2021-01-15T15:56:47.749 INFO:root:teuthology version: 1.0.0-04fddc54 2021-01-15T15:56:47.751 DEBUG:teuthology.run:Teuthology command: teuthology --block --archive /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788567 --name teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi --description rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} --verbose --owner scheduled_teuthology@teuthology --lock -- /tmp/teuthology-worker.bo68_utd.tmp 2021-01-15T15:56:47.775 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T15:56:47.844 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788567 branch: master description: rados/standalone/{mon_election/connectivity supported-random-distro$/{centos_8} workloads/osd} email: ceph-qa@ceph.io first_in_suite: false job_id: '5788567' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.2' overrides: admin_socket: branch: master ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: master sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 owner: scheduled_teuthology@teuthology priority: 80 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 sleep_before_teardown: 0 suite: rados suite_branch: master suite_path: /home/teuthworker/src/git.ceph.com_ceph_master/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master timestamp: 2021-01-15_07:01:02 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.1426 2021-01-15T15:56:47.845 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_master/qa; will attempt to use it 2021-01-15T15:56:47.846 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_master/qa/tasks 2021-01-15T15:56:47.846 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-01-15T15:56:47.849 INFO:teuthology.task.internal:Checking packages... 2021-01-15T15:56:47.877 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '892c51dd3c5f7108e766bea30cd5e0d801b0abd3' 2021-01-15T15:56:47.878 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-01-15T15:56:47.878 INFO:teuthology.packaging:ref: None 2021-01-15T15:56:47.879 INFO:teuthology.packaging:tag: None 2021-01-15T15:56:47.879 INFO:teuthology.packaging:branch: master 2021-01-15T15:56:47.880 INFO:teuthology.packaging:sha1: 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T15:56:47.880 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=master 2021-01-15T15:56:48.510 INFO:teuthology.task.internal:Found packages for ceph version 16.0.0-9042.gbe473e78 2021-01-15T15:56:48.511 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-01-15T15:56:48.527 INFO:teuthology.task.internal:no buildpackages task found 2021-01-15T15:56:48.528 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2021-01-15T15:56:48.557 INFO:teuthology.task.internal.lock_machines:Locking machines... 2021-01-15T15:56:48.557 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T15:56:48.612 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788567'} 2021-01-15T15:56:48.631 DEBUG:teuthology.lock.ops:locked ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T15:56:48.657 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb035160d0>, signals=[15]) 2021-01-15T15:56:48.956 INFO:teuthology.provision.fog.smithi145:Scheduling deploy of centos 8.2 2021-01-15T15:56:49.883 INFO:teuthology.orchestra.console:Power off smithi145 2021-01-15T15:56:49.884 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-01-15T15:56:49.904 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T15:56:50.025 INFO:teuthology.orchestra.console:Power off for smithi145 completed 2021-01-15T15:56:50.126 INFO:teuthology.orchestra.console:Power on smithi145 2021-01-15T15:56:50.127 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2021-01-15T15:56:50.147 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T15:56:54.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T15:56:58.282 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-01-15T15:56:58.492 INFO:teuthology.orchestra.console:Power on for smithi145 completed 2021-01-15T15:56:58.594 INFO:teuthology.provision.fog.smithi145:Waiting for deploy to finish 2021-01-15T15:59:30.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T15:59:39.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T15:59:48.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T15:59:57.718 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:00:06.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:00:15.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:00:24.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:01:30.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:01:31.186 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-01-15T16:01:31.187 DEBUG:teuthology.orchestra.run.smithi145:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2021-01-15T16:01:31.523 INFO:teuthology.provision.fog.smithi145:Node is ready 2021-01-15T16:01:31.523 DEBUG:teuthology.orchestra.run.smithi145:> hostname 2021-01-15T16:01:31.537 INFO:teuthology.orchestra.run.smithi145.stdout:smithi145.front.sepia.ceph.com 2021-01-15T16:01:31.538 DEBUG:teuthology.orchestra.run.smithi145:> grep smithi145.front.sepia.ceph.com /etc/hosts 2021-01-15T16:01:31.592 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.15.145 smithi145.front.sepia.ceph.com smithi145 2021-01-15T16:01:31.593 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2021-01-15T16:01:31.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hosts 2021-01-15T16:01:31.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/172.21.15.145/172.21.15.145/g' /etc/hosts 2021-01-15T16:01:31.786 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i -e 's/smithi145.front.sepia.ceph.com/smithi145/g' /etc/hostname 2021-01-15T16:01:31.849 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname smithi145 2021-01-15T16:01:31.913 INFO:teuthology.provision.fog.smithi145:Deploy complete! 2021-01-15T16:01:31.914 DEBUG:teuthology.parallel:result is None 2021-01-15T16:01:32.014 INFO:teuthology.lock.ops:Checking smithi145.front.sepia.ceph.com 2021-01-15T16:01:32.015 INFO:teuthology.lock.ops:New key found. Updating... 2021-01-15T16:01:32.032 INFO:teuthology.lock.ops:Updating [smithi145.front.sepia.ceph.com]: set os type and version on server 2021-01-15T16:01:32.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:01:32.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:01:32.100 DEBUG:teuthology.orchestra.run.smithi145:> true 2021-01-15T16:01:32.243 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2021-01-15T16:01:32.297 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2021-01-15T16:01:32.297 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2021-01-15T16:01:32.349 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2021-01-15T16:01:32.350 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8 (Core)" 2021-01-15T16:01:32.350 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2021-01-15T16:01:32.350 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2021-01-15T16:01:32.351 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8" 2021-01-15T16:01:32.351 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2021-01-15T16:01:32.351 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-15T16:01:32.351 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2021-01-15T16:01:32.351 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-15T16:01:32.352 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2021-01-15T16:01:32.352 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-15T16:01:32.352 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:32.352 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-15T16:01:32.353 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-15T16:01:32.353 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-15T16:01:32.353 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-15T16:01:32.353 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:32.354 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-01-15T16:01:32.394 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2021-01-15T16:01:32.412 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi145.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDexQhZSJWDzR7OTo02+9IG2RmSmoFFl56JJxj952firCrHW0tM0o0QvXvdbJomOLP7CiHWFvqAotg38EGQ9/y7+nPmqMF7yA/WodBCUFb2C/8+mz4W+gRm0W5GRFB6O/nG21XCpnUA0p9H9P6kjPsQF79F2XcWFZSBbtEfXj/B8nOFLeHlFRYXpWRVKCPvxwHm3UkZoe4zGMzVYz4IxxCJ4tNOkcaxQvsCRvNpzalSGcOhlnWZLeslFcSH/yGp2qJsMhM9Y1yWfexPeLlYgfRfnbAA+/NSytHWikqASN4q4D8Pkrq8UwOqGxQReNsSQXODGPQFV93KhiANmOe+SCvbUls1tuLY9wH5TcLnIv7AK6wRsQULtbnrZkXold4M4GM26OQmWlAhnrPoe6QtcpwMFGIzZoxUT23gV2fZbUAFEpJKTBFf4+OBBaqs3TC/mSjtparqnywnCV34May1MYNwEQRyRGCI9+NcFQooU7ql7bF8y3TH6VOX+MD+njyK7bM= 2021-01-15T16:01:32.413 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-01-15T16:01:32.458 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-01-15T16:01:32.471 INFO:teuthology.task.internal:Saving configuration 2021-01-15T16:01:32.481 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-01-15T16:01:32.495 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-01-15T16:01:32.513 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-01-15 15:56:48.618192', 'description': '/home/teuthworker/archive/teuthology-2021-01-15_07:01:02-rados-master-distro-basic-smithi/5788567', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIQ/Zz+ZDb8Adl8xDRxtRm4aH5YQtWy61IcjjmHNvrN9NDPSuyeSS+hYAl/hyvhEDM8K5tj4B9fXHfkE8wiRsIc=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi145.front.sepia.ceph.com'} 2021-01-15T16:01:32.514 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-01-15T16:01:32.524 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-01-15T16:01:32.525 INFO:teuthology.run_tasks:Running task console_log... 2021-01-15T16:01:32.564 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb034eb9d8>, signals=[15]) 2021-01-15T16:01:32.565 INFO:teuthology.run_tasks:Running task internal.connect... 2021-01-15T16:01:32.575 INFO:teuthology.task.internal:Opening connections... 2021-01-15T16:01:32.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:01:32.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:01:32.647 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-01-15T16:01:32.658 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2021-01-15T16:01:32.792 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2021-01-15T16:01:32.793 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2021-01-15T16:01:32.846 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Linux" 2021-01-15T16:01:32.846 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="8 (Core)" 2021-01-15T16:01:32.847 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2021-01-15T16:01:32.847 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2021-01-15T16:01:32.847 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="8" 2021-01-15T16:01:32.847 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el8" 2021-01-15T16:01:32.847 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-01-15T16:01:32.848 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2021-01-15T16:01:32.848 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-01-15T16:01:32.848 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.centos.org/" 2021-01-15T16:01:32.849 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-01-15T16:01:32.849 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:32.849 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-01-15T16:01:32.849 INFO:teuthology.orchestra.run.smithi145.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-01-15T16:01:32.850 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-01-15T16:01:32.850 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-01-15T16:01:32.850 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:32.851 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2021-01-15T16:01:32.866 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-01-15T16:01:32.885 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-01-15T16:01:32.895 INFO:teuthology.task.internal:Checking for old test directory... 2021-01-15T16:01:32.896 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2021-01-15T16:01:32.910 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-01-15T16:01:32.920 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-01-15T16:01:32.923 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2021-01-15T16:01:32.978 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-01-15T16:01:33.009 INFO:teuthology.run_tasks:Running task kernel... 2021-01-15T16:01:33.030 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-01-15T16:01:33.031 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-01-15T16:01:33.031 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2021-01-15T16:01:33.044 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-01-15T16:01:33.045 INFO:teuthology.task.kernel:Running kernel on smithi145: 4.18.0-193.28.1.el8_2.x86_64 2021-01-15T16:01:33.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2021-01-15T16:01:35.746 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - AppStream 36 MB/s | 6.3 MB 00:00 2021-01-15T16:01:36.850 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Base 25 MB/s | 2.3 MB 00:00 2021-01-15T16:01:37.326 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Extras 111 kB/s | 8.6 kB 00:00 2021-01-15T16:01:37.716 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 26 MB/s | 8.8 MB 00:00 2021-01-15T16:01:39.545 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 4.1 MB/s | 23 kB 00:00 2021-01-15T16:01:40.564 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-01-15T16:01:40.564 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2021-01-15T16:01:40.760 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:01:40.771 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:01:40.771 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:01:40.771 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:01:40.771 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:01:40.772 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 4.18.0-240.1.1.el8_3 CentOS-Base 4.3 M 2021-01-15T16:01:40.772 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2021-01-15T16:01:40.772 INFO:teuthology.orchestra.run.smithi145.stdout: linux-firmware noarch 20200619-101.git3890db36.el8_3 CentOS-Base 101 M 2021-01-15T16:01:40.773 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:01:40.773 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 4.18.0-240.1.1.el8_3 CentOS-Base 30 M 2021-01-15T16:01:40.774 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 4.18.0-240.1.1.el8_3 CentOS-Base 26 M 2021-01-15T16:01:40.774 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:40.774 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:01:40.775 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:01:40.775 INFO:teuthology.orchestra.run.smithi145.stdout:Install 3 Packages 2021-01-15T16:01:40.776 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 1 Package 2021-01-15T16:01:40.776 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:01:40.777 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 162 M 2021-01-15T16:01:40.777 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:01:40.875 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-4.18.0-240.1.1.el8_3.x86_64.rpm 42 MB/s | 4.3 MB 00:00 2021-01-15T16:01:42.259 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-core-4.18.0-240.1.1.el8_3.x86_64. 20 MB/s | 30 MB 00:01 2021-01-15T16:01:42.429 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-4.18.0-240.1.1.el8_3.x86_ 15 MB/s | 26 MB 00:01 2021-01-15T16:01:43.367 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): linux-firmware-20200619-101.git3890db36. 41 MB/s | 101 MB 00:02 2021-01-15T16:01:43.370 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:01:43.370 INFO:teuthology.orchestra.run.smithi145.stdout:Total 62 MB/s | 162 MB 00:02 2021-01-15T16:01:43.463 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:01:43.765 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:01:43.765 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:01:45.491 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:01:45.491 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:01:55.461 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:01:57.294 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/5 2021-01-15T16:01:57.906 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-4.18.0-240.1.1.el8_3.x86_64 2/5 2021-01-15T16:02:00.609 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-4.18.0-240.1.1.el8_3.x86_64 2/5 2021-01-15T16:02:00.955 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-4.18.0-240.1.1.el8_3.x86_64 3/5 2021-01-15T16:02:03.841 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-4.18.0-240.1.1.el8_3.x86_64 3/5 2021-01-15T16:02:04.043 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-4.18.0-240.1.1.el8_3.x86_64 4/5 2021-01-15T16:02:04.282 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-01-15T16:02:23.976 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-4.18.0-240.1.1.el8_3.x86_64 5/5 2021-01-15T16:02:24.327 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-01-15T16:02:24.328 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-4.18.0-240.1.1.el8_3.x86_64 1/5 2021-01-15T16:02:24.328 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-4.18.0-240.1.1.el8_3.x86_64 2/5 2021-01-15T16:02:24.328 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-4.18.0-240.1.1.el8_3.x86_64 3/5 2021-01-15T16:02:24.328 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : linux-firmware-20200619-101.git3890db36.el8_3.noarch 4/5 2021-01-15T16:02:24.770 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-01-15T16:02:24.771 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:02:24.771 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2021-01-15T16:02:24.771 INFO:teuthology.orchestra.run.smithi145.stdout: linux-firmware-20200619-101.git3890db36.el8_3.noarch 2021-01-15T16:02:24.772 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:02:24.772 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:02:24.772 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:24.772 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:24.772 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:24.773 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:02:24.773 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:02:24.912 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel --last | head -n 1 2021-01-15T16:02:25.019 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Fri 15 Jan 2021 04:02:03 PM UTC 2021-01-15T16:02:25.020 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:25.021 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-193.28.1.el8_2.x86_64 Expected: 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:25.021 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-01-15T16:02:25.021 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-01-15T16:02:25.022 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-01-15T16:02:25.022 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2021-01-15T16:02:25.022 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel --last | head -n 1 2021-01-15T16:02:25.110 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-4.18.0-240.1.1.el8_3.x86_64 Fri 15 Jan 2021 04:02:03 PM UTC 2021-01-15T16:02:25.111 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:25.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2021-01-15T16:02:25.142 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2021-01-15T16:02:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2021-01-15T16:02:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.02 2021-01-15T16:02:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 81.el8 2021-01-15T16:02:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2021-01-15T16:02:25.143 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 13 Nov 2020 03:02:55 PM UTC 2021-01-15T16:02:25.144 INFO:teuthology.orchestra.run.smithi145.stdout:Group : System Environment/Base 2021-01-15T16:02:25.144 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 9192705 2021-01-15T16:02:25.144 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2021-01-15T16:02:25.144 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Tue 14 Apr 2020 03:18:37 PM UTC, Key ID 05b555b38483c65d 2021-01-15T16:02:25.145 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.02-81.el8.src.rpm 2021-01-15T16:02:25.145 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Tue 14 Apr 2020 02:54:04 PM UTC 2021-01-15T16:02:25.145 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : kbuilder.bsys.centos.org 2021-01-15T16:02:25.145 INFO:teuthology.orchestra.run.smithi145.stdout:Relocations : (not relocatable) 2021-01-15T16:02:25.146 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : CentOS BuildSystem 2021-01-15T16:02:25.147 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2021-01-15T16:02:25.147 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2021-01-15T16:02:25.147 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2021-01-15T16:02:25.148 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2021-01-15T16:02:25.148 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:02:25.149 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-01-15T16:02:25.149 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2021-01-15T16:02:25.149 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2021-01-15T16:02:25.150 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2021-01-15T16:02:25.150 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:02:25.151 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2021-01-15T16:02:25.152 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-01-15T16:02:25.152 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:25.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-01-15T16:02:26.873 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2021-01-15T16:02:28.925 INFO:teuthology.orchestra.run.smithi145.stderr:done 2021-01-15T16:02:28.927 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2021-01-15T16:02:28.944 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.BR4mBUBEfu 2021-01-15T16:02:28.945 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BR4mBUBEfu 2021-01-15T16:02:29.015 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.BR4mBUBEfu 2021-01-15T16:02:29.175 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.BR4mBUBEfu is 4KB 2021-01-15T16:02:29.223 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.BR4mBUBEfu 2021-01-15T16:02:29.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2021-01-15T16:02:29.298 INFO:teuthology.orchestra.run.smithi145.stdout:2951a06464bf41f6a3bf91594147ecff-0-rescue.conf 2021-01-15T16:02:29.299 INFO:teuthology.orchestra.run.smithi145.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.28.1.el8_2.x86_64.conf 2021-01-15T16:02:29.299 INFO:teuthology.orchestra.run.smithi145.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.el8.x86_64.conf 2021-01-15T16:02:29.299 INFO:teuthology.orchestra.run.smithi145.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-240.1.1.el8_3.x86_64.conf 2021-01-15T16:02:29.300 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 2951a06464bf41f6a3bf91594147ecff-4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:02:29.491 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2021-01-15T16:02:59.522 INFO:teuthology.misc:Re-opening connections... 2021-01-15T16:02:59.523 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:02:59.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:02:59.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:03:18.082 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2021-01-15T16:03:18.083 DEBUG:teuthology.misc:waited 18.56049370765686 2021-01-15T16:03:19.084 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:03:19.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:03:19.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:03:22.147 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2021-01-15T16:03:22.147 DEBUG:teuthology.misc:waited 22.62454319000244 2021-01-15T16:03:23.148 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:03:23.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:03:23.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:03:25.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2021-01-15T16:03:25.223 DEBUG:teuthology.misc:waited 25.69990849494934 2021-01-15T16:03:26.223 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:03:26.224 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:03:26.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:03:28.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2021-01-15T16:03:28.291 DEBUG:teuthology.misc:waited 28.767902612686157 2021-01-15T16:03:29.291 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:03:29.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:03:29.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:04:29.295 DEBUG:teuthology.orchestra.remote:timed out 2021-01-15T16:04:29.297 DEBUG:teuthology.misc:waited 89.774160861969 2021-01-15T16:04:30.298 INFO:teuthology.misc:trying to connect to ubuntu@smithi145.front.sepia.ceph.com 2021-01-15T16:04:30.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:04:30.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:04:30.611 DEBUG:teuthology.orchestra.run.smithi145:> true 2021-01-15T16:04:30.944 DEBUG:teuthology.misc:waited 91.42131853103638 2021-01-15T16:04:31.945 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-01-15T16:04:31.946 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.1.1.el8_3.x86_64"... 2021-01-15T16:04:31.946 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2021-01-15T16:04:31.963 INFO:teuthology.orchestra.run.smithi145.stdout:4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:04:31.964 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.1.1.el8_3.x86_64 vs 4.18.0-240.1.1.el8_3.x86_64 2021-01-15T16:04:31.964 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-01-15T16:04:31.964 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-01-15T16:04:32.965 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-01-15T16:04:32.965 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-01-15T16:04:33.054 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2021-01-15T16:04:33.072 INFO:teuthology.run_tasks:Running task internal.base... 2021-01-15T16:04:33.082 INFO:teuthology.task.internal:Creating test directory... 2021-01-15T16:04:33.083 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-01-15T16:04:33.100 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-01-15T16:04:33.110 INFO:teuthology.run_tasks:Running task internal.archive... 2021-01-15T16:04:33.123 INFO:teuthology.task.internal:Creating archive directory... 2021-01-15T16:04:33.123 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-01-15T16:04:33.157 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-01-15T16:04:33.168 INFO:teuthology.task.internal:Enabling coredump saving... 2021-01-15T16:04:33.168 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-01-15T16:04:33.232 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:04:33.244 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:04:33.245 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-01-15T16:04:33.255 INFO:teuthology.task.internal:Configuring sudo... 2021-01-15T16:04:33.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-01-15T16:04:33.312 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-01-15T16:04:33.328 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-01-15T16:04:33.329 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-01-15T16:04:33.367 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-01-15T16:04:33.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-01-15T16:04:33.499 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-01-15T16:04:33.575 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-01-15T16:04:33.625 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:04:33.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-01-15T16:04:33.690 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2021-01-15T16:04:33.789 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-01-15T16:04:34.131 INFO:teuthology.run_tasks:Running task internal.timer... 2021-01-15T16:04:34.141 INFO:teuthology.task.internal:Starting timer... 2021-01-15T16:04:34.141 INFO:teuthology.run_tasks:Running task pcp... 2021-01-15T16:04:34.165 INFO:teuthology.run_tasks:Running task selinux... 2021-01-15T16:04:34.180 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-01-15T16:04:34.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2021-01-15T16:04:34.276 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: [ OK ] 2021-01-15T16:04:34.277 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-01-15T16:04:34.277 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2021-01-15T16:04:34.298 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2021-01-15T16:04:34.298 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2021-01-15T16:04:34.299 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-01-15T16:04:34.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T16:04:34.363 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-01-15T16:04:34.364 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2021-01-15T16:04:34.443 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-01-15T16:04:34.455 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-01-15T16:04:34.461 INFO:teuthology.repo_utils:Fetching master from origin 2021-01-15T16:04:34.499 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2021-01-15T16:04:34.511 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-01-15T16:04:34.513 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-01-15T16:04:34.848 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-01-15T16:04:35.703 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-01-15T16:04:38.368 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-01-15T16:04:38.386 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:38 +0000 (0:00:00.069) 0:00:00.069 ******** 2021-01-15T16:04:39.791 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2021-01-15T16:04:39.862 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:39 +0000 (0:00:01.476) 0:00:01.545 ******** 2021-01-15T16:04:40.178 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-01-15T16:04:40.180 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-01-15T16:04:40.211 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:40 +0000 (0:00:00.348) 0:00:01.894 ******** 2021-01-15T16:04:40.627 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-01-15T16:04:40.629 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-01-15T16:04:40.661 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:40 +0000 (0:00:00.450) 0:00:02.344 ******** 2021-01-15T16:04:40.687 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-01-15T16:04:40.689 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:40.720 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:40 +0000 (0:00:00.059) 0:00:02.403 ******** 2021-01-15T16:04:41.026 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-01-15T16:04:41.028 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T16:04:41.060 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:41 +0000 (0:00:00.339) 0:00:02.742 ******** 2021-01-15T16:04:41.989 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-01-15T16:04:41.991 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610726681.072513-65044429911568/source", "state": "file", "uid": 0} 2021-01-15T16:04:42.029 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:42 +0000 (0:00:00.969) 0:00:03.711 ******** 2021-01-15T16:04:42.465 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtC 2021-01-15T16:04:42.467 INFO:teuthology.task.ansible.out:toHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-01-15T16:04:42.557 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-01-15T16:04:42.564 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-01-15T16:04:42.709 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:42 +0000 (0:00:00.680) 0:00:04.392 ******** 2021-01-15T16:04:43.346 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2021-01-15T16:04:43.418 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-01-15T16:04:43.420 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:43 +0000 (0:00:00.709) 0:00:05.102 ******** 2021-01-15T16:04:43.914 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-01-15T16:04:43.916 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.183242", "end": "2021-01-15 16:04:43.890457", "rc": 0, "start": "2021-01-15 16:04:43.707215", "stderr": "", "stderr_lines": [], "stdout": "0caff10a478f2674721da1965013b225f3ea5800", "stdout_lines": ["0caff10a478f2674721da1965013b225f3ea5800"]} 2021-01-15T16:04:43.949 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:43 +0000 (0:00:00.530) 0:00:05.632 ******** 2021-01-15T16:04:44.150 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011645", "end": "2021-01-15 16:04:44.126871", "failed_when_result": false, "rc": 0, "start": "2021-01-15 16:04:44.115226", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-01-15T16:04:44.184 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.235) 0:00:05.867 ******** 2021-01-15T16:04:44.211 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-01-15T16:04:44.213 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:44.245 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.060) 0:00:05.928 ******** 2021-01-15T16:04:44.270 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:44.306 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.060) 0:00:05.988 ******** 2021-01-15T16:04:44.331 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-01-15T16:04:44.333 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:44.364 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.058) 0:00:06.047 ******** 2021-01-15T16:04:44.390 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-01-15T16:04:44.392 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:44.423 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.058) 0:00:06.106 ******** 2021-01-15T16:04:44.448 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-01-15T16:04:44.450 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:04:44.481 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.058) 0:00:06.164 ******** 2021-01-15T16:04:44.671 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-01-15T16:04:44.673 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T16:04:44.703 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:04:44 +0000 (0:00:00.222) 0:00:06.386 ******** 2021-01-15T16:04:44.937 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-01-15T16:04:45.125 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-01-15T16:04:45.316 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-01-15T16:04:45.504 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-01-15T16:04:45.693 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-01-15T16:04:45.880 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-01-15T16:04:46.069 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-01-15T16:04:46.260 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-01-15T16:04:46.448 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-01-15T16:04:46.646 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-01-15T16:04:46.838 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-01-15T16:04:47.023 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-01-15T16:04:47.212 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-01-15T16:04:47.401 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-01-15T16:04:47.590 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-01-15T16:04:47.787 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-01-15T16:04:47.976 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-01-15T16:04:48.163 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-01-15T16:04:48.351 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-01-15T16:04:48.541 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-01-15T16:04:48.739 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-01-15T16:04:48.927 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-01-15T16:04:49.121 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-01-15T16:04:49.309 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-01-15T16:04:49.518 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-01-15T16:04:49.711 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-01-15T16:04:49.902 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-01-15T16:04:50.095 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-01-15T16:04:50.297 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-01-15T16:04:50.500 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-01-15T16:04:50.682 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-01-15T16:04:50.886 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-01-15T16:04:51.067 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-01-15T16:04:51.285 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-01-15T16:04:51.485 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-01-15T16:04:51.690 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-01-15T16:04:51.882 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-01-15T16:04:52.066 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-01-15T16:04:52.253 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-01-15T16:04:52.462 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-01-15T16:04:52.650 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-01-15T16:04:52.845 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-01-15T16:04:53.039 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-01-15T16:04:53.255 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-01-15T16:04:53.448 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-01-15T16:04:53.638 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-01-15T16:04:53.838 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-01-15T16:04:54.030 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-01-15T16:04:54.220 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-01-15T16:04:54.411 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-01-15T16:04:54.597 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-01-15T16:04:54.789 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-01-15T16:04:54.983 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-01-15T16:04:55.174 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-01-15T16:04:55.370 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-01-15T16:04:55.590 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-01-15T16:04:55.797 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-01-15T16:04:55.988 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-01-15T16:04:56.180 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-01-15T16:04:56.367 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-01-15T16:04:56.558 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-01-15T16:04:56.759 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-01-15T16:04:56.948 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-01-15T16:04:57.137 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-01-15T16:04:57.317 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-01-15T16:04:57.518 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-01-15T16:04:57.706 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-01-15T16:04:57.906 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-01-15T16:04:58.097 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-01-15T16:04:58.292 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-01-15T16:04:58.486 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-01-15T16:04:58.665 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-01-15T16:04:58.866 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-01-15T16:04:59.049 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-01-15T16:04:59.242 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-01-15T16:04:59.443 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-01-15T16:04:59.631 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-01-15T16:04:59.820 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-01-15T16:04:59.822 INFO:teuthology.task.ansible.out: 2021-01-15T16:05:00.017 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-01-15T16:05:00.206 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-01-15T16:05:00.406 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-01-15T16:05:00.597 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-01-15T16:05:00.789 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-01-15T16:05:00.979 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-01-15T16:05:01.167 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-01-15T16:05:01.360 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-01-15T16:05:01.550 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-01-15T16:05:01.749 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-01-15T16:05:01.946 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-01-15T16:05:02.153 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-01-15T16:05:02.332 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-01-15T16:05:02.529 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-01-15T16:05:02.717 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-01-15T16:05:02.906 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-01-15T16:05:03.095 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-01-15T16:05:03.282 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-01-15T16:05:03.472 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-01-15T16:05:03.664 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-01-15T16:05:03.852 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-01-15T16:05:04.040 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-01-15T16:05:04.238 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-01-15T16:05:04.425 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-01-15T16:05:04.612 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-01-15T16:05:04.800 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-01-15T16:05:04.986 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-01-15T16:05:05.195 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-01-15T16:05:05.394 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-01-15T16:05:05.581 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-01-15T16:05:05.778 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-01-15T16:05:05.966 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-01-15T16:05:06.155 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-01-15T16:05:06.354 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-01-15T16:05:06.543 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-01-15T16:05:06.729 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-01-15T16:05:06.927 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-01-15T16:05:07.114 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-01-15T16:05:07.311 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-01-15T16:05:07.501 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-01-15T16:05:07.688 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-01-15T16:05:07.876 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-01-15T16:05:08.062 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-01-15T16:05:08.260 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-01-15T16:05:08.459 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-01-15T16:05:08.647 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-01-15T16:05:08.833 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-01-15T16:05:09.032 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-01-15T16:05:09.219 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-01-15T16:05:09.416 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-01-15T16:05:09.602 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-01-15T16:05:09.800 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-01-15T16:05:09.987 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-01-15T16:05:10.172 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-01-15T16:05:10.360 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-01-15T16:05:10.557 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-01-15T16:05:10.744 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-01-15T16:05:10.931 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-01-15T16:05:12.455 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-01-15T16:05:13.834 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-01-15T16:05:15.113 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-01-15T16:05:15.229 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:15 +0000 (0:00:30.525) 0:00:36.912 ******** 2021-01-15T16:05:15.256 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-01-15T16:05:15.289 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:15 +0000 (0:00:00.060) 0:00:36.972 ******** 2021-01-15T16:05:15.335 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-01-15T16:05:15.338 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-01-15T16:05:15.340 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-01-15T16:05:15.342 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-01-15T16:05:15.343 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkK 2021-01-15T16:05:15.345 INFO:teuthology.task.ansible.out:ztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop 2021-01-15T16:05:15.345 INFO:teuthology.task.ansible.out: XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 2021-01-15T16:05:15.347 INFO:teuthology.task.ansible.out:HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg 2021-01-15T16:05:15.349 INFO:teuthology.task.ansible.out:e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df 2021-01-15T16:05:15.350 INFO:teuthology.task.ansible.out:79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hm 2021-01-15T16:05:15.352 INFO:teuthology.task.ansible.out:p+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b 2021-01-15T16:05:15.354 INFO:teuthology.task.ansible.out:0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAM 2021-01-15T16:05:15.355 INFO:teuthology.task.ansible.out:Xq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jel 2021-01-15T16:05:15.357 INFO:teuthology.task.ansible.out:opez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxux 2021-01-15T16:05:15.357 INFO:teuthology.task.ansible.out:uehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf 2021-01-15T16:05:15.360 INFO:teuthology.task.ansible.out:0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}]}, "changed": false} 2021-01-15T16:05:15.407 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:15 +0000 (0:00:00.117) 0:00:37.090 ******** 2021-01-15T16:05:16.324 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi145.front.sepia.ceph.com -> localhost] => {"after": "0caff10a478f2674721da1965013b225f3ea5800", "attempts": 1, "before": "0caff10a478f2674721da1965013b225f3ea5800", "changed": false, "remote_url_changed": false} 2021-01-15T16:05:16.356 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:16 +0000 (0:00:00.949) 0:00:38.039 ******** 2021-01-15T16:05:16.640 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-01-15T16:05:16.642 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-01-15T16:05:16.854 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-01-15T16:05:17.065 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-01-15T16:05:17.303 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-01-15T16:05:17.305 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-01-15T16:05:17.496 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-01-15T16:05:17.713 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-01-15T16:05:17.715 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-01-15T16:05:17.912 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-01-15T16:05:18.120 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-01-15T16:05:18.324 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-01-15T16:05:18.335 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:18.557 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-01-15T16:05:18.558 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-01-15T16:05:18.778 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-01-15T16:05:18.780 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-01-15T16:05:18.793 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:18.804 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:19.024 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-01-15T16:05:19.251 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-01-15T16:05:19.253 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-01-15T16:05:19.459 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-01-15T16:05:19.684 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-01-15T16:05:19.877 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-01-15T16:05:20.085 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-01-15T16:05:20.294 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-01-15T16:05:20.513 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2021-01-15T16:05:20.722 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-01-15T16:05:20.724 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-01-15T16:05:20.930 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-01-15T16:05:20.932 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-01-15T16:05:21.138 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-01-15T16:05:21.357 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-01-15T16:05:21.569 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-01-15T16:05:21.571 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-01-15T16:05:21.782 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-01-15T16:05:21.991 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-01-15T16:05:22.199 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-01-15T16:05:22.408 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-01-15T16:05:22.617 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-01-15T16:05:22.824 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-01-15T16:05:22.826 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-01-15T16:05:23.031 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-01-15T16:05:23.240 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-01-15T16:05:23.448 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-01-15T16:05:23.666 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-01-15T16:05:23.877 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-01-15T16:05:24.088 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-01-15T16:05:24.297 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-01-15T16:05:24.507 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-01-15T16:05:24.715 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-01-15T16:05:24.922 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-01-15T16:05:24.924 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-01-15T16:05:25.140 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-01-15T16:05:25.142 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-01-15T16:05:25.347 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-01-15T16:05:25.556 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-01-15T16:05:25.766 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-01-15T16:05:25.977 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-01-15T16:05:25.979 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-01-15T16:05:26.214 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-01-15T16:05:26.216 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-01-15T16:05:26.218 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-01-15T16:05:26.409 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-01-15T16:05:26.624 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-01-15T16:05:26.848 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-01-15T16:05:27.070 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-01-15T16:05:27.291 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-01-15T16:05:27.501 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-01-15T16:05:27.504 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-01-15T16:05:27.711 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-01-15T16:05:27.931 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-01-15T16:05:28.154 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-01-15T16:05:28.156 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-01-15T16:05:28.365 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-01-15T16:05:28.577 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-01-15T16:05:28.788 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-01-15T16:05:28.790 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-01-15T16:05:28.998 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-01-15T16:05:29.199 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-01-15T16:05:29.410 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-01-15T16:05:29.620 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-01-15T16:05:29.838 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-01-15T16:05:30.043 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-01-15T16:05:30.255 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-01-15T16:05:30.452 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-01-15T16:05:30.665 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-01-15T16:05:30.893 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-01-15T16:05:31.093 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-01-15T16:05:31.300 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-01-15T16:05:31.511 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-01-15T16:05:31.729 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-01-15T16:05:31.930 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-01-15T16:05:32.151 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-01-15T16:05:32.359 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-01-15T16:05:32.599 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-01-15T16:05:32.819 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-01-15T16:05:33.025 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-01-15T16:05:33.243 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-01-15T16:05:33.245 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-01-15T16:05:33.452 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-01-15T16:05:33.661 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-01-15T16:05:33.867 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-01-15T16:05:34.076 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-01-15T16:05:34.293 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-01-15T16:05:34.337 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-01-15T16:05:34.502 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-01-15T16:05:34.503 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-01-15T16:05:34.711 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-01-15T16:05:34.930 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-01-15T16:05:35.138 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-01-15T16:05:35.346 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-01-15T16:05:35.555 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-01-15T16:05:35.763 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-01-15T16:05:35.973 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-01-15T16:05:36.192 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-01-15T16:05:36.193 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-01-15T16:05:36.395 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-01-15T16:05:36.603 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-01-15T16:05:36.812 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-01-15T16:05:37.021 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-01-15T16:05:37.240 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-01-15T16:05:37.446 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-01-15T16:05:37.656 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-01-15T16:05:37.658 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-01-15T16:05:37.871 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-01-15T16:05:38.092 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-01-15T16:05:38.310 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-01-15T16:05:38.514 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-01-15T16:05:38.724 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-01-15T16:05:38.937 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-01-15T16:05:39.149 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-01-15T16:05:39.350 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-01-15T16:05:39.570 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-01-15T16:05:39.770 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-01-15T16:05:39.989 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-01-15T16:05:40.208 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-01-15T16:05:40.429 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-01-15T16:05:40.640 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-01-15T16:05:40.860 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-01-15T16:05:41.071 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-01-15T16:05:41.280 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-01-15T16:05:41.489 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-01-15T16:05:41.697 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-01-15T16:05:41.699 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-01-15T16:05:41.907 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-01-15T16:05:42.116 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-01-15T16:05:42.316 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-01-15T16:05:42.527 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-01-15T16:05:42.737 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-01-15T16:05:42.947 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-01-15T16:05:43.167 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-01-15T16:05:43.389 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-01-15T16:05:43.613 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-01-15T16:05:43.833 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-01-15T16:05:44.042 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-01-15T16:05:44.252 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-01-15T16:05:44.462 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-01-15T16:05:44.670 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-01-15T16:05:44.876 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-01-15T16:05:45.084 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-01-15T16:05:45.293 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-01-15T16:05:45.380 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:45 +0000 (0:00:29.023) 0:01:07.063 ******** 2021-01-15T16:05:45.418 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-01-15T16:05:45.420 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.430 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.441 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.442 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.453 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.454 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.466 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.477 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.489 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.698 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-01-15T16:05:45.709 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.710 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:45.919 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-01-15T16:05:46.126 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-01-15T16:05:46.137 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.139 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.149 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.160 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.170 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.172 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.181 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.193 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.204 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.205 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.216 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.227 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.228 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.238 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.249 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.251 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.260 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.271 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.273 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.282 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.293 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.305 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.306 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.316 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.327 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.329 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.339 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.350 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.352 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.361 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.372 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.374 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.384 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.395 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.397 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.406 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.417 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.419 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.429 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.440 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.441 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.451 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.462 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.473 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.484 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.495 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.496 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.506 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.517 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.527 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.529 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.539 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.549 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.551 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.561 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.572 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.573 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.583 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.594 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.605 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.606 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.616 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.617 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.627 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.638 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.649 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.651 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.660 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.671 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.673 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.683 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.693 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.695 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.705 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.715 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.717 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.727 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.737 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.739 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.749 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.759 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.761 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.771 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.782 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.783 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.793 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.804 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.806 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.815 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.826 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.828 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.837 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.848 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.859 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.861 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.870 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.881 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.882 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.892 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.903 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.905 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.914 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.925 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.927 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.936 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.947 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.949 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.958 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.969 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.971 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.980 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.991 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:46.992 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.002 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.013 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.014 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.024 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.036 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.046 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.048 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.058 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.069 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.070 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.080 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.130 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:47 +0000 (0:00:01.749) 0:01:08.813 ******** 2021-01-15T16:05:47.155 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-01-15T16:05:47.157 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:47.190 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:47 +0000 (0:00:00.060) 0:01:08.873 ******** 2021-01-15T16:05:47.417 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi145.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-01-15T16:05:47.613 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-01-15T16:05:47.794 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-01-15T16:05:47.983 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-01-15T16:05:48.160 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-01-15T16:05:48.362 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-01-15T16:05:48.558 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-01-15T16:05:48.746 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-01-15T16:05:48.934 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-01-15T16:05:49.111 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-01-15T16:05:49.298 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-01-15T16:05:49.488 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-01-15T16:05:49.680 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-01-15T16:05:49.884 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-01-15T16:05:50.085 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-01-15T16:05:50.273 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-01-15T16:05:50.463 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-01-15T16:05:50.653 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-01-15T16:05:50.832 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-01-15T16:05:51.021 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-01-15T16:05:51.212 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-01-15T16:05:51.391 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-01-15T16:05:51.581 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-01-15T16:05:51.760 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-01-15T16:05:51.948 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-01-15T16:05:52.128 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-01-15T16:05:52.316 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-01-15T16:05:52.506 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-01-15T16:05:52.683 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-01-15T16:05:52.870 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-01-15T16:05:53.057 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-01-15T16:05:53.245 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-01-15T16:05:53.431 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-01-15T16:05:53.619 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-01-15T16:05:53.796 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-01-15T16:05:53.984 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-01-15T16:05:54.171 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-01-15T16:05:55.406 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-01-15T16:05:55.593 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-01-15T16:05:55.782 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-01-15T16:05:55.969 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-01-15T16:05:56.155 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-01-15T16:05:56.343 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-01-15T16:05:56.530 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-01-15T16:05:56.710 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-01-15T16:05:56.899 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-01-15T16:05:57.097 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-01-15T16:05:57.284 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-01-15T16:05:57.338 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:57 +0000 (0:00:10.146) 0:01:19.020 ******** 2021-01-15T16:05:57.842 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "checksum": "e880c991f1562524752e643d227b8575ac6e2ed1", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "fd5444fc24d885aa1caaed5c7ef528f8", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610726757.3590539-125646481454010/source", "state": "file", "uid": 0} 2021-01-15T16:05:57.889 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:57 +0000 (0:00:00.551) 0:01:19.572 ******** 2021-01-15T16:05:57.916 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-01-15T16:05:57.918 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => { "msg": "Host smithi145.front.sepia.ceph.com is running CentOS 8.2 (Core)" } 2021-01-15T16:05:57.954 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:57 +0000 (0:00:00.065) 0:01:19.637 ******** 2021-01-15T16:05:57.991 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-01-15T16:05:57.993 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-01-15T16:05:58.025 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.071) 0:01:19.708 ******** 2021-01-15T16:05:58.051 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-01-15T16:05:58.054 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:58.088 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.060) 0:01:19.769 ******** 2021-01-15T16:05:58.686 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-01-15T16:05:58.688 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.399974", "end": "2021-01-15 16:05:58.661352", "rc": 0, "start": "2021-01-15 16:05:58.261378", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-01-15T16:05:58.722 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.635) 0:01:20.405 ******** 2021-01-15T16:05:58.748 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-01-15T16:05:58.750 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:58.782 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.060) 0:01:20.465 ******** 2021-01-15T16:05:58.809 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-01-15T16:05:58.811 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:58.842 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.059) 0:01:20.525 ******** 2021-01-15T16:05:58.868 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-01-15T16:05:58.870 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:58.903 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.060) 0:01:20.585 ******** 2021-01-15T16:05:58.928 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:58.968 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:58 +0000 (0:00:00.065) 0:01:20.651 ******** 2021-01-15T16:05:58.996 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-01-15T16:05:58.998 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.050 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.082) 0:01:20.733 ******** 2021-01-15T16:05:59.079 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-01-15T16:05:59.080 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.127 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.076) 0:01:20.810 ******** 2021-01-15T16:05:59.163 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-01-15T16:05:59.165 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T16:05:59.198 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.069) 0:01:20.879 ******** 2021-01-15T16:05:59.222 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-01-15T16:05:59.223 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.255 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.058) 0:01:20.938 ******** 2021-01-15T16:05:59.280 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-01-15T16:05:59.282 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.313 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.058) 0:01:20.996 ******** 2021-01-15T16:05:59.340 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-01-15T16:05:59.342 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T16:05:59.381 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.067) 0:01:21.063 ******** 2021-01-15T16:05:59.406 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-01-15T16:05:59.408 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.440 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.060) 0:01:21.123 ******** 2021-01-15T16:05:59.468 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T16:05:59.501 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.059) 0:01:21.183 ******** 2021-01-15T16:05:59.527 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-01-15T16:05:59.528 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.559 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.058) 0:01:21.242 ******** 2021-01-15T16:05:59.596 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-01-15T16:05:59.597 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.632 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.072) 0:01:21.315 ******** 2021-01-15T16:05:59.659 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-01-15T16:05:59.660 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.693 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.061) 0:01:21.376 ******** 2021-01-15T16:05:59.721 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-01-15T16:05:59.723 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.759 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.065) 0:01:21.442 ******** 2021-01-15T16:05:59.787 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-01-15T16:05:59.788 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T16:05:59.821 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.062) 0:01:21.504 ******** 2021-01-15T16:05:59.848 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-01-15T16:05:59.850 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.882 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.060) 0:01:21.565 ******** 2021-01-15T16:05:59.909 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-01-15T16:05:59.911 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:05:59.945 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:05:59 +0000 (0:00:00.063) 0:01:21.628 ******** 2021-01-15T16:05:59.972 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.006 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.060) 0:01:21.688 ******** 2021-01-15T16:06:00.032 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-01-15T16:06:00.033 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.065 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.059) 0:01:21.748 ******** 2021-01-15T16:06:00.091 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-01-15T16:06:00.093 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.124 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.059) 0:01:21.807 ******** 2021-01-15T16:06:00.150 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-01-15T16:06:00.152 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.184 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.059) 0:01:21.867 ******** 2021-01-15T16:06:00.211 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-01-15T16:06:00.212 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-01-15T16:06:00.245 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.060) 0:01:21.927 ******** 2021-01-15T16:06:00.270 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-01-15T16:06:00.272 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.304 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.059) 0:01:21.986 ******** 2021-01-15T16:06:00.329 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-01-15T16:06:00.331 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:00.364 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.059) 0:01:22.046 ******** 2021-01-15T16:06:00.585 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-01-15T16:06:00.587 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T16:06:00.620 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:00 +0000 (0:00:00.256) 0:01:22.303 ******** 2021-01-15T16:06:01.341 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-01-15T16:06:01.342 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-01-15T16:06:01.778 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-01-15T16:06:01.813 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:01 +0000 (0:00:01.192) 0:01:23.496 ******** 2021-01-15T16:06:02.287 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-01-15T16:06:02.289 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-01-15T16:06:02.715 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-01-15T16:06:02.755 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:02 +0000 (0:00:00.941) 0:01:24.438 ******** 2021-01-15T16:06:02.792 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:02.825 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:02 +0000 (0:00:00.069) 0:01:24.508 ******** 2021-01-15T16:06:02.851 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-01-15T16:06:02.853 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:02.884 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:02 +0000 (0:00:00.059) 0:01:24.567 ******** 2021-01-15T16:06:02.910 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:02.945 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:02 +0000 (0:00:00.059) 0:01:24.627 ******** 2021-01-15T16:06:02.972 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-01-15T16:06:02.974 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.018 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.074) 0:01:24.701 ******** 2021-01-15T16:06:03.045 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-01-15T16:06:03.047 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.080 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.061) 0:01:24.762 ******** 2021-01-15T16:06:03.106 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-01-15T16:06:03.109 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.148 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.068) 0:01:24.831 ******** 2021-01-15T16:06:03.176 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-01-15T16:06:03.178 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.222 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.073) 0:01:24.905 ******** 2021-01-15T16:06:03.248 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-01-15T16:06:03.249 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.285 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.062) 0:01:24.967 ******** 2021-01-15T16:06:03.311 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-01-15T16:06:03.313 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:03.354 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:03 +0000 (0:00:00.068) 0:01:25.036 ******** 2021-01-15T16:06:04.521 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-01-15T16:06:04.523 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.981856", "end": "2021-01-15 16:06:04.500373", "rc": 0, "start": "2021-01-15 16:06:03.518517", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:06:04.555 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:04 +0000 (0:00:01.201) 0:01:26.238 ******** 2021-01-15T16:06:11.534 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-01-15T16:06:11.536 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: krb5-workstation", "Installed: krb5-workstation-1.18.2-5.el8.x86_64", "Installed: krb5-libs-1.18.2-5.el8.x86_64", "Installed: libkadm5-1.18.2-5.el8.x86_64", "Removed: krb5-workstation-1.17-18.el8.x86_64", "Removed: libkadm5-1.17-18.el8.x86_64", "Removed: krb5-libs-1.17-18.el8.x86_64"]} 2021-01-15T16:06:11.568 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:11 +0000 (0:00:07.012) 0:01:33.251 ******** 2021-01-15T16:06:11.595 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-01-15T16:06:11.596 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:11.629 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:11 +0000 (0:00:00.061) 0:01:33.312 ******** 2021-01-15T16:06:11.657 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-01-15T16:06:11.659 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:11.692 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:11 +0000 (0:00:00.061) 0:01:33.373 ******** 2021-01-15T16:06:11.721 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:11.753 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:11 +0000 (0:00:00.062) 0:01:33.436 ******** 2021-01-15T16:06:11.787 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:11.815 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:11 +0000 (0:00:00.062) 0:01:33.498 ******** 2021-01-15T16:06:12.282 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-01-15T16:06:12.284 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-01-15T16:06:12.317 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:12 +0000 (0:00:00.502) 0:01:34.000 ******** 2021-01-15T16:06:15.219 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-01-15T16:06:15.221 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2021-01-15T16:06:15.264 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:15 +0000 (0:00:02.946) 0:01:36.947 ******** 2021-01-15T16:06:15.770 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi145.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-01-15T16:06:16.246 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-01-15T16:06:16.690 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-01-15T16:06:16.746 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:16 +0000 (0:00:01.478) 0:01:38.425 ******** 2021-01-15T16:06:16.950 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-01-15T16:06:16.952 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T16:06:16.985 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:16 +0000 (0:00:00.242) 0:01:38.667 ******** 2021-01-15T16:06:17.533 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-01-15T16:06:17.535 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610726777.029655-43624803516900/source", "state": "file", "uid": 0} 2021-01-15T16:06:17.951 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-01-15T16:06:18.382 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-01-15T16:06:18.423 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:18 +0000 (0:00:01.438) 0:01:40.106 ******** 2021-01-15T16:06:18.449 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-01-15T16:06:18.451 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:18.484 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:18 +0000 (0:00:00.060) 0:01:40.166 ******** 2021-01-15T16:06:18.520 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-01-15T16:06:18.521 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-01-15T16:06:18.555 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:18 +0000 (0:00:00.071) 0:01:40.238 ******** 2021-01-15T16:06:18.753 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi145.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-01-15T16:06:18.785 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:18 +0000 (0:00:00.230) 0:01:40.468 ******** 2021-01-15T16:06:18.833 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-01-15T16:06:18.835 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T16:06:18.856 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T16:06:18.867 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-01-15T16:06:18.918 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:18 +0000 (0:00:00.132) 0:01:40.601 ******** 2021-01-15T16:06:19.946 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-01-15T16:06:19.948 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2021-01-15T16:06:19.981 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:19 +0000 (0:00:01.063) 0:01:41.664 ******** 2021-01-15T16:06:20.009 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:20.043 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:20 +0000 (0:00:00.061) 0:01:41.726 ******** 2021-01-15T16:06:20.045 INFO:teuthology.task.ansible.out: 2021-01-15T16:06:20.489 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-01-15T16:06:20.491 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-01-15T16:06:20.523 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:20 +0000 (0:00:00.479) 0:01:42.206 ******** 2021-01-15T16:06:20.980 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-01-15T16:06:20.981 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-01-15T16:06:21.020 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:21 +0000 (0:00:00.497) 0:01:42.703 ******** 2021-01-15T16:06:21.049 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi145.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-01-15T16:06:21.073 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-01-15T16:06:21.108 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:21 +0000 (0:00:00.087) 0:01:42.790 ******** 2021-01-15T16:06:21.323 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-01-15T16:06:21.325 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T16:06:21.367 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:21 +0000 (0:00:00.259) 0:01:43.050 ******** 2021-01-15T16:06:21.593 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-01-15T16:06:21.633 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:21 +0000 (0:00:00.265) 0:01:43.316 ******** 2021-01-15T16:06:21.665 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:06:21.698 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:21 +0000 (0:00:00.065) 0:01:43.381 ******** 2021-01-15T16:06:22.194 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-01-15T16:06:22.227 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:22 +0000 (0:00:00.528) 0:01:43.909 ******** 2021-01-15T16:06:22.924 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-01-15T16:06:22.925 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-01-15 16:04:00 UTC", "ActiveEnterTimestampMonotonic": "13526184", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket system.slice -.mount nss-lookup.target local-fs.target network.target var-run.mount systemd-tmpfiles-setup.service basic.target sysinit.target tmp.mount time-sync.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-01-15 16:04:00 UTC", "AssertTimestampMonotonic": "13525297", "Before": "shutdown.target getty@tty1.service xdm.service plymouth-quit.service multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-01-15 16:04:00 UTC", "ConditionTimestampMonotonic": "13525296", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "Co 2021-01-15T16:06:22.927 INFO:teuthology.task.ansible.out:ntrolGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "939", "ExecMainStartTimestamp": "Fri 2021-01-15 16:04:00 UTC", "ExecMainStartTimestampMonotonic": "13526145", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-01-15 16:04:00 UTC", "InactiveExitTimestampMonotonic": "13526184", "InvocationID": "dfd79df5d47d40d799f6b1573d952583", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", " 2021-01-15T16:06:22.928 INFO:teuthology.task.ansible.out:LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127261", "LimitNPROCSoft": "127261", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127261", "LimitSIGPENDINGSoft": "127261", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "939", "MemoryAccounting": "yes", "MemoryCurrent": "1781760", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", 2021-01-15T16:06:22.930 INFO:teuthology.task.ansible.out:"Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-01-15 16:04:00 UTC", "StateChangeTimestampMonotonic": "13526184", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203618", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-01-15 16:04:00 UTC", "WatchdogTimestampMonotonic": "13526183", "WatchdogUSec": "0"}} 2021-01-15T16:06:22.962 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:22 +0000 (0:00:00.732) 0:01:44.642 ******** 2021-01-15T16:06:23.168 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-01-15T16:06:23.170 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001759", "end": "2021-01-15 16:06:23.150819", "rc": 0, "start": "2021-01-15 16:06:23.149060", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-01-15T16:06:23.203 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:23 +0000 (0:00:00.244) 0:01:44.886 ******** 2021-01-15T16:06:25.970 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils", "Installed: python3-libsemanage-2.9-3.el8.x86_64", "Installed: libsemanage-2.9-3.el8.x86_64", "Removed: python3-libsemanage-2.9-2.el8.x86_64", "Removed: libsemanage-2.9-2.el8.x86_64"]} 2021-01-15T16:06:26.009 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:06:26 +0000 (0:00:02.805) 0:01:47.691 ******** 2021-01-15T16:07:04.197 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-targeted-3.14.3-54.el8.noarch", "Installed: selinux-policy-3.14.3-54.el8.noarch", "Removed: selinux-policy-3.14.3-41.el8_2.8.noarch", "Removed: selinux-policy-targeted-3.14.3-41.el8_2.8.noarch"]} 2021-01-15T16:07:04.232 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:04 +0000 (0:00:38.223) 0:02:25.915 ******** 2021-01-15T16:07:04.935 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-01-15T16:07:04.936 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-01-15T16:07:04.970 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:04 +0000 (0:00:00.737) 0:02:26.652 ******** 2021-01-15T16:07:21.443 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-01-15T16:07:21.446 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.284316", "end": "2021-01-15 16:07:21.420163", "failed_when_result": false, "rc": 0, "start": "2021-01-15 16:07:05.135847", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-01-15T16:07:21.476 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:21 +0000 (0:00:16.506) 0:02:43.159 ******** 2021-01-15T16:07:21.935 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-01-15T16:07:21.937 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-01-15T16:07:21.974 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:21 +0000 (0:00:00.497) 0:02:43.657 ******** 2021-01-15T16:07:22.188 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-01-15T16:07:22.190 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.021065", "end": "2021-01-15 16:07:22.167561", "rc": 0, "start": "2021-01-15 16:07:22.146496", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:07:22.223 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:22 +0000 (0:00:00.248) 0:02:43.906 ******** 2021-01-15T16:07:22.427 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.013201", "end": "2021-01-15 16:07:22.406088", "rc": 0, "start": "2021-01-15 16:07:22.392887", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:07:22.462 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:22 +0000 (0:00:00.239) 0:02:44.145 ******** 2021-01-15T16:07:39.168 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.502010", "end": "2021-01-15 16:07:39.141284", "rc": 0, "start": "2021-01-15 16:07:22.639274", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:07:39.221 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:16.758) 0:03:00.903 ******** 2021-01-15T16:07:39.438 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-01-15T16:07:39.480 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:00.255) 0:03:01.159 ******** 2021-01-15T16:07:39.524 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-01-15T16:07:39.556 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:00.079) 0:03:01.238 ******** 2021-01-15T16:07:39.594 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-01-15T16:07:39.625 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:00.069) 0:03:01.307 ******** 2021-01-15T16:07:39.667 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-01-15T16:07:39.672 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-01-15T16:07:39.706 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:00.079) 0:03:01.386 ******** 2021-01-15T16:07:39.745 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-01-15T16:07:39.775 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:39 +0000 (0:00:00.071) 0:03:01.458 ******** 2021-01-15T16:07:39.964 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-01-15T16:07:39.966 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-01-15T16:07:40.003 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:40 +0000 (0:00:00.227) 0:03:01.686 ******** 2021-01-15T16:07:40.193 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-01-15T16:07:40.233 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:40 +0000 (0:00:00.229) 0:03:01.916 ******** 2021-01-15T16:07:40.260 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:40.296 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:40 +0000 (0:00:00.063) 0:03:01.979 ******** 2021-01-15T16:07:40.320 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:40.352 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:40 +0000 (0:00:00.056) 0:03:02.035 ******** 2021-01-15T16:07:40.565 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi145.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-01-15T16:07:40.602 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:40 +0000 (0:00:00.246) 0:03:02.282 ******** 2021-01-15T16:07:41.142 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-01-15T16:07:41.181 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:41 +0000 (0:00:00.581) 0:03:02.864 ******** 2021-01-15T16:07:41.223 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi145"}, "changed": false} 2021-01-15T16:07:41.258 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:41 +0000 (0:00:00.073) 0:03:02.938 ******** 2021-01-15T16:07:42.272 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi145.front.sepia.ceph.com", "ansible_hostname": "smithi145", "ansible_nodename": "smithi145"}, "changed": true, "name": "smithi145"} 2021-01-15T16:07:42.309 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:42 +0000 (0:00:01.051) 0:03:03.990 ******** 2021-01-15T16:07:42.777 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-01-15T16:07:42.826 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:42 +0000 (0:00:00.513) 0:03:04.503 ******** 2021-01-15T16:07:43.297 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-01-15T16:07:43.299 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-01-15T16:07:43.334 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:43 +0000 (0:00:00.512) 0:03:05.016 ******** 2021-01-15T16:07:43.974 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-01-15T16:07:44.006 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:44 +0000 (0:00:00.673) 0:03:05.689 ******** 2021-01-15T16:07:44.465 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-01-15T16:07:44.498 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:44 +0000 (0:00:00.491) 0:03:06.181 ******** 2021-01-15T16:07:44.687 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-01-15T16:07:44.724 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:44 +0000 (0:00:00.225) 0:03:06.406 ******** 2021-01-15T16:07:44.913 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-01-15T16:07:44.949 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:44 +0000 (0:00:00.225) 0:03:06.632 ******** 2021-01-15T16:07:45.165 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-01-15T16:07:45.170 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-01-15T16:07:45.204 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:45 +0000 (0:00:00.253) 0:03:06.885 ******** 2021-01-15T16:07:45.661 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-01-15T16:07:45.663 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-01-15T16:07:45.695 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:45 +0000 (0:00:00.493) 0:03:07.378 ******** 2021-01-15T16:07:46.160 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-01-15T16:07:46.195 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:46 +0000 (0:00:00.499) 0:03:07.878 ******** 2021-01-15T16:07:48.366 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-01-15T16:07:48.368 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.976333", "end": "2021-01-15 16:07:48.338902", "rc": 0, "start": "2021-01-15 16:07:46.362569", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:07:48.409 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:48 +0000 (0:00:02.213) 0:03:10.091 ******** 2021-01-15T16:07:48.970 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-01-15T16:07:49.414 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-01-15T16:07:49.864 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-01-15T16:07:49.900 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:49 +0000 (0:00:01.491) 0:03:11.583 ******** 2021-01-15T16:07:50.376 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-01-15T16:07:50.441 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:50 +0000 (0:00:00.539) 0:03:12.123 ******** 2021-01-15T16:07:50.944 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-01-15T16:07:51.417 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-01-15T16:07:51.859 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-01-15T16:07:51.920 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:51 +0000 (0:00:01.480) 0:03:13.603 ******** 2021-01-15T16:07:51.951 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-01-15T16:07:52.006 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.084) 0:03:13.687 ******** 2021-01-15T16:07:52.044 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.077 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.072) 0:03:13.760 ******** 2021-01-15T16:07:52.129 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.151 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.215 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.132) 0:03:13.892 ******** 2021-01-15T16:07:52.250 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.286 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.076) 0:03:13.969 ******** 2021-01-15T16:07:52.338 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-01-15T16:07:52.370 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.082) 0:03:14.051 ******** 2021-01-15T16:07:52.422 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.465 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.096) 0:03:14.148 ******** 2021-01-15T16:07:52.514 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-01-15T16:07:52.516 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.562 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.097) 0:03:14.245 ******** 2021-01-15T16:07:52.781 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007177", "end": "2021-01-15 16:07:52.756075", "msg": "non-zero return code", "rc": 1, "start": "2021-01-15 16:07:52.748898", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-01-15T16:07:52.818 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.252) 0:03:14.498 ******** 2021-01-15T16:07:52.853 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:07:52.885 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:52 +0000 (0:00:00.070) 0:03:14.568 ******** 2021-01-15T16:07:55.169 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-01-15T16:07:55.171 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-01-15T16:07:55.203 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:55 +0000 (0:00:02.317) 0:03:16.886 ******** 2021-01-15T16:07:57.350 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-01-15T16:07:57.351 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2021-01-15T16:07:57.386 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:07:57 +0000 (0:00:02.183) 0:03:19.069 ******** 2021-01-15T16:08:39.456 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-01-15T16:08:39.457 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: util-linux-2.32.1-24.el8.x86_64", "Installed: util-linux-user-2.32.1-24.el8.x86_64", "Installed: python3-libdnf-0.48.0-5.el8.x86_64", "Installed: libdnf-0.48.0-5.el8.x86_64", "Installed: python3-librepo-1.12.0-2.el8.x86_64", "Installed: boost-chrono-1.66.0-10.el8.x86_64", "Installed: xfsprogs-5.0.0-4.el8.x86_64", "Installed: libfdisk-2.32.1-24.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: gdb-8.2-12.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: nfs-utils-1:2.3.3-35.el8.x86_64", "Installed: gcc-8.3.1-5.1.el8.x86_64", "Installed: gdb-headless-8.2-12.el8.x86_64", "Installed: libuuid-2.32.1-24.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: gcc-gdb-plugin-8.3.1-5.1.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: python3-virtualenv-15.1.0-19.module_el8.3.0+562+e162826a.noarch", "Installed: libgcc-8.3.1-5.1.el8.x86_64", "Installed: libgcrypt-1.8.5-4.el8.x86_64", "Installed: yum-utils-4.0.17-5.el8.no 2021-01-15T16:08:39.459 INFO:teuthology.task.ansible.out:arch", "Installed: libgomp-8.3.1-5.1.el8.x86_64", "Installed: yum-4.2.23-4.el8.noarch", "Installed: perl-Git-2.27.0-1.el8.noarch", "Installed: boost-system-1.66.0-10.el8.x86_64", "Installed: boost-thread-1.66.0-10.el8.x86_64", "Installed: librepo-1.12.0-2.el8.x86_64", "Installed: podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64", "Installed: podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64", "Installed: podman-docker-2.0.5-5.module_el8.3.0+512+b3b58dca.noarch", "Installed: dnf-4.2.23-4.el8.noarch", "Installed: dnf-data-4.2.23-4.el8.noarch", "Installed: dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: python3-nose-1.3.7-30.module_el8.3.0+562+e162826a.noarch", "Installed: libsmartcols-2.32.1-24.el8.x86_64", "Installed: openssl-1:1.1.1g-12.el8_3.x86_64", "Installed: lsof-4.93.2-1.el8.x86_64", "Installed: valgrind-1:3.16.0-2.el8.x86_64", "Installed: libblkid-2.32.1-24.el8.x86_64", "Installed: git-2.27.0-1.el8.x86_64", "Installed: openssl-libs-1:1.1.1g-12.el8_3.x86_64", "Installed: git-core-2.27.0-1.el8.x86_64", "Installed: git-core-doc-2.27.0-1.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: python3-hawkey-0.48.0-5.el8.x86_64", "Installed: cpp-8.3.1-5.1.el8.x86_64", "Installed: libmodulemd-2.9.4-2.el8.x86_64", "Installed: libmount-2.32.1-24.el8.x86_64", "Installed: python3-dnf-4.2.23-4.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: sysstat-11.7.3-5.el8.x86_64", "Installed: seavgabios-bin-1.13.0-2.module_el8.3.0+555+a55c8938.noarch", "Removed: python3-virtualenv-15.1.0-19.module_el8.1.0+245+c39af44f.noarch", "Removed: openssl-1:1.1.1c-15.el8.x86_64", "Removed: qemu-img-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm- 2021-01-15T16:08:39.461 INFO:teuthology.task.ansible.out:common-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm-core-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: openssl-libs-1:1.1.1c-15.el8.x86_64", "Removed: librepo-1.11.0-2.el8.x86_64", "Removed: libsmartcols-2.32.1-22.el8.x86_64", "Removed: gcc-8.3.1-4.5.el8.x86_64", "Removed: gcc-gdb-plugin-8.3.1-4.5.el8.x86_64", "Removed: gdb-8.2-6.el8.x86_64", "Removed: gdb-headless-8.2-6.el8.x86_64", "Removed: seavgabios-bin-1.11.1-4.module_el8.1.0+248+298dec18.noarch", "Removed: boost-chrono-1.66.0-6.el8.x86_64", "Removed: git-2.18.2-2.el8_1.x86_64", "Removed: git-core-2.18.2-2.el8_1.x86_64", "Removed: git-core-doc-2.18.2-2.el8_1.noarch", "Removed: boost-system-1.66.0-6.el8.x86_64", "Removed: boost-thread-1.66.0-6.el8.x86_64", "Removed: libuuid-2.32.1-22.el8.x86_64", "Removed: podman-1.6.4-4.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-docker-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: podman-manpages-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: libblkid-2.32.1-22.el8.x86_64", "Removed: libdnf-0.35.1-9.el8_1.x86_64", "Removed: perl-Git-2.18.2-2.el8_1.noarch", "Removed: lsof-4.91-2.el8.x86_64", "Removed: cpp-8.3.1-4.5.el8.x86_64", "Removed: sysstat-11.7.3-2.el8.x86_64", "Removed: libfdisk-2.32.1-22.el8.x86_64", "Removed: python3-dnf-4.2.7-7.el8_1.noarch", "Removed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: libgcc-8.3.1-5.el8.0.2.x86_64", "Removed: libgcrypt-1.8.3-4.el8.x86_64", "Removed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Removed: libgomp-8.3.1-4.5.el8.x86_64", "Removed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Removed: python3-librepo-1.11.0-2.el8.x86_64", "Removed: util-linux-2.32.1-22.el8.x86_64", "Removed: util-linux-user-2.32.1-22.el8.x86_64", "Removed: valgrind-1:3.15.0-9.el8.x86_64", "Removed: python3-nose-1.3.7-30.module_el8.1.0+245+c39af44f.noarch", "Removed: dnf-4.2.7-7.el8_1.noarch", "Removed: dnf-data-4.2.7-7.el8_1.noarch", "Removed: libmount-2.32.1-22.el8.x86_64", "Removed: dnf-plu 2021-01-15T16:08:39.462 INFO:teuthology.task.ansible.out:gins-core-4.0.8-3.el8.noarch", "Removed: nfs-utils-1:2.3.3-26.el8.x86_64", "Removed: xfsprogs-5.0.0-2.el8.x86_64", "Removed: yum-4.2.7-7.el8_1.noarch", "Removed: yum-utils-4.0.8-3.el8.noarch"]} 2021-01-15T16:08:39.506 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:39 +0000 (0:00:42.119) 0:04:01.189 ******** 2021-01-15T16:08:39.533 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-01-15T16:08:39.536 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:39.568 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:39 +0000 (0:00:00.061) 0:04:01.250 ******** 2021-01-15T16:08:40.646 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-01-15T16:08:40.649 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-01-15T16:08:40.683 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:40 +0000 (0:00:01.114) 0:04:02.365 ******** 2021-01-15T16:08:40.710 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:40.746 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:40 +0000 (0:00:00.063) 0:04:02.429 ******** 2021-01-15T16:08:41.048 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-01-15T16:08:41.050 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "polkit.service system.slice sysinit.target basic.target dbus.service dbus.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ipset.service shutdown.target iptables.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNode 2021-01-15T16:08:41.052 INFO:teuthology.task.ansible.out:s": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127261", "LimitNPROCSoft": "127261", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "L 2021-01-15T16:08:41.054 INFO:teuthology.task.ansible.out:imitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127261", "LimitSIGPENDINGSoft": "127261", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWe 2021-01-15T16:08:41.055 INFO:teuthology.task.ansible.out:ight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203618", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-01-15T16:08:41.084 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:41 +0000 (0:00:00.336) 0:04:02.766 ******** 2021-01-15T16:08:41.373 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-01-15T16:08:41.375 INFO:teuthology.task.ansible.out:fatal: [smithi145.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-01-15T16:08:41.406 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:41 +0000 (0:00:00.323) 0:04:03.089 ******** 2021-01-15T16:08:42.139 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-01-15T16:08:42.145 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-01-15T16:08:42.176 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.769) 0:04:03.859 ******** 2021-01-15T16:08:42.209 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.254 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.077) 0:04:03.937 ******** 2021-01-15T16:08:42.280 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.313 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.059) 0:04:03.996 ******** 2021-01-15T16:08:42.339 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.372 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.058) 0:04:04.055 ******** 2021-01-15T16:08:42.408 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-01-15T16:08:42.410 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.448 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.076) 0:04:04.131 ******** 2021-01-15T16:08:42.479 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.511 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.062) 0:04:04.194 ******** 2021-01-15T16:08:42.538 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-01-15T16:08:42.540 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.582 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.070) 0:04:04.265 ******** 2021-01-15T16:08:42.623 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-01-15T16:08:42.663 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.080) 0:04:04.346 ******** 2021-01-15T16:08:42.700 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.739 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.072) 0:04:04.419 ******** 2021-01-15T16:08:42.763 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.801 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.064) 0:04:04.484 ******** 2021-01-15T16:08:42.850 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.852 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.888 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.082) 0:04:04.566 ******** 2021-01-15T16:08:42.912 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:42.948 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:42 +0000 (0:00:00.062) 0:04:04.629 ******** 2021-01-15T16:08:42.976 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-01-15T16:08:43.012 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.063) 0:04:04.692 ******** 2021-01-15T16:08:43.037 INFO:teuthology.task.ansible.out: 2021-01-15T16:08:43.038 INFO:teuthology.task.ansible.out:TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.075 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.065) 0:04:04.757 ******** 2021-01-15T16:08:43.114 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-01-15T16:08:43.116 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.150 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.074) 0:04:04.832 ******** 2021-01-15T16:08:43.177 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.210 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.060) 0:04:04.893 ******** 2021-01-15T16:08:43.237 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.270 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.059) 0:04:04.953 ******** 2021-01-15T16:08:43.297 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.329 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.059) 0:04:05.012 ******** 2021-01-15T16:08:43.358 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.395 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.065) 0:04:05.078 ******** 2021-01-15T16:08:43.423 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-01-15T16:08:43.425 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.460 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.064) 0:04:05.142 ******** 2021-01-15T16:08:43.487 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.520 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.060) 0:04:05.203 ******** 2021-01-15T16:08:43.547 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.591 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.071) 0:04:05.274 ******** 2021-01-15T16:08:43.623 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.659 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.067) 0:04:05.342 ******** 2021-01-15T16:08:43.690 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.731 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.071) 0:04:05.414 ******** 2021-01-15T16:08:43.759 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.794 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.063) 0:04:05.477 ******** 2021-01-15T16:08:43.821 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.855 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.061) 0:04:05.538 ******** 2021-01-15T16:08:43.881 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-01-15T16:08:43.883 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.916 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.060) 0:04:05.598 ******** 2021-01-15T16:08:43.943 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:43.977 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:43 +0000 (0:00:00.061) 0:04:05.659 ******** 2021-01-15T16:08:44.004 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-01-15T16:08:44.006 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.037 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.060) 0:04:05.720 ******** 2021-01-15T16:08:44.064 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-01-15T16:08:44.066 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.099 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.059) 0:04:05.780 ******** 2021-01-15T16:08:44.136 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-01-15T16:08:44.138 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.194 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.095) 0:04:05.876 ******** 2021-01-15T16:08:44.219 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-01-15T16:08:44.220 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.258 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.062) 0:04:05.938 ******** 2021-01-15T16:08:44.282 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.322 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.066) 0:04:06.005 ******** 2021-01-15T16:08:44.361 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.402 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.077) 0:04:06.082 ******** 2021-01-15T16:08:44.438 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-01-15T16:08:44.440 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.495 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.094) 0:04:06.177 ******** 2021-01-15T16:08:44.524 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.557 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.063) 0:04:06.240 ******** 2021-01-15T16:08:44.583 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.616 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.059) 0:04:06.299 ******** 2021-01-15T16:08:44.645 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.696 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.079) 0:04:06.378 ******** 2021-01-15T16:08:44.725 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.758 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.061) 0:04:06.440 ******** 2021-01-15T16:08:44.785 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:44.818 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:44 +0000 (0:00:00.060) 0:04:06.500 ******** 2021-01-15T16:08:45.024 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-01-15T16:08:45.026 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-01-15T16:08:45.057 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.239) 0:04:06.740 ******** 2021-01-15T16:08:45.262 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T16:08:45.297 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.237) 0:04:06.978 ******** 2021-01-15T16:08:45.493 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-01-15T16:08:45.494 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-01-15T16:08:45.527 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.232) 0:04:07.210 ******** 2021-01-15T16:08:45.554 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.587 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.059) 0:04:07.270 ******** 2021-01-15T16:08:45.615 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.650 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.059) 0:04:07.330 ******** 2021-01-15T16:08:45.676 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.710 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.062) 0:04:07.392 ******** 2021-01-15T16:08:45.739 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-01-15T16:08:45.741 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.772 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.062) 0:04:07.455 ******** 2021-01-15T16:08:45.799 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.833 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.060) 0:04:07.515 ******** 2021-01-15T16:08:45.858 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-01-15T16:08:45.860 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.891 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.059) 0:04:07.574 ******** 2021-01-15T16:08:45.918 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:45.952 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:45 +0000 (0:00:00.061) 0:04:07.635 ******** 2021-01-15T16:08:45.980 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-01-15T16:08:45.982 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.014 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.060) 0:04:07.696 ******** 2021-01-15T16:08:46.040 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.074 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:07.756 ******** 2021-01-15T16:08:46.099 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-01-15T16:08:46.101 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.140 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.066) 0:04:07.823 ******** 2021-01-15T16:08:46.167 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-01-15T16:08:46.169 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.203 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.063) 0:04:07.886 ******** 2021-01-15T16:08:46.229 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-01-15T16:08:46.231 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.264 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.060) 0:04:07.947 ******** 2021-01-15T16:08:46.291 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.325 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.061) 0:04:08.008 ******** 2021-01-15T16:08:46.351 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-01-15T16:08:46.353 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.385 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.060) 0:04:08.068 ******** 2021-01-15T16:08:46.411 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-01-15T16:08:46.413 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.445 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:08.127 ******** 2021-01-15T16:08:46.471 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-01-15T16:08:46.472 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.504 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:08.187 ******** 2021-01-15T16:08:46.531 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-01-15T16:08:46.533 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.564 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:08.247 ******** 2021-01-15T16:08:46.600 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-01-15T16:08:46.602 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.636 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.072) 0:04:08.319 ******** 2021-01-15T16:08:46.662 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-01-15T16:08:46.664 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.697 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.060) 0:04:08.380 ******** 2021-01-15T16:08:46.723 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-01-15T16:08:46.725 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.758 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.061) 0:04:08.441 ******** 2021-01-15T16:08:46.784 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-01-15T16:08:46.786 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.817 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:08.500 ******** 2021-01-15T16:08:46.844 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-01-15T16:08:46.845 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.877 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.059) 0:04:08.560 ******** 2021-01-15T16:08:46.904 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-01-15T16:08:46.938 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:46 +0000 (0:00:00.061) 0:04:08.621 ******** 2021-01-15T16:08:46.975 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-01-15T16:08:46.976 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:46.987 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-01-15T16:08:46.999 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.010 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.027 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.061 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.123) 0:04:08.744 ******** 2021-01-15T16:08:47.250 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi145.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.002898", "end": "2021-01-15 16:08:47.227906", "rc": 0, "start": "2021-01-15 16:08:47.225008", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2021-01-15T16:08:47.287 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.225) 0:04:08.970 ******** 2021-01-15T16:08:47.313 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.357 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.070) 0:04:09.040 ******** 2021-01-15T16:08:47.394 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.426 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.069) 0:04:09.109 ******** 2021-01-15T16:08:47.454 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.488 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.060) 0:04:09.170 ******** 2021-01-15T16:08:47.514 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:47.547 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:47 +0000 (0:00:00.060) 0:04:09.230 ******** 2021-01-15T16:08:52.687 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-01-15T16:08:52.689 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper", "Installed: device-mapper-event-8:1.02.171-5.el8.x86_64", "Installed: lvm2-8:2.03.09-5.el8.x86_64", "Installed: device-mapper-event-libs-8:1.02.171-5.el8.x86_64", "Installed: lvm2-libs-8:2.03.09-5.el8.x86_64", "Installed: device-mapper-libs-8:1.02.171-5.el8.x86_64", "Installed: device-mapper-8:1.02.171-5.el8.x86_64", "Removed: device-mapper-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-event-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-event-libs-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-libs-8:1.02.169-3.el8.x86_64", "Removed: lvm2-8:2.03.08-3.el8.x86_64", "Removed: lvm2-libs-8:2.03.08-3.el8.x86_64"]} 2021-01-15T16:08:52.730 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:52 +0000 (0:00:05.183) 0:04:14.413 ******** 2021-01-15T16:08:52.793 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-01-15T16:08:52.795 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356469760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137810, 'block_used': 12957298, 'inode_total': 61054976, 'inode_available': 60986353, 'inode_used': 68623, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137810, "block_size": 4096, "block_total": 240095108, "block_used": 12957298, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986353, "inode_total": 61054976, "inode_used": 68623, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356469760, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}} 2021-01-15T16:08:52.837 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:52 +0000 (0:00:00.106) 0:04:14.520 ******** 2021-01-15T16:08:53.039 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-01-15T16:08:53.041 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005401", "end": "2021-01-15 16:08:53.016840", "rc": 0, "start": "2021-01-15 16:08:53.011439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:08:53.078 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:53 +0000 (0:00:00.241) 0:04:14.761 ******** 2021-01-15T16:08:53.139 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-01-15T16:08:53.140 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356469760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137810, 'block_used': 12957298, 'inode_total': 61054976, 'inode_available': 60986353, 'inode_used': 68623, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137810, "block_size": 4096, "block_total": 240095108, "block_used": 12957298, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986353, "inode_total": 61054976, "inode_used": 68623, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356469760, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:08:53.173 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:53 +0000 (0:00:00.094) 0:04:14.856 ******** 2021-01-15T16:08:53.213 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-01-15T16:08:53.273 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:08:53 +0000 (0:00:00.094) 0:04:14.950 ******** 2021-01-15T16:08:55.017 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-01-15T16:08:55.019 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxilypJ949NaT59LKvBs6aTWvlFYiznnJ8k'], 'uuids': ['37f01dd6-a642-4400-bfad-fbccec1dcfe0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.512444", "end": "2021-01-15 16:08:54.990241", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxilypJ949NaT59LKvBs6aTWvlFYiznnJ8k"], "labels": [], "masters": [], "uuids": ["37f01dd6-a642-4400-bfad-fbccec1dcfe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:08:53.477797", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:08:56.277 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-e0CDIT-hpBz-ARJZ-tGwk-0UbG-eoYy-fhCEUa', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', 'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.069229", "end": "2021-01-15 16:08:56.245103", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-e0CDIT-hpBz-ARJZ-tGwk-0UbG-eoYy-fhCEUa", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:08:55.175874", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-01-15T16:08:56.279 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:08:57.529 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiqxfsEOfv9uKEV271aG5LpOarUiq1FqKd'], 'uuids': ['3f79ffcf-c684-4ba2-aba1-d0c0c787d04a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.058245", "end": "2021-01-15 16:08:57.497667", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiqxfsEOfv9uKEV271aG5LpOarUiq1FqKd"], "labels": [], "masters": [], "uuids": ["3f79ffcf-c684-4ba2-aba1-d0c0c787d04a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:08:56.439422", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:08:58.773 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi0DS9nEBRfZYpsoBh9tKtQBUGBtF3GYBe'], 'uuids': ['2739f69a-7cb5-4d89-8b2e-cae1490ca003'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.056261", "end": "2021-01-15 16:08:58.739989", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi0DS9nEBRfZYpsoBh9tKtQBUGBtF3GYBe"], "labels": [], "masters": [], "uuids": ["2739f69a-7cb5-4d89-8b2e-cae1490ca003"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:08:57.683728", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:08:59.997 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi3239KLOvXFKX3ENu0do6JRqdwbAvuNAl'], 'uuids': ['7eb72d29-54f1-421e-80ff-e4401135fc2a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050134", "end": "2021-01-15 16:08:59.973082", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi3239KLOvXFKX3ENu0do6JRqdwbAvuNAl"], "labels": [], "masters": [], "uuids": ["7eb72d29-54f1-421e-80ff-e4401135fc2a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:08:58.922948", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:09:00.020 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLDP', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP', 'scsi-35000c5009293fa4d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP', 'wwn-0x5000c5009293fa4d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XLDP', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293fa4d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP-part1', 'scsi-35000c5009293fa4d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP-part1', 'wwn-0x5000c5009293fa4d-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP", "scsi-35000c5009293fa4d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-01-15T16:09:00.022 INFO:teuthology.task.ansible.out:Z1W5XLDP-part1", "scsi-35000c5009293fa4d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:01.266 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiPinauw8kgB7spfo30TK7JAZsCYLwKvUb'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.059221", "end": "2021-01-15 16:09:01.232579", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiPinauw8kgB7spfo30TK7JAZsCYLwKvUb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:00.173358", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-01-15T16:09:01.312 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:01 +0000 (0:00:08.044) 0:04:22.995 ******** 2021-01-15T16:09:01.676 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-01-15T16:09:01.678 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxilypJ949NaT59LKvBs6aTWvlFYiznnJ8k'], 'uuids': ['37f01dd6-a642-4400-bfad-fbccec1dcfe0'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.139542", "end": "2021-01-15 16:09:01.647074", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxilypJ949NaT59LKvBs6aTWvlFYiznnJ8k"], "labels": [], "masters": [], "uuids": ["37f01dd6-a642-4400-bfad-fbccec1dcfe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:01.507532", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.136877 s, 843 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.136877 s, 843 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:01.899 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-e0CDIT-hpBz-ARJZ-tGwk-0UbG-eoYy-fhCEUa', 'nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN', 'nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037808", "end": "2021-01-15 16:09:01.869847", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-e0CDIT-hpBz-ARJZ-tGwk-0UbG-eoYy-fhCEUa", "nvme-INTEL_SSDPEDMD400G4_PHFT620500N9400BGN", "nvme-nvme.8086-504846543632303530304e3934303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:01.832039", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0350973 s, 3.3 GB/s", "std 2021-01-15T16:09:01.901 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0350973 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:02.237 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiqxfsEOfv9uKEV271aG5LpOarUiq1FqKd'], 'uuids': ['3f79ffcf-c684-4ba2-aba1-d0c0c787d04a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.150347", "end": "2021-01-15 16:09:02.204064", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiqxfsEOfv9uKEV271aG5LpOarUiq1FqKd"], "labels": [], "masters": [], "uuids": ["3f79ffcf-c684-4ba2-aba1-d0c0c787d04a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:02.053717", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147709 s, 781 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147709 s, 781 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:02.552 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi0DS9nEBRfZYpsoBh9tKtQBUGBtF3GYBe'], 'uuids': ['2739f69a-7cb5-4d89-8b2e-cae1490ca003'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142381", "end": "2021-01-15 16:09:02.523880", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi0DS9nEBRfZYpsoBh9tKtQBUGBtF3GYBe"], "labels": [], "masters": [], "uuids": ["2739f69a-7cb5-4d89-8b2e-cae1490ca003"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:02.381499", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139765 s, 825 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139765 s, 825 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:02.876 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi3239KLOvXFKX3ENu0do6JRqdwbAvuNAl'], 'uuids': ['7eb72d29-54f1-421e-80ff-e4401135fc2a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.141224", "end": "2021-01-15 16:09:02.844305", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxi3239KLOvXFKX3ENu0do6JRqdwbAvuNAl"], "labels": [], "masters": [], "uuids": ["7eb72d29-54f1-421e-80ff-e4401135fc2a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:02.703081", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138649 s, 832 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138649 s, 832 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:02.878 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLDP', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP', 'scsi-35000c5009293fa4d', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP', 'wwn-0x5000c5009293fa4d'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5XLDP', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c5009293fa4d', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP-part1', 'scsi-35000c5009293fa4d-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP-part1', 'wwn-0x5000c5009293fa4d-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XLDP", "scsi-35000c5009293fa4d", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP", "wwn-0x5000c5009293fa4d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XLDP-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5XLDP-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-01-15T16:09:02.880 INFO:teuthology.task.ansible.out:Z1W5XLDP-part1", "scsi-35000c5009293fa4d-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5XLDP-part1", "wwn-0x5000c5009293fa4d-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XLDP", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009293fa4d"}}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:03.196 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiPinauw8kgB7spfo30TK7JAZsCYLwKvUb'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140382", "end": "2021-01-15 16:09:03.174857", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TQSaVAidmesBhXRjKhmixoX81MDqdUxiPinauw8kgB7spfo30TK7JAZsCYLwKvUb"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-01-15 16:09:03.034475", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137847 s, 837 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137847 s, 837 MB/s"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:03.267 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:03 +0000 (0:00:01.954) 0:04:24.949 ******** 2021-01-15T16:09:03.524 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi145.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.062216", "end": "2021-01-15 16:09:03.495734", "rc": 0, "start": "2021-01-15 16:09:03.433518", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:03.577 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:03 +0000 (0:00:00.310) 0:04:25.259 ******** 2021-01-15T16:09:03.808 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047756", "end": "2021-01-15 16:09:03.787442", "rc": 0, "start": "2021-01-15 16:09:03.739686", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:03.844 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:03 +0000 (0:00:00.267) 0:04:25.527 ******** 2021-01-15T16:09:03.872 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-01-15T16:09:03.923 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:03 +0000 (0:00:00.078) 0:04:25.605 ******** 2021-01-15T16:09:03.951 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:03.993 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:03 +0000 (0:00:00.070) 0:04:25.676 ******** 2021-01-15T16:09:04.021 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.060 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.067) 0:04:25.743 ******** 2021-01-15T16:09:04.087 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-01-15T16:09:04.089 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.126 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.065) 0:04:25.808 ******** 2021-01-15T16:09:04.152 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-01-15T16:09:04.154 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.185 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.060) 0:04:25.868 ******** 2021-01-15T16:09:04.372 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-01-15T16:09:04.374 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-01-15T16:09:04.410 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.224) 0:04:26.093 ******** 2021-01-15T16:09:04.447 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-01-15T16:09:04.449 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356469760, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137810, 'block_used': 12957298, 'inode_total': 61054976, 'inode_available': 60986353, 'inode_used': 68623, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137810, "block_size": 4096, "block_total": 240095108, "block_used": 12957298, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986353, "inode_total": 61054976, "inode_used": 68623, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356469760, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.484 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.072) 0:04:26.165 ******** 2021-01-15T16:09:04.510 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-01-15T16:09:04.512 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.545 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.062) 0:04:26.228 ******** 2021-01-15T16:09:04.583 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.621 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.076) 0:04:26.304 ******** 2021-01-15T16:09:04.649 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T16:09:04.651 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.682 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.060) 0:04:26.365 ******** 2021-01-15T16:09:04.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-01-15T16:09:04.710 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.746 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.061) 0:04:26.427 ******** 2021-01-15T16:09:04.771 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-01-15T16:09:04.773 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.804 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.060) 0:04:26.487 ******** 2021-01-15T16:09:04.842 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-01-15T16:09:04.844 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:04.877 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:04 +0000 (0:00:00.072) 0:04:26.560 ******** 2021-01-15T16:09:05.568 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-01-15T16:09:05.570 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-01-15T16:09:05.604 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:05 +0000 (0:00:00.726) 0:04:27.286 ******** 2021-01-15T16:09:06.226 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-01-15T16:09:06.229 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T16:09:06.566 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T16:09:06.891 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T16:09:07.223 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T16:09:07.558 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-01-15T16:09:07.592 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:07 +0000 (0:00:01.988) 0:04:29.275 ******** 2021-01-15T16:09:07.785 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-01-15T16:09:07.787 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-01-15T16:09:07.821 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:07 +0000 (0:00:00.228) 0:04:29.504 ******** 2021-01-15T16:09:08.039 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T16:09:08.205 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T16:09:08.382 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T16:09:08.560 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-01-15T16:09:08.571 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:08.607 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:08 +0000 (0:00:00.786) 0:04:30.290 ******** 2021-01-15T16:09:08.634 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:08.667 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:08 +0000 (0:00:00.059) 0:04:30.350 ******** 2021-01-15T16:09:08.693 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-01-15T16:09:08.695 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:08.728 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:08 +0000 (0:00:00.061) 0:04:30.411 ******** 2021-01-15T16:09:08.922 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-01-15T16:09:08.924 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T16:09:08.957 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:08 +0000 (0:00:00.228) 0:04:30.640 ******** 2021-01-15T16:09:08.994 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-01-15T16:09:08.996 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-01-15T16:09:09.029 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:09 +0000 (0:00:00.072) 0:04:30.712 ******** 2021-01-15T16:09:10.559 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-01-15T16:09:10.561 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true} 2021-01-15T16:09:10.593 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:10 +0000 (0:00:01.563) 0:04:32.276 ******** 2021-01-15T16:09:11.149 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-01-15T16:09:11.151 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-01-15T16:09:11.183 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.589) 0:04:32.866 ******** 2021-01-15T16:09:11.209 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-01-15T16:09:11.211 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.244 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.060) 0:04:32.927 ******** 2021-01-15T16:09:11.274 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.306 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.061) 0:04:32.989 ******** 2021-01-15T16:09:11.332 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-01-15T16:09:11.334 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.367 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.060) 0:04:33.049 ******** 2021-01-15T16:09:11.392 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-01-15T16:09:11.394 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.439 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.071) 0:04:33.121 ******** 2021-01-15T16:09:11.467 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-01-15T16:09:11.468 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.500 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.061) 0:04:33.183 ******** 2021-01-15T16:09:11.526 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.558 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.057) 0:04:33.241 ******** 2021-01-15T16:09:11.597 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-01-15T16:09:11.598 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.631 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.072) 0:04:33.314 ******** 2021-01-15T16:09:11.657 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.690 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.059) 0:04:33.373 ******** 2021-01-15T16:09:11.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-01-15T16:09:11.718 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.753 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.062) 0:04:33.435 ******** 2021-01-15T16:09:11.779 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-01-15T16:09:11.781 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.812 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.059) 0:04:33.495 ******** 2021-01-15T16:09:11.839 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-01-15T16:09:11.840 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.872 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.059) 0:04:33.555 ******** 2021-01-15T16:09:11.899 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-01-15T16:09:11.900 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.931 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.059) 0:04:33.614 ******** 2021-01-15T16:09:11.957 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-01-15T16:09:11.959 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:11.991 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:11 +0000 (0:00:00.059) 0:04:33.673 ******** 2021-01-15T16:09:12.017 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-01-15T16:09:12.019 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:12.050 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:12 +0000 (0:00:00.059) 0:04:33.733 ******** 2021-01-15T16:09:12.086 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-01-15T16:09:12.088 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:12.119 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:12 +0000 (0:00:00.069) 0:04:33.802 ******** 2021-01-15T16:09:12.579 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-01-15T16:09:12.581 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-01-15T16:09:12.618 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:12 +0000 (0:00:00.498) 0:04:34.301 ******** 2021-01-15T16:09:12.918 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-01-15T16:09:12.919 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-01-15 16:03:59 UTC", "ActiveEnterTimestampMonotonic": "13303379", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount ntpd.service basic.target sysinit.target systemd-journald.socket ntpdate.service sntp.service systemd-tmpfiles-setup.service tmp.mount system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-01-15 16:03:58 UTC", "AssertTimestampMonotonic": "11920325", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-01-15 16:03:58 UTC", "ConditionTimestampMonotonic": "11920296", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0 2021-01-15T16:09:12.920 INFO:teuthology.task.ansible.out:", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "874", "ExecMainStartTimestamp": "Fri 2021-01-15 16:03:59 UTC", "ExecMainStartTimestampMonotonic": "13182731", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-01-15 16:03:58 UTC", "InactiveExitTimestampMonotonic": "11931172", "InvocationID": "f51824ac8789467d8db3b32422c4547d", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity 2021-01-15T16:09:12.922 INFO:teuthology.task.ansible.out:", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127261", "LimitNPROCSoft": "127261", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127261", "LimitSIGPENDINGSoft": "127261", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "874", "MemoryAccounting": "yes", "MemoryCurrent": "3301376", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOn 2021-01-15T16:09:12.923 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-01-15 16:03:59 UTC", "StateChangeTimestampMonotonic": "13303379", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203618", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-01-15 16:03:59 UTC", "WatchdogTimestampMonotonic": "13182755", "WatchdogUSec": "0"}} 2021-01-15T16:09:12.954 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:12 +0000 (0:00:00.336) 0:04:34.637 ******** 2021-01-15T16:09:12.980 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T16:09:12.982 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-01-15T16:09:13.015 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.060) 0:04:34.697 ******** 2021-01-15T16:09:13.042 INFO:teuthology.task.ansible.out: 2021-01-15T16:09:13.044 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-01-15T16:09:13.074 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.059) 0:04:34.757 ******** 2021-01-15T16:09:13.101 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-01-15T16:09:13.103 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:13.136 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.061) 0:04:34.818 ******** 2021-01-15T16:09:13.162 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:13.199 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.063) 0:04:34.882 ******** 2021-01-15T16:09:13.225 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-01-15T16:09:13.227 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:13.258 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.059) 0:04:34.941 ******** 2021-01-15T16:09:13.456 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-01-15T16:09:13.490 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.231) 0:04:35.173 ******** 2021-01-15T16:09:13.950 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-01-15T16:09:13.952 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-01-15T16:09:13.986 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:13 +0000 (0:00:00.495) 0:04:35.669 ******** 2021-01-15T16:09:15.307 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-01-15T16:09:15.312 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-01-15T16:09:15.346 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:01.360) 0:04:37.029 ******** 2021-01-15T16:09:15.373 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-01-15T16:09:15.375 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:15.410 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.063) 0:04:37.093 ******** 2021-01-15T16:09:15.437 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-01-15T16:09:15.438 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-01-15T16:09:15.472 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.061) 0:04:37.155 ******** 2021-01-15T16:09:15.666 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-01-15T16:09:15.668 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-01-15T16:09:15.702 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.230) 0:04:37.385 ******** 2021-01-15T16:09:15.740 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:15.781 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.079) 0:04:37.464 ******** 2021-01-15T16:09:15.810 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-01-15T16:09:15.812 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-01-15T16:09:15.848 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.067) 0:04:37.531 ******** 2021-01-15T16:09:15.875 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-01-15T16:09:15.877 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:15.916 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:15 +0000 (0:00:00.067) 0:04:37.599 ******** 2021-01-15T16:09:16.103 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-01-15T16:09:16.105 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-01-15T16:09:16.151 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.234) 0:04:37.834 ******** 2021-01-15T16:09:16.347 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-01-15T16:09:16.349 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-01-15T16:09:16.383 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.231) 0:04:38.065 ******** 2021-01-15T16:09:16.409 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-01-15T16:09:16.411 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.444 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.060) 0:04:38.125 ******** 2021-01-15T16:09:16.479 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-01-15T16:09:16.481 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.492 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.525 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.082) 0:04:38.208 ******** 2021-01-15T16:09:16.573 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-01-15T16:09:16.575 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi145.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.577 INFO:teuthology.task.ansible.out: 2021-01-15T16:09:16.585 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.596 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.598 INFO:teuthology.task.ansible.out: skipping: [smithi145.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.631 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.106) 0:04:38.314 ******** 2021-01-15T16:09:16.659 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-01-15T16:09:16.660 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.707 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.075) 0:04:38.390 ******** 2021-01-15T16:09:16.735 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-01-15T16:09:16.737 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.768 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.060) 0:04:38.451 ******** 2021-01-15T16:09:16.795 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-01-15T16:09:16.797 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.840 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.071) 0:04:38.523 ******** 2021-01-15T16:09:16.871 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.920 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.080) 0:04:38.603 ******** 2021-01-15T16:09:16.948 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:16.985 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:16 +0000 (0:00:00.064) 0:04:38.668 ******** 2021-01-15T16:09:17.022 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:17.058 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:17 +0000 (0:00:00.072) 0:04:38.740 ******** 2021-01-15T16:09:17.084 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-01-15T16:09:17.086 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:17.120 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:17 +0000 (0:00:00.062) 0:04:38.803 ******** 2021-01-15T16:09:17.146 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:17.196 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:17 +0000 (0:00:00.076) 0:04:38.879 ******** 2021-01-15T16:09:17.413 INFO:teuthology.task.ansible.out: 2021-01-15T16:09:17.414 INFO:teuthology.task.ansible.out:TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-01-15T16:09:17.415 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004920", "end": "2021-01-15 16:09:17.378019", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-01-15 16:09:17.373099", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:17.596 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003103", "end": "2021-01-15 16:09:17.557664", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-01-15 16:09:17.554561", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:17.782 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002876", "end": "2021-01-15 16:09:17.736735", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-01-15 16:09:17.733859", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:17.952 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003082", "end": "2021-01-15 16:09:17.920606", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-01-15 16:09:17.917524", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:17.988 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:17 +0000 (0:00:00.791) 0:04:39.671 ******** 2021-01-15T16:09:18.015 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-01-15T16:09:18.050 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:18 +0000 (0:00:00.062) 0:04:39.733 ******** 2021-01-15T16:09:18.233 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-01-15T16:09:18.332 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-01-15T16:09:18.365 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:18 +0000 (0:00:00.314) 0:04:40.048 ******** 2021-01-15T16:09:19.011 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi145.front.sepia.ceph.com] 2021-01-15T16:09:19.085 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:19 +0000 (0:00:00.720) 0:04:40.768 ******** 2021-01-15T16:09:19.111 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T16:09:19.112 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:19.148 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:19 +0000 (0:00:00.062) 0:04:40.831 ******** 2021-01-15T16:09:19.174 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T16:09:19.176 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-01-15T16:09:19.210 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:19 +0000 (0:00:00.061) 0:04:40.893 ******** 2021-01-15T16:09:19.246 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-01-15T16:09:19.248 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-01-15T16:09:19.281 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:19 +0000 (0:00:00.071) 0:04:40.964 ******** 2021-01-15T16:09:20.295 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-01-15T16:09:20.297 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: podman", "Installed: podman-docker"]} 2021-01-15T16:09:20.331 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:20 +0000 (0:00:01.049) 0:04:42.013 ******** 2021-01-15T16:09:20.357 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T16:09:20.358 INFO:teuthology.task.ansible.out:ok: [smithi145.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-01-15T16:09:20.392 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:20 +0000 (0:00:00.060) 0:04:42.074 ******** 2021-01-15T16:09:20.421 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-01-15T16:09:20.423 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:20.455 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:20 +0000 (0:00:00.063) 0:04:42.138 ******** 2021-01-15T16:09:20.796 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [smithi145.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T16:09:20.984 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-01-15T16:09:21.021 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:21 +0000 (0:00:00.566) 0:04:42.704 ******** 2021-01-15T16:09:21.654 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-01-15T16:09:21.655 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "ad23d23287af05eb6dfc6847a1957988954f5ccf", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "97bf7a9f95b52e04e40f96163cd1717c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2134, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610726961.0583394-255462242709018/source", "state": "file", "uid": 0} 2021-01-15T16:09:22.168 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "ad23d23287af05eb6dfc6847a1957988954f5ccf", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "97bf7a9f95b52e04e40f96163cd1717c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2134, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1610726961.6827257-259108506155775/source", "state": "file", "uid": 0} 2021-01-15T16:09:22.203 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:22 +0000 (0:00:01.181) 0:04:43.886 ******** 2021-01-15T16:09:25.292 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-01-15T16:09:25.293 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-70jns7b0-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-70jns7b0-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-01-15T16:09:25.295 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-01-15T16:09:25.327 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:25 +0000 (0:00:03.123) 0:04:47.009 ******** 2021-01-15T16:09:25.353 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:25.394 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:25 +0000 (0:00:00.067) 0:04:47.077 ******** 2021-01-15T16:09:25.432 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-01-15T16:09:25.434 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:25.476 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:25 +0000 (0:00:00.081) 0:04:47.159 ******** 2021-01-15T16:09:25.846 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-01-15T16:09:25.848 INFO:teuthology.task.ansible.out:changed: [smithi145.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.172257", "end": "2021-01-15 16:09:25.817603", "rc": 0, "start": "2021-01-15 16:09:25.645346", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-01-15T16:09:25.884 INFO:teuthology.task.ansible.out:Friday 15 January 2021 16:09:25 +0000 (0:00:00.408) 0:04:47.567 ******** 2021-01-15T16:09:25.911 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-01-15T16:09:25.912 INFO:teuthology.task.ansible.out:skipping: [smithi145.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-01-15T16:09:26.009 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-01-15T16:09:26.013 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-01-15T16:09:26.018 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* 2021-01-15T16:09:26.020 INFO:teuthology.task.ansible.out:smithi145.front.sepia.ceph.com : ok=126 changed=46 unreachable=0 failed=0 skipped=183 rescued=0 ignored=3 Friday 15 January 2021 16:09:26 +0000 (0:00:00.133) 0:04:47.701 ******** =============================================================================== testnode : Install packages -------------------------------------------- 42.12s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.22s users : Create all admin users with sudo access. ----------------------- 30.53s users : Update authorized_keys using the keys repo --------------------- 29.02s common : nrpe - Load SELinux policy package ---------------------------- 16.76s common : nrpe - Remove SELinux policy package -------------------------- 16.51s users : Remove revoked users ------------------------------------------- 10.15s testnode : Zap all non-root disks --------------------------------------- 8.04s common : Install Kerberos Packages (RedHat) ----------------------------- 7.01s testnode : Make sure rpm dependencies are installed --------------------- 5.18s container-host : Install registries-conf-ctl ---------------------------- 3.12s common : Make sure smartmontools is installed --------------------------- 2.95s common : nrpe - Install SELinux dependencies ---------------------------- 2.81s testnode : Ensure ceph packages are not present. ------------------------ 2.32s testnode : Rebuild rpmdb ------------------------------------------------ 2.21s testnode : Ensure ceph dependency packages are not present. ------------- 2.18s testnode : Create logical volume(s) ------------------------------------- 1.99s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.95s users : Update authorized_keys for each user with literal keys ---------- 1.75s testnode : Create xfs filesystem on /dev/vg_nvme/lv_5 ------------------- 1.56s 2021-01-15T16:09:26.304 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2021-01-15T16:09:26.305 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-01-15T16:09:26.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-01-15T16:09:26.380 DEBUG:teuthology.orchestra.run.smithi145:> true 2021-01-15T16:09:26.470 INFO:teuthology.run_tasks:Running task clock... 2021-01-15T16:09:26.481 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-01-15T16:09:26.481 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-01-15T16:09:26.482 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-01-15T16:09:26.542 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-01-15T16:09:26.557 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-01-15T16:09:26.595 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2021-01-15T16:09:26.606 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2021-01-15T16:09:26.621 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-01-15T16:09:26.635 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-01-15T16:09:26.693 INFO:teuthology.orchestra.run.smithi145.stderr:bash: ntpq: command not found 2021-01-15T16:09:26.695 INFO:teuthology.orchestra.run.smithi145.stdout:210 Number of sources = 4 2021-01-15T16:09:26.696 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-01-15T16:09:26.696 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2021-01-15T16:09:26.696 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T16:09:26.696 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T16:09:26.697 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T16:09:26.697 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-01-15T16:09:26.697 INFO:teuthology.run_tasks:Running task install... 2021-01-15T16:09:26.707 DEBUG:teuthology.task.install:project ceph 2021-01-15T16:09:26.708 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'}} 2021-01-15T16:09:26.708 DEBUG:teuthology.task.install:config {'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'} 2021-01-15T16:09:26.709 INFO:teuthology.task.install:Using flavor: basic 2021-01-15T16:09:26.716 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-01-15T16:09:26.717 INFO:teuthology.task.install:extra packages: [] 2021-01-15T16:09:26.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2021-01-15T16:09:26.769 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2021-01-15T16:09:26.770 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-15T16:09:26.771 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '892c51dd3c5f7108e766bea30cd5e0d801b0abd3', 'tag': None, 'wait_for_package': False} 2021-01-15T16:09:26.771 DEBUG:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2021-01-15T16:09:26.843 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2021-01-15T16:09:26.844 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-15T16:09:26.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T16:09:27.016 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/ 2021-01-15T16:09:27.016 INFO:teuthology.task.install.rpm:Package version is 16.0.0-9031.g892c51dd 2021-01-15T16:09:27.149 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/master/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-01-15T16:09:27.150 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:09:27.151 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-01-15T16:09:27.178 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-01-15T16:09:27.179 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/892c51dd3c5f7108e766bea30cd5e0d801b0abd3/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-01-15T16:09:27.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-01-15T16:09:27.319 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-01-15T16:09:27.383 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2021-01-15T16:09:27.384 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2021-01-15T16:09:27.729 INFO:teuthology.orchestra.run.smithi145.stdout:107 files removed 2021-01-15T16:09:27.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw 2021-01-15T16:09:28.666 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 135 kB/s | 73 kB 00:00 2021-01-15T16:09:28.907 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 73 kB/s | 15 kB 00:00 2021-01-15T16:09:29.141 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 6.8 kB/s | 1.4 kB 00:00 2021-01-15T16:09:29.311 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - AppStream 42 MB/s | 6.3 MB 00:00 2021-01-15T16:09:30.394 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Base 23 MB/s | 2.3 MB 00:00 2021-01-15T16:09:30.837 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-8 - Extras 133 kB/s | 8.6 kB 00:00 2021-01-15T16:09:31.200 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 28 MB/s | 8.8 MB 00:00 2021-01-15T16:09:33.073 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 3.6 MB/s | 23 kB 00:00 2021-01-15T16:09:34.223 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:09:34.227 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:09:34.228 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:09:34.228 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:09:34.228 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:09:34.229 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 12 M 2021-01-15T16:09:34.230 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2021-01-15T16:09:34.231 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.7 M 2021-01-15T16:09:34.231 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.2 M 2021-01-15T16:09:34.231 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:09:34.232 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 6.0 M 2021-01-15T16:09:34.232 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 24 M 2021-01-15T16:09:34.233 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 25 k 2021-01-15T16:09:34.233 INFO:teuthology.orchestra.run.smithi145.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-01-15T16:09:34.235 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2021-01-15T16:09:34.235 INFO:teuthology.orchestra.run.smithi145.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-01-15T16:09:34.236 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 793 k 2021-01-15T16:09:34.237 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-01-15T16:09:34.237 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-01-15T16:09:34.238 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 486 k 2021-01-15T16:09:34.238 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-01-15T16:09:34.239 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 4.0 M 2021-01-15T16:09:34.240 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-01-15T16:09:34.240 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-01-15T16:09:34.241 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse 2021-01-15T16:09:34.241 INFO:teuthology.orchestra.run.smithi145.stdout: x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 40 k 2021-01-15T16:09:34.241 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 77 k 2021-01-15T16:09:34.241 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 186 k 2021-01-15T16:09:34.242 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-01-15T16:09:34.242 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 387 k 2021-01-15T16:09:34.242 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 368 k 2021-01-15T16:09:34.242 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 114 k 2021-01-15T16:09:34.243 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:09:34.243 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:09:34.243 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:09:34.243 INFO:teuthology.orchestra.run.smithi145.stdout:Install 22 Packages 2021-01-15T16:09:34.243 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 2 Packages 2021-01-15T16:09:34.244 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:09:34.244 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 57 M 2021-01-15T16:09:34.245 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:09:36.881 INFO:teuthology.orchestra.run.smithi145.stdout:(1/24): ceph-base-16.0.0-9031.g892c51dd.el8.x86 2.3 MB/s | 6.0 MB 00:02 2021-01-15T16:09:36.972 INFO:teuthology.orchestra.run.smithi145.stdout:(2/24): ceph-selinux-16.0.0-9031.g892c51dd.el8. 271 kB/s | 25 kB 00:00 2021-01-15T16:09:37.249 INFO:teuthology.orchestra.run.smithi145.stdout:(3/24): libcephfs2-16.0.0-9031.g892c51dd.el8.x8 2.8 MB/s | 793 kB 00:00 2021-01-15T16:09:37.424 INFO:teuthology.orchestra.run.smithi145.stdout:(4/24): libradosstriper1-16.0.0-9031.g892c51dd. 2.7 MB/s | 486 kB 00:00 2021-01-15T16:09:37.525 INFO:teuthology.orchestra.run.smithi145.stdout:(5/24): ceph-radosgw-16.0.0-9031.g892c51dd.el8. 3.7 MB/s | 12 MB 00:03 2021-01-15T16:09:37.609 INFO:teuthology.orchestra.run.smithi145.stdout:(6/24): python3-ceph-argparse-16.0.0-9031.g892c 482 kB/s | 40 kB 00:00 2021-01-15T16:09:37.693 INFO:teuthology.orchestra.run.smithi145.stdout:(7/24): python3-ceph-common-16.0.0-9031.g892c51 930 kB/s | 77 kB 00:00 2021-01-15T16:09:37.784 INFO:teuthology.orchestra.run.smithi145.stdout:(8/24): python3-cephfs-16.0.0-9031.g892c51dd.el 2.0 MB/s | 186 kB 00:00 2021-01-15T16:09:37.880 INFO:teuthology.orchestra.run.smithi145.stdout:(9/24): python3-rados-16.0.0-9031.g892c51dd.el8 3.9 MB/s | 387 kB 00:00 2021-01-15T16:09:37.958 INFO:teuthology.orchestra.run.smithi145.stdout:(10/24): python3-rbd-16.0.0-9031.g892c51dd.el8. 4.6 MB/s | 368 kB 00:00 2021-01-15T16:09:38.039 INFO:teuthology.orchestra.run.smithi145.stdout:(11/24): python3-rgw-16.0.0-9031.g892c51dd.el8. 1.4 MB/s | 114 kB 00:00 2021-01-15T16:09:38.080 INFO:teuthology.orchestra.run.smithi145.stdout:(12/24): librgw2-16.0.0-9031.g892c51dd.el8.x86_ 6.1 MB/s | 4.0 MB 00:00 2021-01-15T16:09:38.148 INFO:teuthology.orchestra.run.smithi145.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 3.2 MB/s | 353 kB 00:00 2021-01-15T16:09:38.150 INFO:teuthology.orchestra.run.smithi145.stdout:(14/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.6 MB/s | 259 kB 00:00 2021-01-15T16:09:38.151 INFO:teuthology.orchestra.run.smithi145.stdout:(15/24): python3-prettytable-0.7.2-14.el8.noarc 17 MB/s | 44 kB 00:00 2021-01-15T16:09:38.152 INFO:teuthology.orchestra.run.smithi145.stdout:(16/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 22 MB/s | 47 kB 00:00 2021-01-15T16:09:38.370 INFO:teuthology.orchestra.run.smithi145.stdout:(17/24): ceph-common-16.0.0-9031.g892c51dd.el8. 5.7 MB/s | 24 MB 00:04 2021-01-15T16:09:38.511 INFO:teuthology.orchestra.run.smithi145.stdout:(18/24): fmt-6.2.1-1.el8.x86_64.rpm 323 kB/s | 116 kB 00:00 2021-01-15T16:09:38.515 INFO:teuthology.orchestra.run.smithi145.stdout:(19/24): gperftools-libs-2.7-6.el8.x86_64.rpm 874 kB/s | 315 kB 00:00 2021-01-15T16:09:38.530 INFO:teuthology.orchestra.run.smithi145.stdout:(20/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.3 MB/s | 75 kB 00:00 2021-01-15T16:09:38.531 INFO:teuthology.orchestra.run.smithi145.stdout:(21/24): liboath-2.6.2-3.el8.x86_64.rpm 2.9 MB/s | 59 kB 00:00 2021-01-15T16:09:38.590 INFO:teuthology.orchestra.run.smithi145.stdout:(22/24): leveldb-1.22-1.el8.x86_64.rpm 823 kB/s | 181 kB 00:00 2021-01-15T16:09:38.729 INFO:teuthology.orchestra.run.smithi145.stdout:(23/24): librbd1-16.0.0-9031.g892c51dd.el8.x86_ 16 MB/s | 3.2 MB 00:00 2021-01-15T16:09:38.810 INFO:teuthology.orchestra.run.smithi145.stdout:(24/24): librados2-16.0.0-9031.g892c51dd.el8.x8 13 MB/s | 3.7 MB 00:00 2021-01-15T16:09:38.814 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:09:38.815 INFO:teuthology.orchestra.run.smithi145.stdout:Total 12 MB/s | 57 MB 00:04 2021-01-15T16:09:38.819 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:09:38.862 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:09:38.863 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:09:39.179 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:09:39.182 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:09:40.260 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:09:40.420 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-01-15T16:09:40.614 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-01-15T16:09:40.788 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-01-15T16:09:41.197 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-01-15T16:09:41.405 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 4/26 2021-01-15T16:09:41.493 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 4/26 2021-01-15T16:09:41.687 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 5/26 2021-01-15T16:09:41.856 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 6/26 2021-01-15T16:09:42.200 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 6/26 2021-01-15T16:09:42.480 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T16:09:42.541 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T16:09:42.696 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-01-15T16:09:42.838 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-01-15T16:09:43.246 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-01-15T16:09:43.363 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T16:09:43.438 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T16:09:43.593 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-01-15T16:09:43.731 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8. 12/26 2021-01-15T16:09:43.883 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 13/26 2021-01-15T16:09:44.047 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 14/26 2021-01-15T16:09:44.223 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 15/26 2021-01-15T16:09:44.371 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 16/26 2021-01-15T16:09:44.423 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 16/26 2021-01-15T16:09:44.618 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-01-15T16:09:44.821 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 18/26 2021-01-15T16:09:44.972 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-01-15T16:09:45.088 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x8 20/26 2021-01-15T16:09:48.617 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T16:09:48.779 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T16:09:49.491 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 21/26 2021-01-15T16:09:49.739 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 22/26 2021-01-15T16:09:49.817 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 22/26 2021-01-15T16:09:49.929 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T16:10:06.633 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T16:10:06.634 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2021-01-15T16:10:06.634 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2021-01-15T16:10:06.634 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2021-01-15T16:10:06.634 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /run 2021-01-15T16:10:06.635 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2021-01-15T16:10:06.635 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2021-01-15T16:10:06.635 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2021-01-15T16:10:06.635 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2021-01-15T16:10:06.635 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /dev 2021-01-15T16:10:06.636 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:07.867 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 24/26 2021-01-15T16:10:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 24/26 2021-01-15T16:10:07.950 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-01-15T16:10:08.152 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-01-15T16:10:08.153 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T16:10:08.899 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T16:10:08.899 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 1/26 2021-01-15T16:10:08.899 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2/26 2021-01-15T16:10:08.900 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 3/26 2021-01-15T16:10:08.900 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 4/26 2021-01-15T16:10:08.900 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 5/26 2021-01-15T16:10:08.900 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_6 6/26 2021-01-15T16:10:08.901 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 7/26 2021-01-15T16:10:08.901 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8. 8/26 2021-01-15T16:10:08.901 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x8 9/26 2021-01-15T16:10:08.901 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 10/26 2021-01-15T16:10:08.902 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 11/26 2021-01-15T16:10:08.902 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 12/26 2021-01-15T16:10:08.902 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 13/26 2021-01-15T16:10:08.903 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-01-15T16:10:08.903 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-01-15T16:10:08.903 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-01-15T16:10:08.903 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-01-15T16:10:08.903 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-01-15T16:10:08.904 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 19/26 2021-01-15T16:10:08.904 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-01-15T16:10:08.904 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-01-15T16:10:08.904 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-01-15T16:10:08.904 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 23/26 2021-01-15T16:10:08.905 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-01-15T16:10:08.905 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 25/26 2021-01-15T16:10:09.737 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-01-15T16:10:09.738 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:09.738 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2021-01-15T16:10:09.738 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.738 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.739 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:09.739 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:09.739 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.739 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.740 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.740 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.740 INFO:teuthology.orchestra.run.smithi145.stdout: fmt-6.2.1-1.el8.x86_64 2021-01-15T16:10:09.740 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.7-6.el8.x86_64 2021-01-15T16:10:09.740 INFO:teuthology.orchestra.run.smithi145.stdout: leveldb-1.22-1.el8.x86_64 2021-01-15T16:10:09.741 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.741 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.2-3.el8.x86_64 2021-01-15T16:10:09.741 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-01-15T16:10:09.741 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.741 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-01-15T16:10:09.742 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.742 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.3.1-3.el8.x86_64 2021-01-15T16:10:09.742 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-01-15T16:10:09.742 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.742 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.743 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.743 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-01-15T16:10:09.743 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.743 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.743 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:09.744 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:09.744 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:09.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-test 2021-01-15T16:10:10.402 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:37 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:10.850 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:10.851 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:10.851 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:10:10.852 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:10.852 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:10.852 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 49 M 2021-01-15T16:10:10.852 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:10:10.853 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-01-15T16:10:10.853 INFO:teuthology.orchestra.run.smithi145.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-01-15T16:10:10.853 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-01-15T16:10:10.853 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-01-15T16:10:10.853 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-01-15T16:10:10.854 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:10.854 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:10.854 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:10.854 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2021-01-15T16:10:10.855 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:10.855 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 50 M 2021-01-15T16:10:10.855 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 218 M 2021-01-15T16:10:10.856 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:10.889 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.6 MB/s | 161 kB 00:00 2021-01-15T16:10:10.893 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.7 MB/s | 187 kB 00:00 2021-01-15T16:10:10.895 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 49 MB/s | 302 kB 00:00 2021-01-15T16:10:10.898 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 48 MB/s | 250 kB 00:00 2021-01-15T16:10:10.982 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 795 kB/s | 69 kB 00:00 2021-01-15T16:10:13.265 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): ceph-test-16.0.0-9031.g892c51dd.el8.x86_ 21 MB/s | 49 MB 00:02 2021-01-15T16:10:13.268 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:13.269 INFO:teuthology.orchestra.run.smithi145.stdout:Total 21 MB/s | 50 MB 00:02 2021-01-15T16:10:13.273 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:13.298 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:13.299 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:13.558 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:13.560 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:14.022 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:14.209 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-01-15T16:10:14.376 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-01-15T16:10:14.568 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-01-15T16:10:14.707 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-01-15T16:10:14.787 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-01-15T16:10:19.441 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-01-15T16:10:19.721 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 6/6 2021-01-15T16:10:19.953 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 6/6 2021-01-15T16:10:19.954 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 1/6 2021-01-15T16:10:19.954 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-01-15T16:10:19.954 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-01-15T16:10:19.955 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-01-15T16:10:19.955 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-01-15T16:10:20.314 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-01-15T16:10:20.314 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:20.315 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:20.315 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:16.0.0-9031.g892c51dd.el8.x86_64 jq-1.5-12.el8.x86_64 2021-01-15T16:10:20.315 INFO:teuthology.orchestra.run.smithi145.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-01-15T16:10:20.315 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-01-15T16:10:20.316 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:20.316 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:20.385 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph 2021-01-15T16:10:20.835 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:47 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:21.310 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:21.315 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2021-01-15T16:10:21.315 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:10:21.315 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2021-01-15T16:10:21.315 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:21.316 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 6.4 k 2021-01-15T16:10:21.316 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:10:21.316 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 2.4 M 2021-01-15T16:10:21.316 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 1.5 M 2021-01-15T16:10:21.316 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 200 k 2021-01-15T16:10:21.317 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 4.9 M 2021-01-15T16:10:21.317 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 18 M 2021-01-15T16:10:21.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-01-15T16:10:21.317 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-01-15T16:10:21.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-01-15T16:10:21.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.5.1-1.el8 epel 171 k 2021-01-15T16:10:21.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-01-15T16:10:21.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-01-15T16:10:21.318 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-01-15T16:10:21.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-01-15T16:10:21.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-01-15T16:10:21.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-01-15T16:10:21.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-01-15T16:10:21.319 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-01-15T16:10:21.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-01-15T16:10:21.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-01-15T16:10:21.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-01-15T16:10:21.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-01-15T16:10:21.320 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-01-15T16:10:21.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-01-15T16:10:21.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-01-15T16:10:21.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-01-15T16:10:21.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-01-15T16:10:21.321 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-01-15T16:10:21.322 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-01-15T16:10:21.322 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-01-15T16:10:21.322 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-01-15T16:10:21.322 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-01-15T16:10:21.322 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-01-15T16:10:21.323 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-01-15T16:10:21.323 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2021-01-15T16:10:21.323 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-01-15T16:10:21.323 INFO:teuthology.orchestra.run.smithi145.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-01-15T16:10:21.323 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:21.324 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:21.324 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================= 2021-01-15T16:10:21.324 INFO:teuthology.orchestra.run.smithi145.stdout:Install 36 Packages 2021-01-15T16:10:21.324 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:21.325 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 32 M 2021-01-15T16:10:21.325 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 113 M 2021-01-15T16:10:21.325 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:21.611 INFO:teuthology.orchestra.run.smithi145.stdout:(1/36): ceph-16.0.0-9031.g892c51dd.el8.x86_64.r 22 kB/s | 6.4 kB 00:00 2021-01-15T16:10:22.201 INFO:teuthology.orchestra.run.smithi145.stdout:(2/36): ceph-mds-16.0.0-9031.g892c51dd.el8.x86_ 2.7 MB/s | 2.4 MB 00:00 2021-01-15T16:10:22.832 INFO:teuthology.orchestra.run.smithi145.stdout:(3/36): ceph-mon-16.0.0-9031.g892c51dd.el8.x86_ 4.1 MB/s | 4.9 MB 00:01 2021-01-15T16:10:22.902 INFO:teuthology.orchestra.run.smithi145.stdout:(4/36): ceph-mgr-modules-core-16.0.0-9031.g892c 2.8 MB/s | 200 kB 00:00 2021-01-15T16:10:22.938 INFO:teuthology.orchestra.run.smithi145.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.0 MB/s | 214 kB 00:00 2021-01-15T16:10:22.957 INFO:teuthology.orchestra.run.smithi145.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 78 MB/s | 1.5 MB 00:00 2021-01-15T16:10:22.959 INFO:teuthology.orchestra.run.smithi145.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 59 MB/s | 157 kB 00:00 2021-01-15T16:10:22.961 INFO:teuthology.orchestra.run.smithi145.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2021-01-15T16:10:22.962 INFO:teuthology.orchestra.run.smithi145.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2021-01-15T16:10:22.964 INFO:teuthology.orchestra.run.smithi145.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 21 MB/s | 27 kB 00:00 2021-01-15T16:10:22.970 INFO:teuthology.orchestra.run.smithi145.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 72 MB/s | 457 kB 00:00 2021-01-15T16:10:22.973 INFO:teuthology.orchestra.run.smithi145.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 63 MB/s | 195 kB 00:00 2021-01-15T16:10:22.975 INFO:teuthology.orchestra.run.smithi145.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2021-01-15T16:10:22.977 INFO:teuthology.orchestra.run.smithi145.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 57 MB/s | 123 kB 00:00 2021-01-15T16:10:22.980 INFO:teuthology.orchestra.run.smithi145.stdout:(15/36): python3-urllib3-1.24.2-4.el8.noarch.rp 58 MB/s | 176 kB 00:00 2021-01-15T16:10:23.059 INFO:teuthology.orchestra.run.smithi145.stdout:(16/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 560 kB/s | 44 kB 00:00 2021-01-15T16:10:23.117 INFO:teuthology.orchestra.run.smithi145.stdout:(17/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.1 MB/s | 185 kB 00:00 2021-01-15T16:10:23.138 INFO:teuthology.orchestra.run.smithi145.stdout:(18/36): python3-cheroot-8.5.1-1.el8.noarch.rpm 8.2 MB/s | 171 kB 00:00 2021-01-15T16:10:23.178 INFO:teuthology.orchestra.run.smithi145.stdout:(19/36): python3-cherrypy-18.4.0-1.el8.noarch.r 9.4 MB/s | 384 kB 00:00 2021-01-15T16:10:23.202 INFO:teuthology.orchestra.run.smithi145.stdout:(20/36): python3-cssselect-0.9.2-10.el8.noarch. 2.0 MB/s | 40 kB 00:00 2021-01-15T16:10:23.218 INFO:teuthology.orchestra.run.smithi145.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 568 kB/s | 11 kB 00:00 2021-01-15T16:10:23.244 INFO:teuthology.orchestra.run.smithi145.stdout:(22/36): python3-jaraco-functools-2.0-4.el8.noa 891 kB/s | 18 kB 00:00 2021-01-15T16:10:23.257 INFO:teuthology.orchestra.run.smithi145.stdout:(23/36): python3-logutils-0.3.5-11.el8.noarch.r 2.4 MB/s | 49 kB 00:00 2021-01-15T16:10:23.278 INFO:teuthology.orchestra.run.smithi145.stdout:(24/36): python3-more-itertools-7.2.0-3.el8.noa 2.8 MB/s | 59 kB 00:00 2021-01-15T16:10:23.300 INFO:teuthology.orchestra.run.smithi145.stdout:(25/36): ceph-mgr-16.0.0-9031.g892c51dd.el8.x86 793 kB/s | 1.5 MB 00:01 2021-01-15T16:10:23.301 INFO:teuthology.orchestra.run.smithi145.stdout:(26/36): python3-pecan-1.3.2-9.el8.noarch.rpm 13 MB/s | 283 kB 00:00 2021-01-15T16:10:23.321 INFO:teuthology.orchestra.run.smithi145.stdout:(27/36): python3-simplegeneric-0.8.1-17.el8.noa 950 kB/s | 19 kB 00:00 2021-01-15T16:10:23.339 INFO:teuthology.orchestra.run.smithi145.stdout:(28/36): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2021-01-15T16:10:23.347 INFO:teuthology.orchestra.run.smithi145.stdout:(29/36): python3-portend-2.6-1.el8.noarch.rpm 322 kB/s | 16 kB 00:00 2021-01-15T16:10:23.360 INFO:teuthology.orchestra.run.smithi145.stdout:(30/36): python3-tempora-1.14.1-5.el8.noarch.rp 1.4 MB/s | 29 kB 00:00 2021-01-15T16:10:23.364 INFO:teuthology.orchestra.run.smithi145.stdout:(31/36): python3-trustme-0.6.0-4.el8.noarch.rpm 1.6 MB/s | 27 kB 00:00 2021-01-15T16:10:23.383 INFO:teuthology.orchestra.run.smithi145.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 10 MB/s | 241 kB 00:00 2021-01-15T16:10:23.403 INFO:teuthology.orchestra.run.smithi145.stdout:(33/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2021-01-15T16:10:23.412 INFO:teuthology.orchestra.run.smithi145.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 5.0 MB/s | 251 kB 00:00 2021-01-15T16:10:23.426 INFO:teuthology.orchestra.run.smithi145.stdout:(35/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2021-01-15T16:10:24.667 INFO:teuthology.orchestra.run.smithi145.stdout:(36/36): ceph-osd-16.0.0-9031.g892c51dd.el8.x86 7.3 MB/s | 18 MB 00:02 2021-01-15T16:10:24.670 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:24.670 INFO:teuthology.orchestra.run.smithi145.stdout:Total 9.6 MB/s | 32 MB 00:03 2021-01-15T16:10:24.679 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:24.720 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:24.720 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:24.983 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:24.986 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:26.181 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:26.404 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2021-01-15T16:10:26.571 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2021-01-15T16:10:26.794 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2021-01-15T16:10:26.955 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2021-01-15T16:10:27.095 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 5/36 2021-01-15T16:10:27.217 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2021-01-15T16:10:27.382 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2021-01-15T16:10:27.566 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2021-01-15T16:10:27.677 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2021-01-15T16:10:27.847 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2021-01-15T16:10:28.057 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.5.1-1.el8.noarch 11/36 2021-01-15T16:10:28.190 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2021-01-15T16:10:28.354 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2021-01-15T16:10:28.565 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2021-01-15T16:10:28.813 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 15/36 2021-01-15T16:10:28.984 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2021-01-15T16:10:29.207 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2021-01-15T16:10:29.374 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2021-01-15T16:10:29.583 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2021-01-15T16:10:29.767 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2021-01-15T16:10:30.106 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2021-01-15T16:10:30.332 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2021-01-15T16:10:30.519 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2021-01-15T16:10:30.665 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2021-01-15T16:10:30.818 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2021-01-15T16:10:31.029 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2021-01-15T16:10:31.258 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2021-01-15T16:10:31.481 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2021-01-15T16:10:31.794 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2021-01-15T16:10:32.064 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2021-01-15T16:10:32.438 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8. 31/36 2021-01-15T16:10:32.590 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 32/36 2021-01-15T16:10:34.115 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 32/36 2021-01-15T16:10:34.330 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 33/36 2021-01-15T16:10:34.765 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 33/36 2021-01-15T16:10:34.939 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 34/36 2021-01-15T16:10:35.164 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 34/36 2021-01-15T16:10:35.570 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 35/36 2021-01-15T16:10:35.591 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 35/36 2021-01-15T16:10:35.711 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 36/36 2021-01-15T16:10:36.055 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 36/36 2021-01-15T16:10:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 1/36 2021-01-15T16:10:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 2/36 2021-01-15T16:10:36.056 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 3/36 2021-01-15T16:10:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 4/36 2021-01-15T16:10:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 5/36 2021-01-15T16:10:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8. 6/36 2021-01-15T16:10:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2021-01-15T16:10:36.057 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2021-01-15T16:10:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2021-01-15T16:10:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2021-01-15T16:10:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2021-01-15T16:10:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2021-01-15T16:10:36.058 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2021-01-15T16:10:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2021-01-15T16:10:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2021-01-15T16:10:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2021-01-15T16:10:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 17/36 2021-01-15T16:10:36.059 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2021-01-15T16:10:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2021-01-15T16:10:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.5.1-1.el8.noarch 20/36 2021-01-15T16:10:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2021-01-15T16:10:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2021-01-15T16:10:36.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2021-01-15T16:10:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2021-01-15T16:10:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2021-01-15T16:10:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2021-01-15T16:10:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2021-01-15T16:10:36.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2021-01-15T16:10:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2021-01-15T16:10:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2021-01-15T16:10:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2021-01-15T16:10:36.062 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 32/36 2021-01-15T16:10:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2021-01-15T16:10:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2021-01-15T16:10:36.063 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2021-01-15T16:10:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2021-01-15T16:10:36.870 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:36.871 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:36.871 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:10:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-01-15T16:10:36.872 INFO:teuthology.orchestra.run.smithi145.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-01-15T16:10:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-01-15T16:10:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.5.1-1.el8.noarch 2021-01-15T16:10:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-01-15T16:10:36.873 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-01-15T16:10:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-01-15T16:10:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-6.2-6.el8.noarch 2021-01-15T16:10:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-01-15T16:10:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-01-15T16:10:36.874 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-01-15T16:10:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.0.6-13.el8.noarch 2021-01-15T16:10:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-01-15T16:10:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-01-15T16:10:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-01-15T16:10:36.875 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-2.6-1.el8.noarch 2021-01-15T16:10:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-01-15T16:10:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2017.2-9.el8.noarch 2021-01-15T16:10:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-01-15T16:10:36.876 INFO:teuthology.orchestra.run.smithi145.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-01-15T16:10:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-01-15T16:10:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-01-15T16:10:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-01-15T16:10:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-01-15T16:10:36.877 INFO:teuthology.orchestra.run.smithi145.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-01-15T16:10:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-01-15T16:10:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-01-15T16:10:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-01-15T16:10:36.878 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-01-15T16:10:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-01-15T16:10:36.879 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:36.879 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:37.094 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-base 2021-01-15T16:10:37.553 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:04 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:37.885 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-base-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:10:38.003 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:38.004 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:10:38.004 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:38.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install cephadm 2021-01-15T16:10:38.509 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:05 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:39.000 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:39.001 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:39.001 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:10:39.002 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:39.002 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:39.003 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 66 k 2021-01-15T16:10:39.003 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:39.003 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:39.003 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:39.003 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:10:39.004 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:39.004 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 66 k 2021-01-15T16:10:39.004 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 262 k 2021-01-15T16:10:39.004 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:39.414 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm-16.0.0-9031.g892c51dd.el8.noarch.rpm 159 kB/s | 66 kB 00:00 2021-01-15T16:10:39.416 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:39.416 INFO:teuthology.orchestra.run.smithi145.stdout:Total 158 kB/s | 66 kB 00:00 2021-01-15T16:10:39.416 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:39.424 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:39.425 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:39.426 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:39.428 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:39.495 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:41.326 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T16:10:41.485 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T16:10:41.658 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T16:10:41.940 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/1 2021-01-15T16:10:41.941 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:41.941 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:41.941 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:10:41.941 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:41.942 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:42.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-immutable-object-cache 2021-01-15T16:10:42.594 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:09 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:43.055 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:43.056 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:43.056 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2021-01-15T16:10:43.057 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:43.057 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:43.058 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 163 k 2021-01-15T16:10:43.059 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:43.059 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:43.060 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:43.061 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:10:43.062 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:43.062 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 163 k 2021-01-15T16:10:43.063 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 587 k 2021-01-15T16:10:43.063 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:43.750 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-immutable-object-cache-16.0.0-9031.g892c51 234 kB/s | 163 kB 00:00 2021-01-15T16:10:43.751 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:43.751 INFO:teuthology.orchestra.run.smithi145.stdout:Total 234 kB/s | 163 kB 00:00 2021-01-15T16:10:43.752 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:43.765 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:43.766 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:43.815 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:43.817 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:43.926 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:44.127 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T16:10:44.476 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T16:10:44.725 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd. 1/1 2021-01-15T16:10:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:44.726 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:44.726 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:10:44.727 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:44.727 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:44.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr 2021-01-15T16:10:45.342 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:12 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:45.684 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-mgr-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:10:45.797 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:45.799 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:10:45.799 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:45.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-dashboard 2021-01-15T16:10:46.299 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:13 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:47.435 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:47.436 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:47.436 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:10:47.436 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:47.437 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:47.437 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 2.7 M 2021-01-15T16:10:47.437 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:10:47.437 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 24 k 2021-01-15T16:10:47.438 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 8.8 k 2021-01-15T16:10:47.438 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-01-15T16:10:47.438 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-01-15T16:10:47.438 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-01-15T16:10:47.438 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:47.439 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:47.439 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:10:47.439 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2021-01-15T16:10:47.439 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:47.440 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.0 M 2021-01-15T16:10:47.440 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 14 M 2021-01-15T16:10:47.440 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:47.739 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): ceph-prometheus-alerts-16.0.0-9031.g892c 29 kB/s | 8.8 kB 00:00 2021-01-15T16:10:47.770 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2021-01-15T16:10:47.799 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): ceph-grafana-dashboards-16.0.0-9031.g892 67 kB/s | 24 kB 00:00 2021-01-15T16:10:47.831 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 560 kB/s | 34 kB 00:00 2021-01-15T16:10:47.897 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.0 MB/s | 196 kB 00:00 2021-01-15T16:10:48.261 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): ceph-mgr-dashboard-16.0.0-9031.g892c51dd 3.3 MB/s | 2.7 MB 00:00 2021-01-15T16:10:48.265 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:48.266 INFO:teuthology.orchestra.run.smithi145.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2021-01-15T16:10:48.266 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:48.276 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:48.277 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:48.396 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:48.398 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:48.682 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:48.884 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-01-15T16:10:49.154 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-01-15T16:10:49.369 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-01-15T16:10:49.474 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.n 4/6 2021-01-15T16:10:50.472 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8. 5/6 2021-01-15T16:10:50.719 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 6/6 2021-01-15T16:10:50.941 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 6/6 2021-01-15T16:10:50.942 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8. 1/6 2021-01-15T16:10:50.942 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarc 2/6 2021-01-15T16:10:50.942 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.n 3/6 2021-01-15T16:10:50.942 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-01-15T16:10:50.943 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-01-15T16:10:51.207 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-01-15T16:10:51.207 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:51.207 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:10:51.208 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:10:51.208 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:10:51.208 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:10:51.208 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-01-15T16:10:51.209 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-01-15T16:10:51.209 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.4.1-12.el8.noarch 2021-01-15T16:10:51.209 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:51.209 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:10:51.280 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-diskprediction-local 2021-01-15T16:10:51.727 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:18 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:10:52.174 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:10:52.175 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2021-01-15T16:10:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:10:52.176 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2021-01-15T16:10:52.176 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:10:52.176 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 8.6 M 2021-01-15T16:10:52.176 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:10:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-01-15T16:10:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-01-15T16:10:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-01-15T16:10:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-01-15T16:10:52.177 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-01-15T16:10:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-01-15T16:10:52.178 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-01-15T16:10:52.179 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:52.180 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:10:52.180 INFO:teuthology.orchestra.run.smithi145.stdout:===================================================================================================== 2021-01-15T16:10:52.180 INFO:teuthology.orchestra.run.smithi145.stdout:Install 8 Packages 2021-01-15T16:10:52.180 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:10:52.181 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 36 M 2021-01-15T16:10:52.181 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 194 M 2021-01-15T16:10:52.181 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:10:52.294 INFO:teuthology.orchestra.run.smithi145.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 37 MB/s | 4.3 MB 00:00 2021-01-15T16:10:52.323 INFO:teuthology.orchestra.run.smithi145.stdout:(2/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 30 MB/s | 4.3 MB 00:00 2021-01-15T16:10:52.331 INFO:teuthology.orchestra.run.smithi145.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 30 MB/s | 225 kB 00:00 2021-01-15T16:10:52.377 INFO:teuthology.orchestra.run.smithi145.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 45 MB/s | 3.7 MB 00:00 2021-01-15T16:10:52.395 INFO:teuthology.orchestra.run.smithi145.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 35 MB/s | 639 kB 00:00 2021-01-15T16:10:52.403 INFO:teuthology.orchestra.run.smithi145.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 22 MB/s | 169 kB 00:00 2021-01-15T16:10:52.764 INFO:teuthology.orchestra.run.smithi145.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 34 MB/s | 15 MB 00:00 2021-01-15T16:10:53.915 INFO:teuthology.orchestra.run.smithi145.stdout:(8/8): ceph-mgr-diskprediction-local-16.0.0-903 4.9 MB/s | 8.6 MB 00:01 2021-01-15T16:10:53.919 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:10:53.919 INFO:teuthology.orchestra.run.smithi145.stdout:Total 21 MB/s | 36 MB 00:01 2021-01-15T16:10:53.929 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:10:53.950 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:10:53.950 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:10:54.241 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:10:54.243 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:10:54.794 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:10:54.941 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-01-15T16:10:55.107 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-01-15T16:10:55.249 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-01-15T16:10:55.734 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-01-15T16:10:55.883 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-01-15T16:10:56.395 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-01-15T16:10:56.549 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-01-15T16:10:57.253 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-01-15T16:10:57.546 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-01-15T16:10:57.724 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-01-15T16:10:59.644 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-01-15T16:11:00.938 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-01-15T16:11:01.115 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 8/8 2021-01-15T16:11:01.378 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 8/8 2021-01-15T16:11:01.379 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51d 1/8 2021-01-15T16:11:01.379 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-01-15T16:11:01.379 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-01-15T16:11:01.379 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-01-15T16:11:01.380 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-01-15T16:11:01.380 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-01-15T16:11:01.380 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-01-15T16:11:01.688 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-01-15T16:11:01.688 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:01.688 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:01.688 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:11:01.689 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-01-15T16:11:01.689 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-01-15T16:11:01.689 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.3-5.el8.x86_64 2021-01-15T16:11:01.689 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-01-15T16:11:01.690 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-01-15T16:11:01.690 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-01-15T16:11:01.690 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-01-15T16:11:01.690 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:01.691 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:01.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-rook 2021-01-15T16:11:02.209 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:29 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:02.658 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:02.660 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2021-01-15T16:11:02.661 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:11:02.661 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2021-01-15T16:11:02.661 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:02.661 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 27 k 2021-01-15T16:11:02.661 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:11:02.662 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-01-15T16:11:02.662 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-01-15T16:11:02.662 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-01-15T16:11:02.662 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-01-15T16:11:02.663 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-01-15T16:11:02.663 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-01-15T16:11:02.663 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-01-15T16:11:02.663 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-01-15T16:11:02.663 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-01-15T16:11:02.664 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-01-15T16:11:02.664 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-01-15T16:11:02.664 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-01-15T16:11:02.664 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:02.665 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:02.665 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2021-01-15T16:11:02.665 INFO:teuthology.orchestra.run.smithi145.stdout:Install 13 Packages 2021-01-15T16:11:02.665 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 1.9 M 2021-01-15T16:11:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 24 M 2021-01-15T16:11:02.666 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:02.695 INFO:teuthology.orchestra.run.smithi145.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 848 kB/s | 27 kB 00:00 2021-01-15T16:11:02.698 INFO:teuthology.orchestra.run.smithi145.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 580 kB/s | 20 kB 00:00 2021-01-15T16:11:02.699 INFO:teuthology.orchestra.run.smithi145.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 33 MB/s | 126 kB 00:00 2021-01-15T16:11:02.702 INFO:teuthology.orchestra.run.smithi145.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 33 MB/s | 110 kB 00:00 2021-01-15T16:11:02.704 INFO:teuthology.orchestra.run.smithi145.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 29 MB/s | 43 kB 00:00 2021-01-15T16:11:02.705 INFO:teuthology.orchestra.run.smithi145.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 29 MB/s | 155 kB 00:00 2021-01-15T16:11:02.753 INFO:teuthology.orchestra.run.smithi145.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 665 kB/s | 33 kB 00:00 2021-01-15T16:11:02.763 INFO:teuthology.orchestra.run.smithi145.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 280 kB/s | 16 kB 00:00 2021-01-15T16:11:02.786 INFO:teuthology.orchestra.run.smithi145.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.5 MB/s | 81 kB 00:00 2021-01-15T16:11:02.802 INFO:teuthology.orchestra.run.smithi145.stdout:(10/13): python3-rsa-3.4.2-16.el8.noarch.rpm 4.3 MB/s | 73 kB 00:00 2021-01-15T16:11:02.819 INFO:teuthology.orchestra.run.smithi145.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.6 MB/s | 61 kB 00:00 2021-01-15T16:11:02.883 INFO:teuthology.orchestra.run.smithi145.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 9.7 MB/s | 1.2 MB 00:00 2021-01-15T16:11:02.944 INFO:teuthology.orchestra.run.smithi145.stdout:(13/13): ceph-mgr-rook-16.0.0-9031.g892c51dd.el 98 kB/s | 27 kB 00:00 2021-01-15T16:11:02.945 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:02.946 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.8 MB/s | 1.9 MB 00:00 2021-01-15T16:11:02.951 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:02.965 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:02.966 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:03.034 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:03.037 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:03.528 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:03.752 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-01-15T16:11:04.784 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-01-15T16:11:05.022 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-01-15T16:11:05.273 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-01-15T16:11:05.458 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-01-15T16:11:05.675 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-01-15T16:11:05.918 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-01-15T16:11:06.113 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-01-15T16:11:07.165 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-01-15T16:11:07.421 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-01-15T16:11:07.580 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-01-15T16:11:07.764 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-01-15T16:11:07.956 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 13/13 2021-01-15T16:11:08.191 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 13/13 2021-01-15T16:11:08.192 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 1/13 2021-01-15T16:11:08.192 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-01-15T16:11:08.192 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-01-15T16:11:08.192 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-01-15T16:11:08.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-01-15T16:11:08.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-01-15T16:11:08.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-01-15T16:11:08.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-01-15T16:11:08.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-01-15T16:11:08.194 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-01-15T16:11:08.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-01-15T16:11:08.196 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-01-15T16:11:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-01-15T16:11:08.529 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:08.530 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:11:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-01-15T16:11:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-01-15T16:11:08.530 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-01-15T16:11:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-01-15T16:11:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-01-15T16:11:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-01-15T16:11:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-01-15T16:11:08.531 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-01-15T16:11:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-01-15T16:11:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-01-15T16:11:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-01-15T16:11:08.532 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-01-15T16:11:08.533 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:08.533 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:08.601 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-cephadm 2021-01-15T16:11:09.052 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:36 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:09.501 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:09.502 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:09.502 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:11:09.502 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:09.502 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:09.503 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:16.0.0-9031.g892c51dd.el8 ceph-noarch 75 k 2021-01-15T16:11:09.503 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:11:09.503 INFO:teuthology.orchestra.run.smithi145.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-01-15T16:11:09.503 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-01-15T16:11:09.504 INFO:teuthology.orchestra.run.smithi145.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-01-15T16:11:09.504 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-01-15T16:11:09.504 INFO:teuthology.orchestra.run.smithi145.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-01-15T16:11:09.504 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:09.504 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:09.505 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:09.505 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2021-01-15T16:11:09.505 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:09.506 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 5.8 M 2021-01-15T16:11:09.506 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 24 M 2021-01-15T16:11:09.506 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:09.550 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2021-01-15T16:11:09.604 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-01-15T16:11:09.675 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): ceph-mgr-cephadm-16.0.0-9031.g892c51dd.e 439 kB/s | 75 kB 00:00 2021-01-15T16:11:09.914 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 58 kB/s | 21 kB 00:00 2021-01-15T16:11:09.996 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 907 kB/s | 355 kB 00:00 2021-01-15T16:11:10.630 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): python3-remoto-1.1.4-4.el8.noarch.rpm 50 kB/s | 48 kB 00:00 2021-01-15T16:11:10.633 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:10.633 INFO:teuthology.orchestra.run.smithi145.stdout:Total 5.1 MB/s | 5.8 MB 00:01 2021-01-15T16:11:10.636 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:10.647 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:10.647 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:10.728 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:10.730 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:11.001 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:11.247 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-01-15T16:11:11.508 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-01-15T16:11:12.478 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-01-15T16:11:12.820 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-01-15T16:11:13.050 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-01-15T16:11:13.213 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 6/6 2021-01-15T16:11:13.429 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 6/6 2021-01-15T16:11:13.429 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 1/6 2021-01-15T16:11:13.429 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-01-15T16:11:13.430 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-01-15T16:11:13.430 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-01-15T16:11:13.430 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-01-15T16:11:13.719 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-01-15T16:11:13.719 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:13.719 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:13.719 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:16.0.0-9031.g892c51dd.el8.noarch 2021-01-15T16:11:13.720 INFO:teuthology.orchestra.run.smithi145.stdout: python3-apipkg-1.5-6.el8.noarch 2021-01-15T16:11:13.720 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.5.1-5.el8.noarch 2021-01-15T16:11:13.720 INFO:teuthology.orchestra.run.smithi145.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-01-15T16:11:13.720 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-01-15T16:11:13.720 INFO:teuthology.orchestra.run.smithi145.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-01-15T16:11:13.721 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:13.721 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:13.789 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-fuse 2021-01-15T16:11:14.242 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:41 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:14.692 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:14.692 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:14.693 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2021-01-15T16:11:14.693 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:14.693 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:14.693 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 877 k 2021-01-15T16:11:14.694 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2021-01-15T16:11:14.694 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-01-15T16:11:14.694 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-01-15T16:11:14.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:14.694 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:14.695 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:14.695 INFO:teuthology.orchestra.run.smithi145.stdout:Install 3 Packages 2021-01-15T16:11:14.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:14.696 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 981 k 2021-01-15T16:11:14.696 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 3.2 M 2021-01-15T16:11:14.696 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:14.754 INFO:teuthology.orchestra.run.smithi145.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 1.4 MB/s | 82 kB 00:00 2021-01-15T16:11:14.761 INFO:teuthology.orchestra.run.smithi145.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 316 kB/s | 21 kB 00:00 2021-01-15T16:11:14.958 INFO:teuthology.orchestra.run.smithi145.stdout:(3/3): ceph-fuse-16.0.0-9031.g892c51dd.el8.x86_ 3.3 MB/s | 877 kB 00:00 2021-01-15T16:11:14.961 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:14.963 INFO:teuthology.orchestra.run.smithi145.stdout:Total 3.6 MB/s | 981 kB 00:00 2021-01-15T16:11:14.964 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:14.985 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:14.986 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:15.058 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:15.061 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:15.233 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:15.234 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-01-15T16:11:15.234 INFO:teuthology.orchestra.run.smithi145.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-01-15T16:11:15.234 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:15.646 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-01-15T16:11:15.937 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 3/3 2021-01-15T16:11:16.265 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 3/3 2021-01-15T16:11:16.266 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/3 2021-01-15T16:11:16.266 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2021-01-15T16:11:16.592 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2021-01-15T16:11:16.592 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:16.592 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:16.593 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-01-15T16:11:16.593 INFO:teuthology.orchestra.run.smithi145.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-01-15T16:11:16.593 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:16.593 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:16.749 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados-devel 2021-01-15T16:11:17.210 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:44 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:17.701 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:17.702 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:17.702 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2021-01-15T16:11:17.702 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:17.702 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:17.703 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 131 k 2021-01-15T16:11:17.703 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:17.703 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:17.703 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:17.703 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:11:17.704 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:17.704 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 131 k 2021-01-15T16:11:17.704 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 437 k 2021-01-15T16:11:17.704 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:17.874 INFO:teuthology.orchestra.run.smithi145.stdout:librados-devel-16.0.0-9031.g892c51dd.el8.x86_64 762 kB/s | 131 kB 00:00 2021-01-15T16:11:17.875 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:17.875 INFO:teuthology.orchestra.run.smithi145.stdout:Total 757 kB/s | 131 kB 00:00 2021-01-15T16:11:17.876 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:17.883 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:17.883 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:17.932 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:17.935 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:18.044 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:18.211 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:18.451 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:18.759 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:18.759 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:18.760 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:18.760 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:11:18.760 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:18.760 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:18.905 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs2 2021-01-15T16:11:19.363 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:46 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:19.710 INFO:teuthology.orchestra.run.smithi145.stdout:Package libcephfs2-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:19.824 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:19.825 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:19.825 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:19.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs-devel 2021-01-15T16:11:20.336 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:47 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:20.788 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:20.789 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:20.789 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2021-01-15T16:11:20.790 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:20.790 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 24 k 2021-01-15T16:11:20.790 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:20.791 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:20.791 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:20.791 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:11:20.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:20.792 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 24 k 2021-01-15T16:11:20.792 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 89 k 2021-01-15T16:11:20.792 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:21.147 INFO:teuthology.orchestra.run.smithi145.stdout:libcephfs-devel-16.0.0-9031.g892c51dd.el8.x86_6 67 kB/s | 24 kB 00:00 2021-01-15T16:11:21.148 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:21.148 INFO:teuthology.orchestra.run.smithi145.stdout:Total 67 kB/s | 24 kB 00:00 2021-01-15T16:11:21.149 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:21.153 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:21.153 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:21.155 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:21.158 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:21.215 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:21.552 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:21.773 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:21.991 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:21.992 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:11:21.992 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:21.992 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:22.060 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados2 2021-01-15T16:11:22.513 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:49 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:22.852 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:22.967 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:22.968 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:22.968 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:23.023 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librbd1 2021-01-15T16:11:23.476 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:50 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:23.843 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:23.959 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:23.960 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:23.960 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:24.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rados 2021-01-15T16:11:24.467 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:51 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:24.801 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rados-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:24.918 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:24.919 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:24.919 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:24.976 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rgw 2021-01-15T16:11:25.430 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:52 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:25.792 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rgw-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:25.905 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:25.906 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:25.906 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:25.955 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-cephfs 2021-01-15T16:11:26.413 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:53 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:26.742 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-cephfs-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:26.859 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:26.860 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:26.860 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:27.167 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rbd 2021-01-15T16:11:27.626 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:54 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:27.959 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rbd-2:16.0.0-9031.g892c51dd.el8.x86_64 is already installed. 2021-01-15T16:11:28.071 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:28.072 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2021-01-15T16:11:28.072 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:28.121 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-fuse 2021-01-15T16:11:28.571 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:55 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:29.045 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:29.046 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2021-01-15T16:11:29.047 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:29.047 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:29.047 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 89 k 2021-01-15T16:11:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:29.048 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:29.048 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:29.048 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:11:29.048 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:29.049 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 89 k 2021-01-15T16:11:29.049 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 298 k 2021-01-15T16:11:29.049 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:29.414 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-fuse-16.0.0-9031.g892c51dd.el8.x86_64.rpm 242 kB/s | 89 kB 00:00 2021-01-15T16:11:29.415 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:29.415 INFO:teuthology.orchestra.run.smithi145.stdout:Total 241 kB/s | 89 kB 00:00 2021-01-15T16:11:29.415 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:29.427 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:29.428 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:29.478 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:29.480 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:29.588 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:29.767 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:29.987 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:30.261 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:30.262 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:30.262 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:30.262 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:11:30.262 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:30.263 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:30.420 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-mirror 2021-01-15T16:11:30.879 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:57 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:31.348 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:31.349 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:31.349 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2021-01-15T16:11:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:31.350 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:31.350 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 3.6 M 2021-01-15T16:11:31.350 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:31.351 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:31.351 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:31.351 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:11:31.351 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:31.352 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.6 M 2021-01-15T16:11:31.352 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 18 M 2021-01-15T16:11:31.352 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:33.736 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-mirror-16.0.0-9031.g892c51dd.el8.x86_64.rpm 1.5 MB/s | 3.6 MB 00:02 2021-01-15T16:11:33.737 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:33.738 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.5 MB/s | 3.6 MB 00:02 2021-01-15T16:11:33.738 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:33.754 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:33.754 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:33.821 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:33.824 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:34.607 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:34.818 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:35.182 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:35.467 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:35.467 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:11:35.468 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:35.468 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:35.602 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-nbd 2021-01-15T16:11:36.083 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:03 ago on Fri 15 Jan 2021 04:09:33 PM UTC. 2021-01-15T16:11:36.531 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2021-01-15T16:11:36.532 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:36.533 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2021-01-15T16:11:36.533 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:36.533 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2021-01-15T16:11:36.533 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:16.0.0-9031.g892c51dd.el8 ceph 174 k 2021-01-15T16:11:36.533 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:36.534 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2021-01-15T16:11:36.534 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2021-01-15T16:11:36.534 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2021-01-15T16:11:36.534 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:36.535 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 174 k 2021-01-15T16:11:36.535 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 649 k 2021-01-15T16:11:36.535 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2021-01-15T16:11:37.013 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-nbd-16.0.0-9031.g892c51dd.el8.x86_64.rpm 361 kB/s | 174 kB 00:00 2021-01-15T16:11:37.014 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2021-01-15T16:11:37.014 INFO:teuthology.orchestra.run.smithi145.stdout:Total 360 kB/s | 174 kB 00:00 2021-01-15T16:11:37.015 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2021-01-15T16:11:37.028 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2021-01-15T16:11:37.028 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2021-01-15T16:11:37.081 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2021-01-15T16:11:37.083 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2021-01-15T16:11:37.199 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2021-01-15T16:11:37.407 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:37.642 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:37.914 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 1/1 2021-01-15T16:11:37.915 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:37.915 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2021-01-15T16:11:37.915 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:16.0.0-9031.g892c51dd.el8.x86_64 2021-01-15T16:11:37.915 INFO:teuthology.orchestra.run.smithi145.stdout: 2021-01-15T16:11:37.915 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2021-01-15T16:11:38.071 DEBUG:teuthology.parallel:result is None 2021-01-15T16:11:38.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo lsb_release -is 2021-01-15T16:11:38.105 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS 2021-01-15T16:11:38.106 DEBUG:teuthology.misc:System to be installed: CentOS 2021-01-15T16:11:38.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T16:11:38.270 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-01-15T16:11:38.290 INFO:teuthology.orchestra.run.smithi145.stdout:16.0.0-9031.g892c51dd.el8 2021-01-15T16:11:38.290 INFO:teuthology.packaging:The installed version of ceph is 16.0.0-9031.g892c51dd.el8 2021-01-15T16:11:38.291 INFO:teuthology.task.install:The correct ceph version 16.0.0-9031.g892c51dd is installed. 2021-01-15T16:11:38.291 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-01-15T16:11:38.292 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:11:38.292 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-01-15T16:11:38.355 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-01-15T16:11:38.357 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:11:38.358 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2021-01-15T16:11:38.420 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-01-15T16:11:38.483 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-01-15T16:11:38.484 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:11:38.484 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2021-01-15T16:11:38.547 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-01-15T16:11:38.611 INFO:teuthology.run_tasks:Running task workunit... 2021-01-15T16:11:38.627 INFO:tasks.workunit:Pulling workunits from ref 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T16:11:38.628 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-01-15T16:11:38.628 INFO:tasks.workunit:timeout=3h 2021-01-15T16:11:38.629 INFO:tasks.workunit:cleanup=True 2021-01-15T16:11:38.629 DEBUG:teuthology.orchestra.run.smithi145:> stat -- /home/ubuntu/cephtest/mnt.0 2021-01-15T16:11:38.665 INFO:teuthology.orchestra.run.smithi145.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-01-15T16:11:38.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-01-15T16:11:38.666 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-01-15T16:11:38.719 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-01-15T16:11:38.720 DEBUG:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-01-15T16:11:38.773 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 892c51dd3c5f7108e766bea30cd5e0d801b0abd3 2021-01-15T16:11:38.830 INFO:tasks.workunit.client.0.smithi145.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-01-15T16:21:30.170 INFO:tasks.workunit.client.0.smithi145.stderr:Note: switching to '892c51dd3c5f7108e766bea30cd5e0d801b0abd3'. 2021-01-15T16:21:30.216 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.216 INFO:tasks.workunit.client.0.smithi145.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-01-15T16:21:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:changes and commit them, and you can discard any commits you make in this 2021-01-15T16:21:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:state without impacting any branches by switching back to a branch. 2021-01-15T16:21:30.217 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:If you want to create a new branch to retain commits you create, you may 2021-01-15T16:21:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:do so (now or later) by using -c with the switch command. Example: 2021-01-15T16:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr: git switch -c 2021-01-15T16:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr:Or undo this operation with: 2021-01-15T16:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.219 INFO:tasks.workunit.client.0.smithi145.stderr: git switch - 2021-01-15T16:21:30.219 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.219 INFO:tasks.workunit.client.0.smithi145.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-01-15T16:21:30.219 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:21:30.219 INFO:tasks.workunit.client.0.smithi145.stderr:HEAD is now at 892c51dd3c Merge PR #38896 into master 2021-01-15T16:21:30.220 DEBUG:teuthology.orchestra.run.smithi145:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-01-15T16:21:30.236 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2021-01-15T16:21:30.236 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-01-15T16:21:30.291 INFO:tasks.workunit:Running workunits matching osd on client.0... 2021-01-15T16:21:30.292 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2021-01-15T16:21:30.293 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2021-01-15T16:21:30.362 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:21:30.393 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:21:30.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:21:30.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:21:30.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:21:30.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:21:30.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:21:30.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:21:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:21:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:21:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/bad-inc-map 2021-01-15T16:21:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2021-01-15T16:21:30.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2021-01-15T16:21:30.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:41585 2021-01-15T16:21:30.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:41585 2021-01-15T16:21:30.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2021-01-15T16:21:30.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2021-01-15T16:21:30.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none ' 2021-01-15T16:21:30.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:41585 ' 2021-01-15T16:21:30.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2021-01-15T16:21:30.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2021-01-15T16:21:30.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:21:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2021-01-15T16:21:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2021-01-15T16:21:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2021-01-15T16:21:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2021-01-15T16:21:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2021-01-15T16:21:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-01-15T16:21:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:21:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-01-15T16:21:30.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:21:30.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:21:30.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:21:30.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:21:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:21:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:21:30.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:21:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:21:30.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:21:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:21:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:21:30.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:21:30.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:21:30.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:21:30.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:21:30.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:21:30.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:21:30.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:21:30.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-01-15T16:21:30.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:21:30.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:30.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:30.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29068 2021-01-15T16:21:30.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:21:30.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:21:30.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2021-01-15T16:21:30.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:21:30.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:30.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:30.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.29068 2021-01-15T16:21:30.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:21:30.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:21:30.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:21:30.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:21:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2021-01-15T16:21:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2021-01-15T16:21:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2021-01-15T16:21:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2021-01-15T16:21:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2021-01-15T16:21:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:21:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:21:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:21:30.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2021-01-15T16:21:30.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2021-01-15T16:21:30.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:21:30.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:21:30.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:21:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:21:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:30.809 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:30.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:30.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:21:30.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:21:30.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:21:30.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:21:30.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:21:30.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:21:30.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:21:30.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:21:30.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:21:30.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:21:30.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:21:30.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:30.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:30.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29068/ceph-mon.a.asok 2021-01-15T16:21:30.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:21:30.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.29068/ceph-mon.a.asok config get fsid 2021-01-15T16:21:31.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:21:31.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:21:31.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:21:31.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:21:31.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:21:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:21:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:21:31.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:21:31.090 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:21:31.090 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:31.090 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:31.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.29068/ceph-mon.a.asok 2021-01-15T16:21:31.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:21:31.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.29068/ceph-mon.a.asok config get mon_host 2021-01-15T16:21:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2021-01-15T16:21:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/bad-inc-map 2021-01-15T16:21:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:21:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:21:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:21:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/bad-inc-map/x 2021-01-15T16:21:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:21:32.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:21:32.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:21:32.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:21:32.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:21:32.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:32.080 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:32.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:32.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:21:32.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:21:32.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2021-01-15T16:21:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2021-01-15T16:21:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:21:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:21:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:21:32.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/0 2021-01-15T16:21:32.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 ' 2021-01-15T16:21:32.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:21:32.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:21:32.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:21:32.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2021-01-15T16:21:32.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2021-01-15T16:21:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:21:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:21:32.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-01-15T16:21:32.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:21:32.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:21:32.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:21:32.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:21:32.122 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:32.123 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:32.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:32.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:32.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:21:32.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:21:32.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:21:32.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-01-15T16:21:32.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-01-15T16:21:32.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:21:32.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:21:32.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:32.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:21:32.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:21:32.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/0 2021-01-15T16:21:32.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:21:32.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7a8e1f65-4f97-4406-84f8-2865eff00b6d 2021-01-15T16:21:32.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7a8e1f65-4f97-4406-84f8-2865eff00b6d' 2021-01-15T16:21:32.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:21:32.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAMwQFgHZHXBhAATrAw+XfOYDPnAYt2Clf7tg== 2021-01-15T16:21:32.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAMwQFgHZHXBhAATrAw+XfOYDPnAYt2Clf7tg=="}' 2021-01-15T16:21:32.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7a8e1f65-4f97-4406-84f8-2865eff00b6d -i td/bad-inc-map/0/new.json 2021-01-15T16:21:32.137 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 7a8e1f65-4f97-4406-84f8-2865eff00b6d 2021-01-15T16:21:32.461 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:21:32.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/0/new.json 2021-01-15T16:21:32.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAMwQFgHZHXBhAATrAw+XfOYDPnAYt2Clf7tg== --osd-uuid 7a8e1f65-4f97-4406-84f8-2865eff00b6d 2021-01-15T16:21:32.494 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.494+0000 7f4f1a9c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:32.496 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.496+0000 7f4f1a9c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:32.507 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.497+0000 7f4f1a9c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:32.507 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.498+0000 7f4f1a9c9080 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-01-15T16:21:32.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.498+0000 7f4f1a9c9080 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-01-15T16:21:32.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.498+0000 7f4f1a9c9080 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2021-01-15T16:21:32.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:32.551+0000 7f4f1a9c9080 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2021-01-15T16:21:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/0/keyring 2021-01-15T16:21:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:21:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:21:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:21:37.026 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:21:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:21:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:21:37.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:21:37.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:21:37.536 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:21:37.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:37.552+0000 7f9cb0fd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:37.563 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:37.562+0000 7f9cb0fd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:37.564 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:37.563+0000 7f9cb0fd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:21:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:21:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:21:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:21:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:21:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:21:37.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:37.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:21:37.926 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:21:38.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:38.087+0000 7f9cb0fd3080 -1 Falling back to public interface 2021-01-15T16:21:38.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:21:38.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:38.923+0000 7f9cb0fd3080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:21:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:21:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:21:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:21:39.315 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:21:39.719 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30118,v1:127.0.0.1:6803/30118] [v2:127.0.0.1:6804/30118,v1:127.0.0.1:6805/30118] exists,up 7a8e1f65-4f97-4406-84f8-2865eff00b6d 2021-01-15T16:21:39.720 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 2021-01-15T16:21:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:21:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:21:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:21:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2021-01-15T16:21:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2021-01-15T16:21:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:21:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:21:39.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:21:39.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/1 2021-01-15T16:21:39.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 ' 2021-01-15T16:21:39.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:21:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:21:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:21:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2021-01-15T16:21:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2021-01-15T16:21:39.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:21:39.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:21:39.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-01-15T16:21:39.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:21:39.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:21:39.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:21:39.725 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:21:39.725 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:39.726 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:39.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:39.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:39.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:21:39.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:21:39.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:21:39.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-01-15T16:21:39.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-01-15T16:21:39.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:21:39.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:21:39.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:39.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:21:39.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:21:39.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/1 2021-01-15T16:21:39.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:21:39.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 2021-01-15T16:21:39.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5' 2021-01-15T16:21:39.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:21:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQATwQFg+tOiKxAA9hu8QRb8NAL8Lf4qJvpqPw== 2021-01-15T16:21:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQATwQFg+tOiKxAA9hu8QRb8NAL8Lf4qJvpqPw=="}' 2021-01-15T16:21:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 -i td/bad-inc-map/1/new.json 2021-01-15T16:21:40.212 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:21:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/1/new.json 2021-01-15T16:21:40.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQATwQFg+tOiKxAA9hu8QRb8NAL8Lf4qJvpqPw== --osd-uuid 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 2021-01-15T16:21:40.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.245+0000 7ff38de10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.247+0000 7ff38de10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.248+0000 7ff38de10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.249+0000 7ff38de10080 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-01-15T16:21:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.249+0000 7ff38de10080 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-01-15T16:21:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.249+0000 7ff38de10080 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2021-01-15T16:21:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:40.335+0000 7ff38de10080 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2021-01-15T16:21:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/1/keyring 2021-01-15T16:21:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:21:44.586 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:21:44.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:21:44.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:21:45.109 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:21:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:21:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:21:45.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:21:45.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:21:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:45.127+0000 7fa1831d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:45.131 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:45.129+0000 7fa1831d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:45.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:45.130+0000 7fa1831d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:21:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:21:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:21:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:21:45.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:21:45.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:45.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:21:45.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:45.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:21:45.498 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:21:45.647 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:45.647+0000 7fa1831d2080 -1 Falling back to public interface 2021-01-15T16:21:45.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:21:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:46.386+0000 7fa1831d2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:21:46.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:21:46.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:46.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:21:46.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:46.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:21:46.887 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:21:47.298 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/31382,v1:127.0.0.1:6811/31382] [v2:127.0.0.1:6812/31382,v1:127.0.0.1:6813/31382] exists,up 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 2021-01-15T16:21:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:21:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:21:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:21:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2021-01-15T16:21:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/bad-inc-map 2021-01-15T16:21:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:21:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:21:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:21:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/bad-inc-map/2 2021-01-15T16:21:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 ' 2021-01-15T16:21:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:21:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:21:47.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:21:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2021-01-15T16:21:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2021-01-15T16:21:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:21:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:21:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2021-01-15T16:21:47.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:21:47.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:21:47.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:21:47.303 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:21:47.303 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:21:47.304 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:21:47.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:47.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' 2021-01-15T16:21:47.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:21:47.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:21:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:21:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2021-01-15T16:21:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2021-01-15T16:21:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:21:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:21:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:21:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:21:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/bad-inc-map/2 2021-01-15T16:21:47.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:21:47.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a2fc119b-0496-4310-895c-237d3293fdf6 2021-01-15T16:21:47.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a2fc119b-0496-4310-895c-237d3293fdf6' 2021-01-15T16:21:47.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:21:47.308 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 a2fc119b-0496-4310-895c-237d3293fdf6 2021-01-15T16:21:47.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAbwQFgZoqxEhAA6NyWIt63ZwSJohZOG3+aaw== 2021-01-15T16:21:47.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAbwQFgZoqxEhAA6NyWIt63ZwSJohZOG3+aaw=="}' 2021-01-15T16:21:47.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a2fc119b-0496-4310-895c-237d3293fdf6 -i td/bad-inc-map/2/new.json 2021-01-15T16:21:47.778 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:21:47.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/bad-inc-map/2/new.json 2021-01-15T16:21:47.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAbwQFgZoqxEhAA6NyWIt63ZwSJohZOG3+aaw== --osd-uuid a2fc119b-0496-4310-895c-237d3293fdf6 2021-01-15T16:21:47.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.811+0000 7f172abda080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:47.813 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.813+0000 7f172abda080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:47.813 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.813+0000 7f172abda080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:47.814 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.814+0000 7f172abda080 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-01-15T16:21:47.814 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.814+0000 7f172abda080 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-01-15T16:21:47.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.814+0000 7f172abda080 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2021-01-15T16:21:47.869 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:47.869+0000 7f172abda080 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2021-01-15T16:21:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/bad-inc-map/2/keyring 2021-01-15T16:21:52.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:21:52.319 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:21:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:21:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:21:52.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:21:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:21:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=d7d077db-8e58-4bbd-831f-a2ce0b8c191f --auth-supported=none --mon-host=127.0.0.1:41585 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.29068/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:21:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:21:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:21:52.851 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:21:52.868 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:52.868+0000 7fc7731d8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:52.891 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:52.891+0000 7fc7731d8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:52.893 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:52.892+0000 7fc7731d8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:21:53.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:21:53.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:21:53.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:21:53.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:21:53.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:21:53.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:53.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:21:53.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:53.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:21:53.243 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:21:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:53.409+0000 7fc7731d8080 -1 Falling back to public interface 2021-01-15T16:21:53.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:21:54.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:21:54.201+0000 7fc7731d8080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:21:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:21:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:21:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:21:54.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:21:54.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:21:54.635 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:21:55.035 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/32646,v1:127.0.0.1:6819/32646] [v2:127.0.0.1:6820/32646,v1:127.0.0.1:6821/32646] exists,up a2fc119b-0496-4310-895c-237d3293fdf6 2021-01-15T16:21:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:21:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:21:55.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:21:55.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2021-01-15T16:21:55.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2021-01-15T16:21:55.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 8 2021-01-15T16:21:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T16:21:56.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:21:57.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2021-01-15T16:21:58.116 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 1 2021-01-15T16:21:58.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2021-01-15T16:21:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T16:21:59.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2021-01-15T16:22:04.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2021-01-15T16:22:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2021-01-15T16:22:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:22:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2021-01-15T16:22:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:22:04.081 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:04.473 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/30118,v1:127.0.0.1:6803/30118] [v2:127.0.0.1:6804/30118,v1:127.0.0.1:6805/30118] exists,up 7a8e1f65-4f97-4406-84f8-2865eff00b6d 2021-01-15T16:22:04.473 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2021-01-15T16:22:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2021-01-15T16:22:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:22:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:04.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2021-01-15T16:22:04.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:04.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:04.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:22:04.891 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/31382,v1:127.0.0.1:6811/31382] [v2:127.0.0.1:6812/31382,v1:127.0.0.1:6813/31382] exists,up 4e9299c2-9b38-4a8f-99cf-84ae459b1dc5 2021-01-15T16:22:04.891 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:04.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:04.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:04.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:04.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2021-01-15T16:22:04.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2021-01-15T16:22:04.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:04.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:22:04.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:04.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:04.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 10 )) 2021-01-15T16:22:04.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:04.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:04.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:22:05.275 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 14 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/32646,v1:127.0.0.1:6819/32646] [v2:127.0.0.1:6820/32646,v1:127.0.0.1:6821/32646] exists,up a2fc119b-0496-4310-895c-237d3293fdf6 2021-01-15T16:22:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2021-01-15T16:22:05.277 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.124+0000 7fc75d982700 -1 osd.2 15 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.277 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.217+0000 7fc75d982700 -1 osd.2 15 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.277 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.775+0000 7fc75d982700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.278 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.776+0000 7fc75d982700 -1 osd.2 17 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.278 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.349+0000 7fc7596be700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.278 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.349+0000 7fc7596be700 -1 osd.2 19 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.278 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.475+0000 7fc75d982700 -1 osd.2 20 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.279 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.760+0000 7fc7596be700 -1 osd.2 21 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.279 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.887+0000 7fc75d982700 -1 osd.2 22 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.279 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.233+0000 7fc7596be700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.279 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.233+0000 7fc7596be700 -1 osd.2 23 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.279 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.380+0000 7fc75d982700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.280 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.381+0000 7fc75d982700 -1 osd.2 24 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.280 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.615+0000 7fc75d982700 -1 osd.2 25 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.280 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.064+0000 7fc75d982700 -1 osd.2 26 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.280 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.613+0000 7fc7596be700 -1 osd.2 27 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.280 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.613+0000 7fc7596be700 -1 osd.2 27 handle_osd_map injecting map crc failure 2021-01-15T16:22:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2021-01-15T16:22:05.282 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.125+0000 7fc75d982700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2021-01-15T16:22:05.282 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.217+0000 7fc75d982700 10 osd.2 15 handle_osd_map bailing because last < start (16<17) 2021-01-15T16:22:05.282 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.775+0000 7fc75d982700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-01-15T16:22:05.282 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:56.776+0000 7fc75d982700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2021-01-15T16:22:05.283 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.349+0000 7fc7596be700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-01-15T16:22:05.283 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.350+0000 7fc7596be700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2021-01-15T16:22:05.283 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.475+0000 7fc75d982700 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2021-01-15T16:22:05.283 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.761+0000 7fc7596be700 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2021-01-15T16:22:05.284 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:57.888+0000 7fc75d982700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2021-01-15T16:22:05.284 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.233+0000 7fc7596be700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-01-15T16:22:05.284 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.234+0000 7fc7596be700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2021-01-15T16:22:05.285 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.381+0000 7fc75d982700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-01-15T16:22:05.285 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.382+0000 7fc75d982700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2021-01-15T16:22:05.285 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:58.615+0000 7fc75d982700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2021-01-15T16:22:05.285 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.064+0000 7fc75d982700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2021-01-15T16:22:05.285 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.613+0000 7fc7596be700 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2021-01-15T16:22:05.286 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T16:21:59.613+0000 7fc7596be700 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2021-01-15T16:22:05.286 INFO:tasks.workunit.client.0.smithi145.stdout:success 2021-01-15T16:22:05.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2021-01-15T16:22:05.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2021-01-15T16:22:05.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=foo 2021-01-15T16:22:05.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-01-15T16:22:05.786 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' does not exist 2021-01-15T16:22:05.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2021-01-15T16:22:05.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:22:05.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:22:05.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:22:05.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:22:05.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:22:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:22:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2021-01-15T16:22:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-01-15T16:22:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:22:05.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-01-15T16:22:05.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:22:05.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:22:05.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:22:05.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:22:05.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:22:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:22:05.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:22:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:22:05.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:22:05.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:22:05.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:22:05.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:22:05.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:05.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:22:05.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:22:05.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:05.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:22:05.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:22:05.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-01-15T16:22:05.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:22:05.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:05.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:22:05.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29068 2021-01-15T16:22:05.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:22:05.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:22:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T16:22:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/bad-inc-map 0 2021-01-15T16:22:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2021-01-15T16:22:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T16:22:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2021-01-15T16:22:05.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:22:05.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:22:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:22:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:22:05.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:22:05.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:22:05.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:22:05.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:22:05.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:22:05.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:22:05.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:22:05.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:22:05.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:05.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:22:05.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:22:05.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:05.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:22:05.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T16:22:05.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2021-01-15T16:22:05.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:22:05.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:05.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.29068 2021-01-15T16:22:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.29068 2021-01-15T16:22:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:22:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:22:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T16:22:05.931 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T16:22:05.932 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T16:22:06.005 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2021-01-15T16:22:06.006 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2021-01-15T16:22:06.067 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:22:06.076 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:22:06.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:22:06.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:22:06.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:22:06.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:22:06.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:22:06.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:22:06.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:22:06.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:22:06.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/divergent-priors 2021-01-15T16:22:06.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2021-01-15T16:22:06.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2021-01-15T16:22:06.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2021-01-15T16:22:06.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2021-01-15T16:22:06.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2021-01-15T16:22:06.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2021-01-15T16:22:06.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2021-01-15T16:22:06.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2021-01-15T16:22:06.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2021-01-15T16:22:06.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2021-01-15T16:22:06.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2021-01-15T16:22:06.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2021-01-15T16:22:06.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2021-01-15T16:22:06.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2021-01-15T16:22:06.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2021-01-15T16:22:06.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2021-01-15T16:22:06.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2021-01-15T16:22:06.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2021-01-15T16:22:06.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2021-01-15T16:22:06.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2021-01-15T16:22:06.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2021-01-15T16:22:06.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2021-01-15T16:22:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2021-01-15T16:22:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2021-01-15T16:22:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2021-01-15T16:22:06.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2021-01-15T16:22:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none ' 2021-01-15T16:22:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-01-15T16:22:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T16:22:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2021-01-15T16:22:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:22:06.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2021-01-15T16:22:06.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:22:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2021-01-15T16:22:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_2 2021-01-15T16:22:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_3 2021-01-15T16:22:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_divergent_ec' 2021-01-15T16:22:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-01-15T16:22:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-01-15T16:22:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-01-15T16:22:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-01-15T16:22:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:22:06.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:22:06.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:22:06.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:22:06.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:22:06.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:22:06.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:22:06.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:22:06.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:22:06.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:22:06.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:22:06.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:22:06.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:22:06.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:22:06.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:22:06.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:06.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:22:06.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:22:06.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:22:06.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:22:06.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:22:06.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:22:06.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:22:06.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:06.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:06.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:22:06.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:22:06.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:22:06.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-01-15T16:22:06.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:22:06.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:06.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:06.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.33425 2021-01-15T16:22:06.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:22:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:22:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:22:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:22:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2021-01-15T16:22:06.101 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2021-01-15T16:22:06.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2021-01-15T16:22:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2021-01-15T16:22:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:22:06.107 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:22:06.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2021-01-15T16:22:06.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-01-15T16:22:06.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:22:06.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:22:06.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:22:06.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-01-15T16:22:06.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-01-15T16:22:06.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:22:06.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:22:06.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:22:06.299 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:22:06.300 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:06.300 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:06.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:06.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:22:06.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:22:06.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:22:06.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:22:06.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:22:06.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:22:06.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:22:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:22:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:22:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:22:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:22:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:06.484 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:06.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:22:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:22:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get fsid 2021-01-15T16:22:06.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:22:06.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:22:06.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:22:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:22:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:06.587 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:06.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:22:06.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:22:06.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get mon_host 2021-01-15T16:22:06.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2021-01-15T16:22:06.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2021-01-15T16:22:06.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:22:06.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:22:06.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:22:06.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2021-01-15T16:22:06.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:22:07.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:22:07.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:22:07.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:22:07.176 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:22:07.176 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:07.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:07.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:07.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:22:07.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:22:07.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-01-15T16:22:07.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2021-01-15T16:22:07.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:22:07.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:22:07.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:22:07.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:22:07.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:22:07.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:22:07.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:22:07.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:22:07.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:22:07.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:22:07.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:22:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:22:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:22:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:22:07.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:22:07.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:22:07.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:22:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:22:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:07.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:07.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:07.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:07.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:22:07.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:22:07.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:22:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:22:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:22:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:22:07.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:22:07.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:07.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:22:07.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:22:07.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:22:07.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:22:07.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:22:07.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d3275fea-a12c-420f-90e6-0ac2fa60dc87' 2021-01-15T16:22:07.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:22:07.217 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:22:07.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAvwQFg0s1SDRAAYeaLRDbvA3FIY7m9/Yhi8A== 2021-01-15T16:22:07.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAvwQFg0s1SDRAAYeaLRDbvA3FIY7m9/Yhi8A=="}' 2021-01-15T16:22:07.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d3275fea-a12c-420f-90e6-0ac2fa60dc87 -i td/divergent-priors/0/new.json 2021-01-15T16:22:07.564 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:07.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2021-01-15T16:22:07.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAvwQFg0s1SDRAAYeaLRDbvA3FIY7m9/Yhi8A== --osd-uuid d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:22:07.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.605+0000 7f7474484080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:07.607 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.607+0000 7f7474484080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:07.608 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.608+0000 7f7474484080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:07.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.609+0000 7f7474484080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:22:07.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.609+0000 7f7474484080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:22:07.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.609+0000 7f7474484080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:22:07.639 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:07.638+0000 7f7474484080 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-01-15T16:22:12.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-01-15T16:22:12.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:22:12.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:22:12.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:22:12.158 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:22:12.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:22:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:22:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:22:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:22:12.683 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:22:12.711 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:12.710+0000 7fe8e1f19080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:12.721+0000 7fe8e1f19080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:12.722+0000 7fe8e1f19080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:13.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:22:13.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:13.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:22:13.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:13.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:13.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:13.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:13.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:13.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:22:13.085 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:13.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:13.240+0000 7fe8e1f19080 -1 Falling back to public interface 2021-01-15T16:22:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:22:13.975 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:13.974+0000 7fe8e1f19080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:22:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:22:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:22:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:14.490 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:22:14.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:22:14.905 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34471,v1:127.0.0.1:6803/34471] [v2:127.0.0.1:6804/34471,v1:127.0.0.1:6805/34471] exists,up d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:22:14.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:14.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:14.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:14.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-01-15T16:22:14.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2021-01-15T16:22:14.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:22:14.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:22:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:22:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:22:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:22:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:22:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:22:14.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:22:14.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:22:14.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:22:14.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:22:14.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:22:14.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:22:14.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:22:14.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:22:14.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:22:14.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:22:14.910 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:22:14.910 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:14.910 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:14.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:14.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:22:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:22:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:22:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:22:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:22:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:22:14.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:22:14.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:22:14.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 66bb123c-6411-4ea2-9bd5-9eb095d2cddb' 2021-01-15T16:22:14.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:22:14.914 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:22:14.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA2wQFgzcIZNxAA3sqgkC2eRCEnd/2PDws3IA== 2021-01-15T16:22:14.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA2wQFgzcIZNxAA3sqgkC2eRCEnd/2PDws3IA=="}' 2021-01-15T16:22:14.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 66bb123c-6411-4ea2-9bd5-9eb095d2cddb -i td/divergent-priors/1/new.json 2021-01-15T16:22:15.415 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:22:15.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2021-01-15T16:22:15.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA2wQFgzcIZNxAA3sqgkC2eRCEnd/2PDws3IA== --osd-uuid 66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:22:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.456+0000 7f1db57f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:15.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.458+0000 7f1db57f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.459+0000 7f1db57f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.460+0000 7f1db57f5080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:22:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.460+0000 7f1db57f5080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:22:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.460+0000 7f1db57f5080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:22:15.530 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:15.530+0000 7f1db57f5080 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-01-15T16:22:19.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-01-15T16:22:19.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:22:19.688 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:22:19.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:22:19.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:22:20.216 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:22:20.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:22:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:22:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:22:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:22:20.244 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:20.243+0000 7f07bc9c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:20.270 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:20.269+0000 7f07bc9c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:20.271 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:20.271+0000 7f07bc9c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:20.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:22:20.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:22:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:20.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:22:20.618 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:20.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:20.788+0000 7f07bc9c8080 -1 Falling back to public interface 2021-01-15T16:22:21.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:22:21.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:21.636+0000 7f07bc9c8080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:22:22.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:22:22.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:22.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:22:22.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:22.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:22:22.059 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:22:22.472 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/35735,v1:127.0.0.1:6811/35735] [v2:127.0.0.1:6812/35735,v1:127.0.0.1:6813/35735] exists,up 66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:22:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2021-01-15T16:22:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2021-01-15T16:22:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:22:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:22:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:22:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:22:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:22:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:22:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:22:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:22:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:22:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:22:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:22:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:22:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:22:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:22:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:22:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:22:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:22:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:22:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:22:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:22:22.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:22:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:22:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:22:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:22:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:22:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:22:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:22:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:22:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:22:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:22:22.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:22:22.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:22:22.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:22:22.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 42a32ee4-c9a8-4952-b91b-7f19d23416f3' 2021-01-15T16:22:22.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:22:22.481 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:22:22.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA+wQFg8HdiHRAADKfxywNQ7UNd9RdwWjcp9Q== 2021-01-15T16:22:22.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA+wQFg8HdiHRAADKfxywNQ7UNd9RdwWjcp9Q=="}' 2021-01-15T16:22:22.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 42a32ee4-c9a8-4952-b91b-7f19d23416f3 -i td/divergent-priors/2/new.json 2021-01-15T16:22:23.086 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:22:23.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2021-01-15T16:22:23.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA+wQFg8HdiHRAADKfxywNQ7UNd9RdwWjcp9Q== --osd-uuid 42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:22:23.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.128+0000 7fb0ee9ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:23.129 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.129+0000 7fb0ee9ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:23.130 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.130+0000 7fb0ee9ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:23.131 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.131+0000 7fb0ee9ad080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:22:23.131 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.131+0000 7fb0ee9ad080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:22:23.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.131+0000 7fb0ee9ad080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:22:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:23.176+0000 7fb0ee9ad080 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-01-15T16:22:27.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-01-15T16:22:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:22:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:22:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:22:27.962 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:22:28.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:22:28.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:22:28.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:22:28.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:22:28.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:22:28.495 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:22:28.523 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:28.522+0000 7f7e8954f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:28.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:28.552+0000 7f7e8954f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:28.553 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:28.553+0000 7f7e8954f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:22:28.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:22:28.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:22:28.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:22:28.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:22:28.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:22:28.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:28.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:22:28.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:28.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:22:28.895 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:22:29.072 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:29.070+0000 7f7e8954f080 -1 Falling back to public interface 2021-01-15T16:22:29.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:22:29.896 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:22:29.896+0000 7f7e8954f080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:22:30.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:22:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:22:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:22:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:22:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:22:30.325 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:22:30.730 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/36999,v1:127.0.0.1:6819/36999] [v2:127.0.0.1:6820/36999,v1:127.0.0.1:6821/36999] exists,up 42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:22:30.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:22:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:22:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:22:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2021-01-15T16:22:31.538 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:22:31.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2021-01-15T16:22:32.299 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2021-01-15T16:22:32.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2021-01-15T16:22:33.067 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2021-01-15T16:22:33.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2021-01-15T16:22:33.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:22:33.755 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:22:33.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:22:34.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2021-01-15T16:22:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T16:22:35.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2021-01-15T16:22:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T16:22:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2021-01-15T16:22:36.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:22:36.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:22:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:22:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:22:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:22:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:22:36.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:36.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:22:36.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:22:36.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:22:36.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:22:36.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:36.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:22:36.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T16:22:36.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T16:22:36.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T16:22:36.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:36.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:22:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:22:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:22:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2021-01-15T16:22:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:37.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:22:37.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:22:37.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:22:37.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:22:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:22:37.053 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:22:37.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:22:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T16:22:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:37.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T16:22:37.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:22:37.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T16:22:37.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:37.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T16:22:37.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T16:22:37.475 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T16:22:37.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:22:37.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672964 -lt 42949672965 2021-01-15T16:22:37.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:22:38.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:22:38.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:22:39.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2021-01-15T16:22:39.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:39.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:22:39.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:39.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:22:39.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:22:39.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:39.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:22:39.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:22:39.281 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:22:39.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:22:39.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T16:22:39.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2021-01-15T16:22:39.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:22:39.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:22:39.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:22:39.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:22:39.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:22:39.677 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:22:39.677 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:22:39.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:22:39.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:22:39.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:22:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:22:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:22:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:22:39.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:22:39.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:22:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:22:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:22:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:22:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:22:40.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:40.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:22:40.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2021-01-15T16:22:40.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2021-01-15T16:22:40.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2021-01-15T16:22:40.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:40.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:22:40.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T16:22:40.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T16:22:40.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2021-01-15T16:22:40.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:40.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:22:40.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T16:22:40.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T16:22:40.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2021-01-15T16:22:40.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:40.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2021-01-15T16:22:40.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:22:40.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2021-01-15T16:22:40.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2021-01-15T16:22:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-01-15T16:22:40.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:22:40.606 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2021-01-15T16:22:41.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2021-01-15T16:22:41.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:22:42.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:22:42.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:22:42.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2021-01-15T16:22:42.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:42.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T16:22:42.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:42.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:22:42.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T16:22:42.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:42.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T16:22:42.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T16:22:42.427 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T16:22:42.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:22:42.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T16:22:42.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:42.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T16:22:42.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:42.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:22:42.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T16:22:42.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:42.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T16:22:42.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T16:22:42.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:22:42.828 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T16:22:43.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509446 2021-01-15T16:22:43.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:22:43.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:22:43.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:22:43.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:22:43.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:22:43.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:22:43.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:22:43.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:22:43.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:22:43.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:22:43.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:22:44.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:22:44.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:22:44.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:22:44.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:22:44.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:22:44.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:22:44.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:22:44.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2021-01-15T16:22:44.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2021-01-15T16:22:45.050 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:22:45.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2021-01-15T16:22:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2021-01-15T16:22:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2021-01-15T16:22:45.062 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2021-01-15T16:22:45.430 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:22:45.431 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:22:34.186460+0000 0'0 26:11 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:22:45.431 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:22:45.431 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:22:45.432 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:22:45.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2021-01-15T16:22:45.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-01-15T16:22:45.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2021-01-15T16:22:45.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2021-01-15T16:22:45.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-01-15T16:22:45.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2021-01-15T16:22:45.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2021-01-15T16:22:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2021-01-15T16:22:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2021-01-15T16:22:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2021-01-15T16:22:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2021-01-15T16:22:45.444 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2021-01-15T16:22:45.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2021-01-15T16:22:45.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:45.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2021-01-15T16:22:45.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:45.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2021-01-15T16:22:45.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:45.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2021-01-15T16:22:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2021-01-15T16:22:45.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:45.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2021-01-15T16:22:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2021-01-15T16:22:46.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2021-01-15T16:22:46.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2021-01-15T16:22:46.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2021-01-15T16:22:46.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2021-01-15T16:22:46.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2021-01-15T16:22:46.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2021-01-15T16:22:46.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:46.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2021-01-15T16:22:47.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2021-01-15T16:22:47.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2021-01-15T16:22:47.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2021-01-15T16:22:47.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2021-01-15T16:22:47.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2021-01-15T16:22:47.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2021-01-15T16:22:47.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2021-01-15T16:22:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2021-01-15T16:22:48.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2021-01-15T16:22:48.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2021-01-15T16:22:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2021-01-15T16:22:48.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2021-01-15T16:22:48.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2021-01-15T16:22:48.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2021-01-15T16:22:48.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2021-01-15T16:22:48.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:48.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2021-01-15T16:22:49.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2021-01-15T16:22:49.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2021-01-15T16:22:49.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2021-01-15T16:22:49.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2021-01-15T16:22:49.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2021-01-15T16:22:49.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2021-01-15T16:22:49.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2021-01-15T16:22:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2021-01-15T16:22:50.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2021-01-15T16:22:50.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2021-01-15T16:22:50.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2021-01-15T16:22:50.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2021-01-15T16:22:50.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2021-01-15T16:22:50.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2021-01-15T16:22:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2021-01-15T16:22:50.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:50.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2021-01-15T16:22:51.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2021-01-15T16:22:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2021-01-15T16:22:51.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2021-01-15T16:22:51.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2021-01-15T16:22:51.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2021-01-15T16:22:51.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2021-01-15T16:22:51.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:51.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2021-01-15T16:22:52.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2021-01-15T16:22:52.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2021-01-15T16:22:52.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2021-01-15T16:22:52.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2021-01-15T16:22:52.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2021-01-15T16:22:52.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2021-01-15T16:22:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2021-01-15T16:22:52.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:52.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2021-01-15T16:22:53.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2021-01-15T16:22:53.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2021-01-15T16:22:53.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2021-01-15T16:22:53.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2021-01-15T16:22:53.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2021-01-15T16:22:53.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2021-01-15T16:22:53.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:53.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2021-01-15T16:22:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2021-01-15T16:22:54.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2021-01-15T16:22:54.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2021-01-15T16:22:54.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2021-01-15T16:22:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2021-01-15T16:22:54.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2021-01-15T16:22:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2021-01-15T16:22:54.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:54.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2021-01-15T16:22:55.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2021-01-15T16:22:55.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2021-01-15T16:22:55.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2021-01-15T16:22:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2021-01-15T16:22:55.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2021-01-15T16:22:55.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2021-01-15T16:22:55.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2021-01-15T16:22:55.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:55.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2021-01-15T16:22:56.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2021-01-15T16:22:56.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2021-01-15T16:22:56.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2021-01-15T16:22:56.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2021-01-15T16:22:56.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2021-01-15T16:22:56.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2021-01-15T16:22:56.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2021-01-15T16:22:56.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2021-01-15T16:22:56.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:56.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2021-01-15T16:22:57.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2021-01-15T16:22:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2021-01-15T16:22:57.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2021-01-15T16:22:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2021-01-15T16:22:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2021-01-15T16:22:57.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2021-01-15T16:22:57.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2021-01-15T16:22:57.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2021-01-15T16:22:57.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-01-15T16:22:58.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:22:58.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2021-01-15T16:22:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:22:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:22:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:22:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:22:58.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:22:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:22:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:22:58.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:22:58.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:22:58.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:22:58.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:22:58.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:22:58.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:22:58.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:22:58.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:22:58.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:22:58.572 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:22:58.572 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:22:58.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:22:58.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:58.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:22:58.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T16:22:58.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T16:22:58.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T16:22:58.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:58.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:22:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:22:58.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:22:58.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T16:22:58.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:22:58.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:22:59.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T16:22:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T16:22:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T16:22:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:59.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T16:22:59.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:22:59.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T16:22:59.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:59.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T16:22:59.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T16:22:59.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:22:59.024 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T16:22:59.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2021-01-15T16:22:59.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:59.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:22:59.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:59.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:22:59.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:22:59.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:59.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:22:59.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:22:59.431 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:22:59.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:22:59.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T16:22:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:22:59.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T16:22:59.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:22:59.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:22:59.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T16:22:59.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:22:59.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T16:22:59.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T16:22:59.843 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T16:22:59.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:23:00.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509452 2021-01-15T16:23:00.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:23:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:23:01.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:23:01.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T16:23:01.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:23:01.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:01.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:02.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:23:02.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:02.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:02.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:02.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:02.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:02.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:02.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:02.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:23:02.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:02.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:02.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:03.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:23:03.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:23:03.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:23:03.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2021-01-15T16:23:03.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T16:23:03.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2021-01-15T16:23:03.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2021-01-15T16:23:03.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T16:23:03.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2021-01-15T16:23:03.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2021-01-15T16:23:03.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2021-01-15T16:23:03.468 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2021-01-15T16:23:03.847 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:23:03.848 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-01-15T16:22:34.186460+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:23:03.848 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:23:03.848 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:23:03.849 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:23:03.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:03.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2021-01-15T16:23:03.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:23:03.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:23:03.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:23:03.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:03.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:03.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:23:03.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-01-15T16:23:03.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok config set objectstore_blackhole 1 2021-01-15T16:23:03.959 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:23:03.988 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:23:03.988 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:23:03.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:03.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2021-01-15T16:23:03.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:23:03.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:23:03.991 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:23:03.991 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:03.991 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:03.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:23:03.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2021-01-15T16:23:03.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok config set objectstore_blackhole 1 2021-01-15T16:23:04.066 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:23:04.067 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:23:04.067 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:23:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2021-01-15T16:23:04.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2021-01-15T16:23:04.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2021-01-15T16:23:04.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2021-01-15T16:23:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2021-01-15T16:23:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2021-01-15T16:23:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2021-01-15T16:23:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T16:23:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T16:23:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T16:23:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T16:23:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test 2021-01-15T16:23:04.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T16:23:04.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=existing_99 2021-01-15T16:23:04.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T16:23:04.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/divergent-priors 2021-01-15T16:23:04.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T16:23:04.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T16:23:04.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T16:23:04.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T16:23:04.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test existing_99 2021-01-15T16:23:04.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2021-01-15T16:23:04.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=existing_99 2021-01-15T16:23:04.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test existing_99 2021-01-15T16:23:04.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T16:23:04.085 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent object 2021-01-15T16:23:04.085 INFO:tasks.workunit.client.0.smithi145.stdout:create missing divergent object 2021-01-15T16:23:04.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2021-01-15T16:23:04.481 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T16:23:04.482 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:23:04.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2021-01-15T16:23:04.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T16:23:04.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T16:23:04.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T16:23:04.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T16:23:04.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/divergent-priors/1/type 2021-01-15T16:23:04.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T16:23:04.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T16:23:04.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T16:23:04.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T16:23:04.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T16:23:04.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T16:23:04.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T16:23:04.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T16:23:04.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:23:04.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:23:04.488 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:23:04.488 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:04.489 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:04.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:23:04.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.33425/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T16:23:04.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T16:23:04.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T16:23:04.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T16:23:04.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T16:23:04.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:23:04.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:23:04.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:23:04.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:04.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:04.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:23:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T16:23:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok injectdataerr test existing_99 2021-01-15T16:23:04.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2021-01-15T16:23:04.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2021-01-15T16:23:14.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2021-01-15T16:23:14.717 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2021-01-15T16:23:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2021-01-15T16:23:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2021-01-15T16:23:14.718 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2021-01-15T16:23:15.100 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:23:15.100 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2021-01-15T16:23:05.071600+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:23:15.101 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:23:15.101 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:23:15.102 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:23:15.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2021-01-15T16:23:15.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:23:15.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:23:15.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:23:15.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:23:15.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:23:15.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:23:15.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-01-15T16:23:15.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2021-01-15T16:23:15.750 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T16:23:15.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-01-15T16:23:15.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2021-01-15T16:23:16.304 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T16:23:16.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2021-01-15T16:23:16.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2021-01-15T16:23:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T16:23:16.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:16.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2021-01-15T16:23:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T16:23:17.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:17.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2021-01-15T16:23:17.961 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T16:23:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2021-01-15T16:23:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2021-01-15T16:23:17.971 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2021-01-15T16:23:18.344 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:23:18.344 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2021-01-15T16:23:05.071600+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:23:18.344 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:23:18.344 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:23:18.345 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:23:18.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:18.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2021-01-15T16:23:18.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:23:18.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:23:18.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:23:18.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:23:18.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:23:18.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:23:18.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:23:18.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:23:18.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:23:18.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:23:18.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:23:18.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:23:18.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:23:18.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:23:18.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:23:18.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:23:18.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:23:18.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:23:18.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:18.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:18.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:23:18.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:23:18.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:23:18.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:23:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:23:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:23:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:23:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:23:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:23:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:23:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:23:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:23:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:23:18.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2021-01-15T16:23:18.368 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:23:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:23:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:23:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:23:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:23:18.387 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:18.387+0000 7f0df95b4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:18.389 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:18.389+0000 7f0df95b4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:18.391 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:18.390+0000 7f0df95b4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:18.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:23:18.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:23:18.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:23:18.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:23:18.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:23:18.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:23:18.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:23:18.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:23:18.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:23:18.759 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:23:18.908 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:18.908+0000 7f0df95b4080 -1 Falling back to public interface 2021-01-15T16:23:19.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:23:20.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:20.108+0000 7f0df95b4080 -1 osd.0 29 log_to_monitors {default=true} 2021-01-15T16:23:20.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:23:20.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:23:20.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:23:20.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:23:20.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:23:20.165 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:23:20.581 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/40979,v1:127.0.0.1:6803/40979] [v2:127.0.0.1:6804/40979,v1:127.0.0.1:6805/40979] exists,up d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:23:20.582 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:23:20.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:23:20.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:23:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:23:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2021-01-15T16:23:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:23:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:23:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:23:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:23:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:23:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:23:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:23:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:23:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:23:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:23:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:23:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:23:20.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:23:20.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:23:20.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:23:20.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:23:20.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:23:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:23:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:20.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:20.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:23:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:23:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:23:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:23:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:23:20.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:23:20.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:23:20.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:23:20.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:23:20.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:23:20.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:23:20.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:23:20.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:23:20.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2021-01-15T16:23:20.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:23:20.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:23:20.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:23:20.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:23:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:20.610+0000 7f26d5706080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:20.611 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:20.611+0000 7f26d5706080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:20.613 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:20.612+0000 7f26d5706080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:23:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:23:20.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:23:20.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:23:20.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:23:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:23:20.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:23:20.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:23:20.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:23:20.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:23:20.995 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:23:21.133 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:21.132+0000 7f26d5706080 -1 Falling back to public interface 2021-01-15T16:23:21.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:23:22.323 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:23:22.322+0000 7f26d5706080 -1 osd.2 29 log_to_monitors {default=true} 2021-01-15T16:23:22.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:23:22.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:23:22.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:23:22.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:23:22.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:23:22.389 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:23:22.809 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/41468,v1:127.0.0.1:6811/41468] [v2:127.0.0.1:6812/41468,v1:127.0.0.1:6813/41468] exists,up 42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:23:22.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:23:22.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:23:22.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:23:22.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2021-01-15T16:23:23.376 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T16:23:23.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:23.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2021-01-15T16:23:23.912 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2021-01-15T16:23:23.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:23:23.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2021-01-15T16:23:23.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:23:23.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:23:23.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:23:23.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:23:23.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:23:23.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:23:23.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:23:23.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:23:23.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:23:23.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:23:23.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:23:23.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:23:23.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:23:23.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:23:23.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:23:24.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:23:24.438 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:23:24.438 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:23:24.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:23:24.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:23:24.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:23:24.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2021-01-15T16:23:24.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2021-01-15T16:23:24.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855363' 2021-01-15T16:23:24.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:23:24.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:23:24.682 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:23:24.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:23:24.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:23:24.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:23:24.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:23:24.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:23:24.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2021-01-15T16:23:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2021-01-15T16:23:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855363 2-154618822659' 2021-01-15T16:23:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:23:24.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855363 2021-01-15T16:23:24.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:23:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:23:24.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855363 2021-01-15T16:23:24.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:23:24.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2021-01-15T16:23:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2021-01-15T16:23:24.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:23:24.829 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 150323855363 2021-01-15T16:23:25.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855363 -lt 150323855363 2021-01-15T16:23:25.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:23:25.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822659 2021-01-15T16:23:25.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:23:25.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:23:25.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2021-01-15T16:23:25.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:23:25.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2021-01-15T16:23:25.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2021-01-15T16:23:25.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:23:25.229 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 154618822659 2021-01-15T16:23:25.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822659 -lt 154618822659 2021-01-15T16:23:25.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:23:25.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:25.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:26.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:23:26.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:26.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:26.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:26.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:26.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:26.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:26.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:26.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:26.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:26.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:26.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:27.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:23:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:23:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:23:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:27.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:23:27.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:27.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:27.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:27.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:27.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:27.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:27.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:27.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:27.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:27.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:27.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:27.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:28.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:28.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:28.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:28.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:28.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:28.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:28.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:28.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:28.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:28.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:28.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:28.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:23:28.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:28.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:23:28.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:28.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:28.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:28.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:28.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:28.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:28.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:28.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:29.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:29.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:29.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:29.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:29.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:29.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:29.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:29.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:30.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:30.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:30.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:23:30.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:30.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:23:30.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:30.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:30.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:30.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:30.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:30.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:30.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:30.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:31.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:31.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:31.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:31.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:31.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:31.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:31.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:31.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:31.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:31.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:32.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:23:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:23:32.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:32.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:32.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:32.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:32.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:32.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:32.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:32.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:33.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:33.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:33.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:33.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:33.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:33.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:33.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:33.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:33.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:33.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:33.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:33.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:33.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:34.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:34.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:34.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:23:34.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:34.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:23:35.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:35.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:35.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:35.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:35.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:35.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:35.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:35.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:36.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:36.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:36.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:36.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:36.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:36.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:36.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:36.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:36.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:37.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:23:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:23:40.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:40.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:40.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:40.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:40.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:40.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:40.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:40.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:40.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:40.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:40.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:40.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:41.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:41.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:41.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:41.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:41.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:41.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:41.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:41.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:41.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:42.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:23:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:23:48.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:48.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:48.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:48.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:48.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:48.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:48.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:48.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:48.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:48.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:48.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:48.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:49.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:49.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:49.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:49.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:49.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:49.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:49.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:49.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:49.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:49.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:49.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:49.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:23:49.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:23:49.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:23:57.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:23:57.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:23:57.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:23:57.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:23:57.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:23:57.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:23:57.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:23:57.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:23:57.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:23:57.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:23:57.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:23:57.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:23:58.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:23:58.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:23:58.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:23:58.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:23:58.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:23:58.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:23:58.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:23:58.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:23:58.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:23:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:23:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:23:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:23:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:23:59.099 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:23:59.100 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "da92a0d1-bd94-4686-909d-49b274774111", 2021-01-15T16:23:59.100 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:23:59.100 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:23:59.100 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:23:59.098418+0000", 2021-01-15T16:23:59.101 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:23:59.101 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:23:59.101 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:23:59.101 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:23:59.102 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:23:59.102 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:23:59.102 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:23:59.102 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:23:59.103 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:23:59.103 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.103 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:23:59.103 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:23:59.104 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.104 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:23:59.104 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:23:59.104 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:23:59.104 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:23:59.105 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:23:59.105 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.105 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:23:59.105 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.105 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:23:59.106 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.106 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.106 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:23:59.106 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:23:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:23:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:23:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-01-15T16:23:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.108 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.109 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:23:59.109 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.109 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:23:59.109 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:23:59.109 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:23:59.110 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:23:59.110 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:23:59.110 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.110 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:23:59.110 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.111 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:23:59.111 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.111 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:23:59.111 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.112 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.112 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:23:59.112 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.112 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.112 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:23:59.113 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.113 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:23:59.113 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:23:59.113 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:23:59.113 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:23:59.114 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:23:59.114 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:22:06.128762Z", 2021-01-15T16:23:59.114 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:22:06.128762Z", 2021-01-15T16:23:59.114 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:23:59.114 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:23:59.115 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:23:59.116 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:23:59.116 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:23:59.116 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:23:59.116 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:23:59.116 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:23:59.117 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:23:59.117 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:23:59.117 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.117 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:23:59.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.118 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:23:59.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.118 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:23:59.118 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:23:59.118 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:23:59.119 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.119 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.119 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.119 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:23:59.119 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:23:59.120 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:23:59.121 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:23:59.121 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:23:59.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.121 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.121 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.122 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:23:59.122 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:23:59.122 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.122 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:23:59.122 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40, 2021-01-15T16:23:59.123 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:23:59.123 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:22:06.711717+0000", 2021-01-15T16:23:59.123 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:23:23.994035+0000", 2021-01-15T16:23:59.123 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:23:22.403931+0000", 2021-01-15T16:23:59.123 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:23:23.735718+0000", 2021-01-15T16:23:59.124 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:23:59.124 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:23:59.124 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:23:59.124 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:23:59.124 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:23:59.125 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.126 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2021-01-15T16:23:59.126 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:23:59.126 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:23:59.126 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:23:59.126 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:23:59.127 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:23:59.127 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:23:59.127 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:23:59.127 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:23:59.127 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:23:59.128 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:23:59.128 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.128 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:23:59.128 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:23:59.128 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:22:33.371494+0000", 2021-01-15T16:23:59.129 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T16:23:59.129 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T16:23:59.129 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:23:59.129 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:23:59.129 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:23:59.130 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T16:23:59.130 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:23:59.130 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:23:59.130 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:23:59.130 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T16:23:59.131 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T16:23:59.132 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T16:23:59.132 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:23:59.132 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:23:59.132 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:23:59.132 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:23:59.133 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:23:59.133 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:23:59.133 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:23:59.133 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.133 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2021-01-15T16:23:59.134 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:23:59.134 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:23:59.134 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T16:23:59.134 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:23:59.134 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T16:23:59.135 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T16:23:59.135 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T16:23:59.135 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T16:23:59.135 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:23:59.135 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:23:59.136 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:23:59.136 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:23:59.136 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:23:59.136 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:23:59.136 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:23:59.137 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:23:59.137 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:23:59.137 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:23:59.137 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:23:59.137 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:23:59.138 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:23:59.138 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:23:59.138 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:23:59.138 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T16:23:59.138 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:23:59.139 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:23:59.140 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:23:59.140 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:23:59.140 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:23:59.140 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:23:59.140 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T16:23:59.141 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:23:59.141 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:23:59.141 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:23:59.141 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:23:59.141 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.142 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.142 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:23:59.142 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.142 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:23:59.142 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "d3275fea-a12c-420f-90e6-0ac2fa60dc87", 2021-01-15T16:23:59.143 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:23:59.143 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:23:59.143 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:23:59.143 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:23:59.143 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 39, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:23:59.144 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:23:59.145 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.145 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.145 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.145 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:23:59.145 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.146 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.146 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.146 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.146 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:23:59.146 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.147 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.147 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.147 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.147 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:23:59.147 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.148 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.148 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.148 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:23:59.148 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.148 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.149 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.150 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.150 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:23:59.150 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.150 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.150 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.151 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:23:59.151 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.151 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.151 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.151 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.152 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:23:59.152 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.152 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.152 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.152 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.153 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:23:59.153 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.153 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.153 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.153 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:23:59.154 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.154 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.154 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.154 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.154 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 40979 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/40979", 2021-01-15T16:23:59.155 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/40979", 2021-01-15T16:23:59.156 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/40979", 2021-01-15T16:23:59.156 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/40979", 2021-01-15T16:23:59.156 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:23:59.156 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:23:59.156 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:23:59.157 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.157 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.157 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.157 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:23:59.157 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "66bb123c-6411-4ea2-9bd5-9eb095d2cddb", 2021-01-15T16:23:59.158 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:23:59.158 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:23:59.158 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:23:59.158 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:23:59.158 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 23, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:23:59.159 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:23:59.160 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.160 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.160 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.160 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:23:59.160 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.161 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.161 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.161 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.161 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:23:59.161 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.162 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.162 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.162 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.162 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:23:59.162 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.163 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.163 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.163 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:23:59.163 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.163 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.164 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.165 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.165 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:23:59.165 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.165 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.165 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.166 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:23:59.166 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.166 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.166 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.166 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.167 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:23:59.167 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.167 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.167 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.167 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:23:59.168 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.169 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.169 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.169 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.169 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:23:59.169 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 35735 2021-01-15T16:23:59.170 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.170 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.170 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.170 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/35735", 2021-01-15T16:23:59.170 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/35735", 2021-01-15T16:23:59.171 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/35735", 2021-01-15T16:23:59.171 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/35735", 2021-01-15T16:23:59.171 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:23:59.171 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:23:59.171 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.172 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.172 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.172 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:23:59.172 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "42a32ee4-c9a8-4952-b91b-7f19d23416f3", 2021-01-15T16:23:59.172 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:23:59.173 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2021-01-15T16:23:59.174 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:23:59.174 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2021-01-15T16:23:59.174 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:23:59.174 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:23:59.174 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.175 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.175 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.175 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:23:59.175 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.175 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.176 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.176 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.176 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:23:59.176 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.176 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.177 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.177 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.177 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:23:59.177 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.177 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.178 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.178 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:23:59.178 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.178 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.178 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.179 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.180 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:23:59.180 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.180 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.180 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.180 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:23:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:23:59.182 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.182 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.182 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.182 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.182 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:23:59.183 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:23:59.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.183 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:23:59.183 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:23:59.183 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 41468 2021-01-15T16:23:59.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.185 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.185 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/41468", 2021-01-15T16:23:59.185 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/41468", 2021-01-15T16:23:59.185 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/41468", 2021-01-15T16:23:59.186 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/41468", 2021-01-15T16:23:59.186 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:23:59.186 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:23:59.186 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:23:59.186 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.187 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.187 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.187 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:23:59.187 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.187 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:23:59.188 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:23:15.553579+0000", 2021-01-15T16:23:59.188 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:23:59.188 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:23:59.188 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:23:59.188 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:22:14.974367+0000", 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:23:59.189 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:23:16.098687+0000", 2021-01-15T16:23:59.190 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:23:59.190 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:23:59.190 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:23:59.190 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:23:59.190 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:22:22.616294+0000", 2021-01-15T16:23:59.191 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:23:59.191 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.191 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.191 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:23:59.191 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:23:16.649962+0000", 2021-01-15T16:23:59.192 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:23:59.192 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:23:59.192 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:23:59.192 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:23:59.192 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:22:30.869661+0000", 2021-01-15T16:23:59.193 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:23:59.193 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.193 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.193 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:23:59.193 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:23:59.194 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:23:59.194 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:23:59.194 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:23:59.194 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:23:59.194 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:23:59.195 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:23:59.195 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:23:59.195 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:23:59.195 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:23:59.195 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:23:59.196 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:23:59.197 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:23:59.197 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:23:59.197 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:23:59.197 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:23:59.197 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:23:59.198 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:23:59.198 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.198 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.198 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:23:59.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.199 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:23:59.199 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:23:59.199 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/40979,v1:127.0.0.1:6805/40979]", 2021-01-15T16:23:59.199 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:23:59.199 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:23:59.200 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:23:59.200 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:23:59.200 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:23:59.200 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:23:59.200 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:23:59.201 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:23:59.202 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:23:59.202 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:23:59.202 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:23:59.202 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:23:59.202 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:23:59.203 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:23:59.204 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:23:59.204 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:23:59.204 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:23:59.204 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:23:59.205 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:23:59.205 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:23:59.205 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/40979,v1:127.0.0.1:6803/40979]", 2021-01-15T16:23:59.205 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:23:59.205 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/40979,v1:127.0.0.1:6809/40979]", 2021-01-15T16:23:59.206 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/40979,v1:127.0.0.1:6807/40979]", 2021-01-15T16:23:59.206 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:23:59.206 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:23:59.206 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:23:59.207 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:23:59.207 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:23:59.207 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:23:59.207 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:23:59.208 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:23:59.208 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:23:59.208 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:23:59.208 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:23:59.209 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:23:59.209 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:23:59.209 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.209 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.209 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:23:59.210 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:23:59.210 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/35735,v1:127.0.0.1:6813/35735]", 2021-01-15T16:23:59.210 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:23:59.210 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:23:59.210 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:23:59.211 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:23:59.211 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:23:59.211 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:23:59.211 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:23:59.211 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:23:59.212 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:23:59.212 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:23:59.212 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:23:59.212 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:23:59.212 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:23:59.213 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:23:59.213 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:23:59.213 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:23:59.213 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:23:59.214 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:23:59.214 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:23:59.214 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:23:59.214 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:23:59.215 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:23:59.215 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:23:59.215 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:23:59.215 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:23:59.215 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/35735,v1:127.0.0.1:6811/35735]", 2021-01-15T16:23:59.216 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:23:59.216 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/35735,v1:127.0.0.1:6817/35735]", 2021-01-15T16:23:59.216 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/35735,v1:127.0.0.1:6815/35735]", 2021-01-15T16:23:59.216 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:23:59.216 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:23:59.217 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:23:59.217 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:23:59.217 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:23:59.217 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:23:59.218 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:23:59.218 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:23:59.218 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:23:59.218 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:23:59.219 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:23:59.219 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:23:59.219 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:23:59.219 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.220 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.220 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:23:59.220 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:23:59.220 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/41468,v1:127.0.0.1:6813/41468]", 2021-01-15T16:23:59.221 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:23:59.221 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:23:59.221 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:23:59.222 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:23:59.222 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:23:59.222 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:23:59.222 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:23:59.223 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:23:59.223 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:23:59.223 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:23:59.223 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:23:59.223 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:23:59.224 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:23:59.224 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:23:59.224 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:23:59.225 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:23:59.225 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:23:59.225 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:23:59.225 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:23:59.225 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:23:59.226 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:23:59.226 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:23:59.226 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:23:59.226 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:23:59.226 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:23:59.227 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/41468,v1:127.0.0.1:6811/41468]", 2021-01-15T16:23:59.227 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:23:59.227 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/41468,v1:127.0.0.1:6817/41468]", 2021-01-15T16:23:59.227 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/41468,v1:127.0.0.1:6815/41468]", 2021-01-15T16:23:59.228 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:23:59.228 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:23:59.228 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:23:59.228 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:23:59.228 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:23:59.229 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:23:59.229 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:23:59.229 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:23:59.229 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:23:59.230 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:23:59.230 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:23:59.230 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:23:59.230 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:23:59.231 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.231 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.231 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:23:59.231 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:23:59.231 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:23:59.232 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:23:59.232 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.232 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:23:59.232 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.233 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:23:59.233 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 35 2021-01-15T16:23:59.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.233 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.233 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:23:59.234 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:23:59.234 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.234 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.234 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:23:59.235 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 36 2021-01-15T16:23:59.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.235 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.235 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:23:59.236 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 40, 2021-01-15T16:23:59.236 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:23:59.236 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:23:59.236 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.236 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:23:59.237 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:23:59.237 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:23:59.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.237 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:23:59.238 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:23:59.238 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:23:59.238 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.238 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.239 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:23:59.239 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:23:59.239 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:23:59.239 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.240 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.240 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:23:59.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.240 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:23:59.240 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:23:59.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.241 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:23:59.241 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:23:59.242 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.242 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.242 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:23:59.242 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:23:59.243 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.243 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.243 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:23:59.243 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:23:59.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.244 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.244 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:23:59.244 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:23:59.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.245 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:23:59.246 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:23:59.246 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.246 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.246 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:23:59.246 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.248 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:23:59.248 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:23:59.248 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.248 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.248 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:23:59.249 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:23:59.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.249 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:23:59.249 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:23:59.250 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.250 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:23:59.250 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:23:59.250 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.251 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.251 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:23:59.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.251 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:23:59.251 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:23:59.252 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:23:59.253 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.253 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:23:59.253 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.253 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:23:59.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.254 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.254 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.254 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.254 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:23:59.254 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:23:59.255 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:23:59.255 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:23:59.255 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.255 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:23:59.255 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:23:59.256 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.257 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.257 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.257 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.257 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:23:59.257 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:23:59.258 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:23:59.258 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:23:59.258 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.258 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:23:59.258 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:23:59.259 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:23:59.259 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.259 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:23:59.259 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.259 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:23:59.260 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.260 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.260 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:23:59.260 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.260 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:23:59.261 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.262 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.262 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.262 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.262 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:23:59.262 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:23:59.263 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:23:59.263 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:23:59.263 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:23:59.263 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:23:59.263 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:23:59.264 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:23:59.264 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.264 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:23:59.264 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.264 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:23:59.265 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.265 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.265 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:23:59.265 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.265 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:23:59.266 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.266 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.266 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:23:59.267 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:23:59.267 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:23:59.267 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.267 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.267 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.268 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.268 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:23:59.268 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.268 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:23:59.268 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:23:59.269 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.270 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:23:59.270 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:23:59.270 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:23:59.270 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.270 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.271 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:23:59.271 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:23:59.271 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:23:59.271 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.271 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.272 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:23:59.272 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.272 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:23:59.272 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:23:59.273 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:23:59.274 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:23:59.274 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:23:59.274 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:23:59.274 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:23:59.274 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:23:59.275 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:23:59.275 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:23:59.275 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:23:59.275 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:23:59.275 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:23:59.276 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T16:23:59.276 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:23:59.276 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:23:59.276 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:23:59.276 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:23:59.277 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:23:59.278 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:23:59.278 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:23:59.278 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:23:59.278 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:23:59.278 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:23:59.279 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:23:59.279 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:23:59.279 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:23:59.279 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:23:59.279 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:23:59.280 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:23:59.280 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:23:59.280 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:23:59.280 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:23:59.280 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.281 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:23:59.282 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:23:59.282 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.282 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:23:59.282 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:23:59.282 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:23:59.283 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:23:59.283 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:23:59.283 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:23:59.283 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.283 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:23:59.284 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2021-01-15T16:23:59.284 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:23:59.284 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:23:59.284 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:23:59.284 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:23:59.285 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:23:59.285 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:23:59.285 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:23:59.285 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:23:59.285 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:23:59.286 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:23:59.286 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:23:59.286 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:23:59.286 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:23:59.286 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:23:59.287 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:23:59.287 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:23:59.287 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:23:59.287 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:23:59.287 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:23:59.288 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:23:59.289 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:23:59.289 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:23:59.289 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:23:59.289 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:23:59.289 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:23:59.290 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:23:59.290 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:23:59.290 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:23:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:23:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:23:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:23:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:23:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:23:59.292 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:23:59.292 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:23:59.292 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:23:59.292 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:23:59.292 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:23:59.293 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:23:59.294 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:23:59.294 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:23:59.294 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:23:59.294 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:23:59.295 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:23:59.295 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:23:59.295 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:23:59.295 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.296 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2021-01-15T16:23:59.296 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2021-01-15T16:23:59.296 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:23:59.296 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:23:59.296 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:23:59.297 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.297 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:23:59.297 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:23:59.297 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:23:59.297 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18248, 2021-01-15T16:23:59.298 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2092, 2021-01-15T16:23:59.299 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:23:59.299 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16126, 2021-01-15T16:23:59.299 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554552, 2021-01-15T16:23:59.299 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:23:59.299 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:23:59.300 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103861248, 2021-01-15T16:23:59.300 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:23:59.300 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2142208, 2021-01-15T16:23:59.300 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 410198, 2021-01-15T16:23:59.300 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:23:59.301 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:23:59.301 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:23:59.301 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:23:59.301 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16513128 2021-01-15T16:23:59.301 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 1, 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:23:59.302 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:23:59.303 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:23:59.303 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.303 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:23:59.303 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:23:59.303 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:23:59.304 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:23:59.304 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:23:59.304 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.304 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:23:59.304 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:23:59.305 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.305 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:23:59.305 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:23:59.305 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:23:59.305 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:23:59.306 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:23:59.306 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:23:59.306 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:23:59.306 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:23:59.306 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:23:59.307 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18248, 2021-01-15T16:23:59.307 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2092, 2021-01-15T16:23:59.307 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:23:59.307 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16126, 2021-01-15T16:23:59.307 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554552, 2021-01-15T16:23:59.308 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:23:59.308 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:23:59.308 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103861248, 2021-01-15T16:23:59.308 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:23:59.309 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2142208, 2021-01-15T16:23:59.309 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 410198, 2021-01-15T16:23:59.309 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:23:59.309 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:23:59.309 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:23:59.310 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:23:59.310 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16513128 2021-01-15T16:23:59.310 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.310 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:23:59.310 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:23:59.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:23:59.311 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 1, 2021-01-15T16:23:59.311 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:23:59.311 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:23:59.311 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:23:59.312 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.312 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:23:59.312 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:23:59.312 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:23:59.312 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.313 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.314 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:23:59.314 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.314 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:23:59.314 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:23:59.314 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:23:59.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:23:59.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:23:59.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:23:59.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:23:59.315 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:23:59.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:23:59.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:23:59.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:23:59.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:23:59.316 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:23:59.317 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:23:59.317 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:23:59.317 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:23:59.317 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:23:59.317 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:23:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:23:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:23:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:23:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:23:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:23:59.319 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:23:59.320 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:23:59.320 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:23:59.320 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:23:59.320 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:23:59.320 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:23:59.321 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:23:59.321 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:23:59.321 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:23:59.321 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:23:59.321 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:23:59.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:23:59.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:23:59.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:23:59.322 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:23:59.322 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.323 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:23:59.323 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:23:59.323 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:23:59.323 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:23:59.323 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1228800, 2021-01-15T16:23:59.324 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 144900, 2021-01-15T16:23:59.324 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:23:59.324 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:23:59.324 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:23:59.324 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:23:59.325 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:23:59.326 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:23:59.326 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.326 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.326 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:23:59.326 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.327 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:23:59.327 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855370 2021-01-15T16:23:59.327 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.327 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.327 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:23:59.328 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672977 2021-01-15T16:23:59.328 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.328 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.328 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:23:59.328 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822666 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:23:59.329 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T16:23:59.330 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.330 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.330 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:23:59.330 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.331 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:23:59.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:23:59.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:23:59.331 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:23:59.331 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.332 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.332 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:23:59.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:23:59.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:23:59.332 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:23:59.333 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:23:59.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.334 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.334 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:23:59.334 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:23:59.335 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:23:59.335 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:23:59.335 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.335 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:23:59.335 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 4, 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:23:24.898383+0000", 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "services": { 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": { 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "daemons": { 2021-01-15T16:23:59.336 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": "", 2021-01-15T16:23:59.337 INFO:tasks.workunit.client.0.smithi145.stdout: "0": { 2021-01-15T16:23:59.337 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:23:59.337 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:23:59.337 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:23:59.337 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:23:59.338 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:23:59.338 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:23:59.338 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.338 INFO:tasks.workunit.client.0.smithi145.stdout: "2": { 2021-01-15T16:23:59.338 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:23:59.339 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:23:59.339 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:23:59.339 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:23:59.339 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:23:59.339 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:23:59.340 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.340 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.340 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.340 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.340 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 361, 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:23:59.341 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:23:59.342 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:23:59.343 INFO:tasks.workunit.client.0.smithi145.stderr:report 3336410462 2021-01-15T16:23:59.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:23:59.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2021-01-15T16:23:59.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2021-01-15T16:23:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2021-01-15T16:23:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2021-01-15T16:23:59.344 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2021-01-15T16:23:59.486 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:23:59.486 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-01-15T16:23:24.236715+0000 29'100 40:122 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:23:59.487 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:23:59.487 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:23:59.488 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:23:59.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2021-01-15T16:23:59.610 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2021-01-15T16:23:59.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2021-01-15T16:23:59.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2021-01-15T16:23:59.985 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:23:59.985 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-01-15T16:23:24.236715+0000 29'100 40:122 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:23:59.986 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:23:59.986 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:23:59.987 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:23:59.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-01-15T16:23:59.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2021-01-15T16:23:59.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:23:59.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:23:59.998 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:23:59.998 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:23:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:23:59.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:23:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-01-15T16:23:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok set_recovery_delay 100000 2021-01-15T16:24:00.101 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:24:00.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2021-01-15T16:24:00.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2021-01-15T16:24:00.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:24:00.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:24:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:24:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:00.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:24:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2021-01-15T16:24:00.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok set_recovery_delay 100000 2021-01-15T16:24:00.212 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:24:00.221 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2021-01-15T16:24:00.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2021-01-15T16:24:00.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:00.591 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:00.591 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-01-15T16:23:24.236715+0000 29'100 40:122 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:00.592 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:00.592 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:00.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2021-01-15T16:24:01.247 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2021-01-15T16:24:01.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2021-01-15T16:24:01.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:24:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:24:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:24:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:24:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:24:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:24:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:24:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:24:01.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:24:01.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:24:01.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:24:01.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:24:01.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:24:01.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:24:01.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:24:01.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:01.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:01.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:01.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:01.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:01.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:24:01.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:24:01.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:24:01.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:24:01.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:24:01.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:01.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:24:01.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:24:01.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:24:01.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:24:01.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:01.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:24:01.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:24:01.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:24:01.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:24:01.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:24:01.275 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:24:01.288 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:01.287+0000 7f9ddc8d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:01.294 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:01.293+0000 7f9ddc8d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:01.296 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:01.295+0000 7f9ddc8d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:01.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2021-01-15T16:24:01.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2021-01-15T16:24:01.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:01.811+0000 7f9ddc8d5080 -1 Falling back to public interface 2021-01-15T16:24:03.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:03.041+0000 7f9ddc8d5080 -1 osd.1 29 log_to_monitors {default=true} 2021-01-15T16:24:06.659 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2021-01-15T16:24:06.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2021-01-15T16:24:06.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:07.042 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:07.042 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-01-15T16:23:24.236715+0000 40'101 41:125 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:07.043 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:07.043 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:07.044 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:07.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2021-01-15T16:24:07.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:24:07.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:24:07.055 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:24:07.055 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:07.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:24:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2021-01-15T16:24:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok set_recovery_delay 100000 2021-01-15T16:24:07.158 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:24:07.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2021-01-15T16:24:07.913 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2021-01-15T16:24:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2021-01-15T16:24:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:24:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:24:07.925 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:08.380 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/40979,v1:127.0.0.1:6803/40979] [v2:127.0.0.1:6804/40979,v1:127.0.0.1:6805/40979] exists,up d3275fea-a12c-420f-90e6-0ac2fa60dc87 2021-01-15T16:24:08.380 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:08.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:24:08.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:24:08.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:24:08.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2021-01-15T16:24:08.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:08.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:24:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:08.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:24:08.778 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/43089,v1:127.0.0.1:6819/43089] [v2:127.0.0.1:6820/43089,v1:127.0.0.1:6821/43089] exists,up 66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:24:08.778 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:24:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:24:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:24:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2021-01-15T16:24:08.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:08.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:24:08.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:08.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:08.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:24:09.177 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/41468,v1:127.0.0.1:6811/41468] [v2:127.0.0.1:6812/41468,v1:127.0.0.1:6813/41468] exists,up 42a32ee4-c9a8-4952-b91b-7f19d23416f3 2021-01-15T16:24:09.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:24:09.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:24:09.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:24:09.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:09.548 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:09.548 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2021-01-15T16:24:08.003485+0000 29'101 44:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:09.548 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:09.549 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:09.551 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:09.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2021-01-15T16:24:09.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:09.559 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2021-01-15T16:24:09.929 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:09.930 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 peering 2021-01-15T16:24:08.003485+0000 29'101 44:123 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:09.930 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:09.930 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:09.931 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:09.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2021-01-15T16:24:20.346 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2021-01-15T16:24:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2021-01-15T16:24:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:20.726 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:20.727 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-01-15T16:24:08.489675+0000 40'101 45:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:20.727 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:20.727 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:20.728 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:20.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2021-01-15T16:24:20.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:24:20.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:24:20.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:24:20.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:24:20.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:24:20.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:24:20.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2021-01-15T16:24:20.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:20.844 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2021-01-15T16:24:21.214 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:21.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2021-01-15T16:24:08.489675+0000 40'101 45:132 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:21.215 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:21.215 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:21.216 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2021-01-15T16:24:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:24:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:24:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:24:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:24:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:24:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:24:21.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:24:21.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:24:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:24:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:24:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:24:21.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:24:21.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:24:21.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:24:21.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:24:21.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:21.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:21.240 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:21.241 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:21.241 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:21.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:24:21.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:24:21.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:24:21.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:24:21.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:24:21.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:21.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:24:21.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:24:21.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:24:21.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:24:21.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:21.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:24:21.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:24:21.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:24:21.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:24:21.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:24:21.252 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:24:21.257 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:21.256+0000 7f2dbbc2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:21.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:21.258+0000 7f2dbbc2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:21.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:21.260+0000 7f2dbbc2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:21.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:24:21.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:21.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:24:21.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:21.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:21.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:21.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:21.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:21.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:24:21.634 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:21.777+0000 7f2dbbc2d080 -1 Falling back to public interface 2021-01-15T16:24:22.029 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 44 up_thru 44 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/43089,v1:127.0.0.1:6819/43089] [v2:127.0.0.1:6820/43089,v1:127.0.0.1:6821/43089] exists,up 66bb123c-6411-4ea2-9bd5-9eb095d2cddb 2021-01-15T16:24:22.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:24:22.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:24:22.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:24:22.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2021-01-15T16:24:22.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:22.699+0000 7f2dbbc2d080 -1 osd.1 45 log_to_monitors {default=true} 2021-01-15T16:24:42.030 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2021-01-15T16:24:42.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2021-01-15T16:24:42.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:42.410 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:42.411 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-01-15T16:24:24.011283+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:42.411 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:42.411 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:42.413 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:42.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-01-15T16:24:42.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T16:24:42.555 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:24:42.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-01-15T16:24:42.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T16:24:42.696 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:24:42.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2021-01-15T16:24:42.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2021-01-15T16:24:42.837 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:24:42.846 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2021-01-15T16:24:42.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2021-01-15T16:24:42.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2021-01-15T16:24:43.216 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:24:43.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2021-01-15T16:24:24.011283+0000 45'102 48:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:22:33.415506+0000 0'0 2021-01-15T16:22:33.415506+0000 0 2021-01-15T16:24:43.217 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:24:43.217 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:24:43.218 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:24:43.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2021-01-15T16:24:43.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2021-01-15T16:24:43.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:24:43.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2021-01-15T16:24:43.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:24:43.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2021-01-15T16:24:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:24:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2021-01-15T16:24:43.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:24:43.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2021-01-15T16:24:43.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2021-01-15T16:24:43.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-01-15T16:24:43.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:24:43.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2021-01-15T16:23:04.105372+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:24:43.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2021-01-15T16:23:04.105372+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:24:43.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:24:43.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2021-01-15T16:24:43.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-01-15T16:24:43.393 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:24:43.393 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2021-01-15T16:24:43.393 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2021-01-15T16:23:04.105372+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:24:43.393 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:24:08.299+0000 7f9db9da9700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-01-15T16:24:43.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2021-01-15T16:24:43.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:24:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2021-01-15T16:24:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2021-01-15T16:24:43.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T16:24:43.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T16:24:43.396 INFO:tasks.workunit.client.0.smithi145.stdout:success 2021-01-15T16:24:43.887 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T16:24:43.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2021-01-15T16:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:24:43.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:24:44.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:24:44.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-01-15T16:24:44.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:24:44.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:24:44.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:24:44.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:24:44.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:24:44.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:24:44.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:24:44.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:24:44.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:24:44.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:24:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:24:44.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:24:44.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:24:44.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:24:44.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:24:44.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:24:44.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:24:44.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:24:44.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:24:44.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:24:44.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:24:44.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:24:44.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:24:44.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:24:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:24:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:24:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-01-15T16:24:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-01-15T16:24:44.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-01-15T16:24:44.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-01-15T16:24:44.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:24:44.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:24:44.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:24:44.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:24:44.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:24:44.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:24:44.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:24:44.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:24:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:24:44.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:24:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:24:44.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:24:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:24:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:24:44.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:24:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:24:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:24:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:24:44.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:24:44.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:24:44.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:24:44.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:24:44.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:24:44.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:24:44.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:24:44.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:24:44.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-01-15T16:24:44.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:24:44.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.33425 2021-01-15T16:24:44.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:24:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:24:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:24:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-01-15T16:24:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2021-01-15T16:24:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2021-01-15T16:24:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2021-01-15T16:24:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2021-01-15T16:24:44.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2021-01-15T16:24:44.044 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2021-01-15T16:24:44.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2021-01-15T16:24:44.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2021-01-15T16:24:44.045 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:24:44.045 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:24:44.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2021-01-15T16:24:44.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-01-15T16:24:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:24:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:24:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:24:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-01-15T16:24:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-01-15T16:24:44.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:24:44.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:44.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:44.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:24:44.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:24:44.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:24:44.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:24:44.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:24:44.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:24:44.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:24:44.516 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:24:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:24:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:24:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:24:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.518 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:24:44.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:24:44.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get fsid 2021-01-15T16:24:44.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:24:44.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:24:44.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:24:44.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:24:44.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:24:44.618 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:24:44.618 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:24:44.618 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:24:44.618 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:24:44.619 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:44.619 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:44.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:24:44.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:24:44.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get mon_host 2021-01-15T16:24:44.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2021-01-15T16:24:44.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2021-01-15T16:24:44.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:24:44.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:24:44.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:24:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2021-01-15T16:24:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:24:45.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:24:45.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:45.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:45.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:45.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:24:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:24:45.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-01-15T16:24:45.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2021-01-15T16:24:45.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:24:45.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:24:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:24:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:24:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:24:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:24:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:24:45.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:24:45.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:24:45.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:24:45.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:24:45.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:24:45.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:24:45.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:24:45.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:24:45.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:45.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:45.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:45.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:45.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:45.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:24:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:24:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:24:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:24:45.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:24:45.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:24:45.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:24:45.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:45.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:24:45.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:24:45.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:24:45.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:24:45.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:24:45.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c0cb32e2-1310-411a-82ce-6f897da57c9c' 2021-01-15T16:24:45.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:24:45.657 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:24:45.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDNwQFgYQe8JxAAJBCVMX0KeREalk/lxJ6e/g== 2021-01-15T16:24:45.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDNwQFgYQe8JxAAJBCVMX0KeREalk/lxJ6e/g=="}' 2021-01-15T16:24:45.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c0cb32e2-1310-411a-82ce-6f897da57c9c -i td/divergent-priors/0/new.json 2021-01-15T16:24:46.055 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:46.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2021-01-15T16:24:46.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDNwQFgYQe8JxAAJBCVMX0KeREalk/lxJ6e/g== --osd-uuid c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:24:46.096 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.096+0000 7f0f6060a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:46.098 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.098+0000 7f0f6060a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:46.099 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.099+0000 7f0f6060a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:46.100 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.100+0000 7f0f6060a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:24:46.100 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.100+0000 7f0f6060a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:24:46.101 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.100+0000 7f0f6060a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:24:46.138 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:46.138+0000 7f0f6060a080 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-01-15T16:24:50.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-01-15T16:24:50.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:24:50.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:24:50.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:24:50.600 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:24:51.132 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:24:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:24:51.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:51.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:24:51.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:24:51.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:24:51.160 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:51.160+0000 7f2e32770080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:51.181 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:51.180+0000 7f2e32770080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:51.182+0000 7f2e32770080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:24:51.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:51.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:24:51.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:51.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:51.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:51.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:51.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:51.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:24:51.543 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:51.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:51.697+0000 7f2e32770080 -1 Falling back to public interface 2021-01-15T16:24:51.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:24:52.507 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:52.507+0000 7f2e32770080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:24:52.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:24:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:24:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:24:52.942 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:24:53.358 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/45557,v1:127.0.0.1:6803/45557] [v2:127.0.0.1:6804/45557,v1:127.0.0.1:6805/45557] exists,up c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:24:53.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:24:53.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:24:53.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:24:53.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-01-15T16:24:53.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2021-01-15T16:24:53.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:24:53.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:24:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:24:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:24:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:24:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:24:53.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:24:53.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:24:53.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:24:53.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:24:53.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:24:53.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:24:53.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:24:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:24:53.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:24:53.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:24:53.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:24:53.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:24:53.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:24:53.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:24:53.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:24:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:24:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:24:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:24:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:24:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:24:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:24:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:24:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:24:53.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:24:53.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:24:53.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:24:53.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:24:53.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 1cc9002f-f77d-47e8-ab42-303305234f9f' 2021-01-15T16:24:53.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:24:53.370 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:24:53.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVwQFgEeiBFhAAJkocpyC34chSBJhpT5QTWA== 2021-01-15T16:24:53.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVwQFgEeiBFhAAJkocpyC34chSBJhpT5QTWA=="}' 2021-01-15T16:24:53.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1cc9002f-f77d-47e8-ab42-303305234f9f -i td/divergent-priors/1/new.json 2021-01-15T16:24:53.894 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:24:53.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2021-01-15T16:24:53.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVwQFgEeiBFhAAJkocpyC34chSBJhpT5QTWA== --osd-uuid 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:24:53.935 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.934+0000 7f10df76f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:53.936 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.936+0000 7f10df76f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:53.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.937+0000 7f10df76f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:53.939 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.938+0000 7f10df76f080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:24:53.940 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.938+0000 7f10df76f080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:24:53.940 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:53.938+0000 7f10df76f080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:24:54.013 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:54.012+0000 7f10df76f080 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-01-15T16:24:58.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-01-15T16:24:58.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:24:58.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:24:58.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:24:58.496 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:24:59.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:24:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:24:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:24:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:24:59.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:24:59.037 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:24:59.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:59.063+0000 7fa4d2d32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:59.079 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:59.077+0000 7fa4d2d32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:59.080 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:59.078+0000 7fa4d2d32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:24:59.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:24:59.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:24:59.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:24:59.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:24:59.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:24:59.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:24:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:24:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:24:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:24:59.447 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:24:59.595 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:24:59.594+0000 7fa4d2d32080 -1 Falling back to public interface 2021-01-15T16:24:59.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:25:00.321 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:00.321+0000 7fa4d2d32080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:25:00.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:25:00.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:00.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:25:00.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:00.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:25:00.900 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:25:01.332 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/46821,v1:127.0.0.1:6811/46821] [v2:127.0.0.1:6812/46821,v1:127.0.0.1:6813/46821] exists,up 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:25:01.333 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:25:01.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:25:01.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:25:01.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:25:01.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2021-01-15T16:25:01.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2021-01-15T16:25:01.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:25:01.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:25:01.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:25:01.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:25:01.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:25:01.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:25:01.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:25:01.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:25:01.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:25:01.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:25:01.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:25:01.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:25:01.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:25:01.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:25:01.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:25:01.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:25:01.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:25:01.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:25:01.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:25:01.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:25:01.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:01.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:01.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:25:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:25:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:25:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:25:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:25:01.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:25:01.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:25:01.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:01.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:25:01.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:25:01.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:25:01.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:25:01.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:25:01.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 3648422a-0c5d-4029-85e2-e7056e79a86d' 2021-01-15T16:25:01.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:25:01.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDdwQFgGOzoFBAALd9WToWx80mLHnCe0BnJKw== 2021-01-15T16:25:01.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDdwQFgGOzoFBAALd9WToWx80mLHnCe0BnJKw=="}' 2021-01-15T16:25:01.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3648422a-0c5d-4029-85e2-e7056e79a86d -i td/divergent-priors/2/new.json 2021-01-15T16:25:01.830 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:25:01.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2021-01-15T16:25:01.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdwQFgGOzoFBAALd9WToWx80mLHnCe0BnJKw== --osd-uuid 3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:25:01.874 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.870+0000 7f65e270d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:01.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.872+0000 7f65e270d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:01.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.873+0000 7f65e270d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:01.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.873+0000 7f65e270d080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:25:01.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.873+0000 7f65e270d080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:25:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.873+0000 7f65e270d080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:25:01.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:01.927+0000 7f65e270d080 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-01-15T16:25:06.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-01-15T16:25:06.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:25:06.170 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:25:06.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:25:06.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:25:06.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:25:06.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:06.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:25:06.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:25:06.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:25:06.704 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:25:06.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:06.730+0000 7f0847102080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:06.742 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:06.741+0000 7f0847102080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:06.743 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:06.742+0000 7f0847102080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:07.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:25:07.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:25:07.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:25:07.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:25:07.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:25:07.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:07.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:25:07.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:07.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:25:07.105 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:25:07.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:07.258+0000 7f0847102080 -1 Falling back to public interface 2021-01-15T16:25:07.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:25:08.052 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:08.052+0000 7f0847102080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:25:08.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:25:08.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:08.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:25:08.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:08.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:25:08.516 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:25:08.939 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/48085,v1:127.0.0.1:6819/48085] [v2:127.0.0.1:6820/48085,v1:127.0.0.1:6821/48085] exists,up 3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:25:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:25:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:25:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:25:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2021-01-15T16:25:09.603 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:25:09.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2021-01-15T16:25:10.389 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2021-01-15T16:25:10.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2021-01-15T16:25:11.032 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2021-01-15T16:25:11.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2021-01-15T16:25:11.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:25:11.596 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:25:11.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:25:12.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2021-01-15T16:25:13.315 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T16:25:13.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2021-01-15T16:25:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T16:25:14.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2021-01-15T16:25:14.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:25:14.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:25:14.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:25:14.607 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:25:14.607 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:25:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:25:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:14.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:25:14.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:25:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:25:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:25:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:14.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:25:14.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T16:25:14.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T16:25:14.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T16:25:14.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:14.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:25:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:25:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:25:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2021-01-15T16:25:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:15.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:25:15.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:15.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:25:15.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:25:15.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:25:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:25:15.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:15.041 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:25:15.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836487 2021-01-15T16:25:15.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:25:16.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:25:16.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T16:25:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T16:25:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:25:16.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T16:25:16.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:16.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T16:25:16.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T16:25:16.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:25:16.839 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T16:25:17.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2021-01-15T16:25:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:17.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:25:17.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:17.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:25:17.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:25:17.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:17.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:25:17.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:25:17.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:25:17.240 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:25:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T16:25:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2021-01-15T16:25:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:25:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:25:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:25:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:25:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:25:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:25:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:25:17.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:25:17.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:25:17.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:25:17.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:25:17.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:25:17.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:25:17.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:25:17.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:25:18.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:25:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:25:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:25:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:25:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:18.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:25:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2021-01-15T16:25:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2021-01-15T16:25:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2021-01-15T16:25:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:18.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:25:18.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T16:25:18.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T16:25:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2021-01-15T16:25:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:18.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:25:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T16:25:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T16:25:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2021-01-15T16:25:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:18.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2021-01-15T16:25:18.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:18.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:25:18.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2021-01-15T16:25:18.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:18.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2021-01-15T16:25:18.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-01-15T16:25:18.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:18.690 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2021-01-15T16:25:19.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2021-01-15T16:25:19.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:25:20.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:25:20.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2021-01-15T16:25:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:20.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T16:25:20.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:20.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:25:20.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T16:25:20.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:20.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T16:25:20.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T16:25:20.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:25:20.496 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T16:25:20.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T16:25:20.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:20.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T16:25:20.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:20.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:25:20.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T16:25:20.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:20.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T16:25:20.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T16:25:20.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:25:20.896 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T16:25:21.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509446 2021-01-15T16:25:21.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:25:21.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:25:21.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:25:21.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:25:21.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:25:21.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:25:21.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:25:21.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:25:21.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:25:21.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:25:21.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:25:22.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:25:22.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:25:22.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:25:22.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:25:22.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:25:22.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:25:22.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:25:22.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2021-01-15T16:25:22.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2021-01-15T16:25:23.084 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:25:23.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2021-01-15T16:25:23.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2021-01-15T16:25:23.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:25:23.096 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2021-01-15T16:25:23.465 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:25:23.465 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:25:12.252161+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:25:23.465 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:25:23.466 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:25:23.466 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:25:23.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2021-01-15T16:25:23.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-01-15T16:25:23.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2021-01-15T16:25:23.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2021-01-15T16:25:23.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-01-15T16:25:23.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2021-01-15T16:25:23.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2021-01-15T16:25:23.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2021-01-15T16:25:23.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2021-01-15T16:25:23.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2021-01-15T16:25:23.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2021-01-15T16:25:23.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2021-01-15T16:25:23.479 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2021-01-15T16:25:23.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:23.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2021-01-15T16:25:23.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:23.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2021-01-15T16:25:23.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:23.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2021-01-15T16:25:23.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:23.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2021-01-15T16:25:23.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:23.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2021-01-15T16:25:24.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2021-01-15T16:25:24.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2021-01-15T16:25:24.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2021-01-15T16:25:24.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2021-01-15T16:25:24.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2021-01-15T16:25:24.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2021-01-15T16:25:24.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2021-01-15T16:25:24.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:24.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2021-01-15T16:25:25.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2021-01-15T16:25:25.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2021-01-15T16:25:25.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2021-01-15T16:25:25.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2021-01-15T16:25:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2021-01-15T16:25:25.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2021-01-15T16:25:25.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2021-01-15T16:25:25.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:25.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2021-01-15T16:25:26.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2021-01-15T16:25:26.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2021-01-15T16:25:26.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2021-01-15T16:25:26.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2021-01-15T16:25:26.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2021-01-15T16:25:26.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2021-01-15T16:25:26.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:26.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2021-01-15T16:25:27.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2021-01-15T16:25:27.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2021-01-15T16:25:27.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2021-01-15T16:25:27.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2021-01-15T16:25:27.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2021-01-15T16:25:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2021-01-15T16:25:27.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2021-01-15T16:25:27.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:27.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2021-01-15T16:25:28.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2021-01-15T16:25:28.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2021-01-15T16:25:28.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2021-01-15T16:25:28.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2021-01-15T16:25:28.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2021-01-15T16:25:28.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2021-01-15T16:25:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2021-01-15T16:25:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2021-01-15T16:25:29.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2021-01-15T16:25:29.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2021-01-15T16:25:29.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2021-01-15T16:25:29.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2021-01-15T16:25:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2021-01-15T16:25:29.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2021-01-15T16:25:29.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2021-01-15T16:25:29.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:29.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2021-01-15T16:25:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2021-01-15T16:25:30.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2021-01-15T16:25:30.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2021-01-15T16:25:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2021-01-15T16:25:30.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2021-01-15T16:25:30.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2021-01-15T16:25:30.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2021-01-15T16:25:30.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:30.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2021-01-15T16:25:31.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2021-01-15T16:25:31.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2021-01-15T16:25:31.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2021-01-15T16:25:31.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2021-01-15T16:25:31.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2021-01-15T16:25:31.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2021-01-15T16:25:31.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2021-01-15T16:25:31.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:31.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2021-01-15T16:25:32.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2021-01-15T16:25:32.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2021-01-15T16:25:32.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2021-01-15T16:25:32.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2021-01-15T16:25:32.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2021-01-15T16:25:32.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2021-01-15T16:25:32.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:32.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2021-01-15T16:25:33.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2021-01-15T16:25:33.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2021-01-15T16:25:33.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2021-01-15T16:25:33.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2021-01-15T16:25:33.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2021-01-15T16:25:33.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2021-01-15T16:25:33.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2021-01-15T16:25:33.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:33.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2021-01-15T16:25:34.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2021-01-15T16:25:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2021-01-15T16:25:34.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2021-01-15T16:25:34.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2021-01-15T16:25:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2021-01-15T16:25:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2021-01-15T16:25:34.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2021-01-15T16:25:34.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:34.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2021-01-15T16:25:35.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2021-01-15T16:25:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2021-01-15T16:25:35.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2021-01-15T16:25:35.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2021-01-15T16:25:35.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2021-01-15T16:25:35.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2021-01-15T16:25:35.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2021-01-15T16:25:35.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:35.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2021-01-15T16:25:36.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2021-01-15T16:25:36.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2021-01-15T16:25:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:25:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2021-01-15T16:25:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:25:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:25:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:25:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:25:36.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:25:36.232 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:25:36.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:25:36.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:25:36.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:25:36.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:25:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:25:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:25:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:25:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:25:36.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:25:36.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:25:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:25:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:25:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:25:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:36.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:25:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T16:25:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T16:25:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T16:25:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:36.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:25:36.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:25:36.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:25:36.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T16:25:36.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:25:36.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:25:37.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T16:25:37.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T16:25:37.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T16:25:37.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:37.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T16:25:37.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:25:37.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T16:25:37.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:37.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T16:25:37.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T16:25:37.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:37.116 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T16:25:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2021-01-15T16:25:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:25:38.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:25:38.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:25:38.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2021-01-15T16:25:38.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:38.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:25:38.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:38.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:25:38.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:25:38.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:38.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:25:38.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:25:38.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:25:38.923 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:25:39.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T16:25:39.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:25:39.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T16:25:39.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:25:39.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:25:39.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T16:25:39.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:25:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T16:25:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T16:25:39.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:25:39.322 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T16:25:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509452 2021-01-15T16:25:39.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:25:39.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:25:39.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:25:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:25:40.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:25:40.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:25:40.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:25:40.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:25:40.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:25:40.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:25:40.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:25:40.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:25:40.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:25:40.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:25:40.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:25:41.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:25:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:25:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:25:41.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2021-01-15T16:25:41.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T16:25:41.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2021-01-15T16:25:41.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2021-01-15T16:25:41.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T16:25:41.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2021-01-15T16:25:41.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2021-01-15T16:25:41.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:25:41.562 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2021-01-15T16:25:41.937 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:25:41.937 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-01-15T16:25:12.252161+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:25:41.937 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:25:41.937 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:25:41.938 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:25:41.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:41.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2021-01-15T16:25:41.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:25:41.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:25:41.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:25:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:25:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:25:41.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:25:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:25:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok config set objectstore_blackhole 1 2021-01-15T16:25:42.045 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:25:42.046 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:25:42.047 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:25:42.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:42.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2021-01-15T16:25:42.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:25:42.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:25:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:25:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:25:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:25:42.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:25:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:25:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok config set objectstore_blackhole 1 2021-01-15T16:25:42.153 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:25:42.153 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:25:42.154 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:25:42.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2021-01-15T16:25:42.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2021-01-15T16:25:42.162 INFO:tasks.workunit.client.0.smithi145.stdout:create new divergent objects 2021-01-15T16:25:42.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:25:42.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:25:42.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2021-01-15T16:25:42.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2021-01-15T16:25:42.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2021-01-15T16:25:42.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2021-01-15T16:25:42.164 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent objects 2021-01-15T16:25:42.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-01-15T16:25:42.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2021-01-15T16:25:42.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2021-01-15T16:25:42.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2021-01-15T16:25:42.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2021-01-15T16:25:42.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2021-01-15T16:25:42.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-01-15T16:25:42.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2021-01-15T16:25:42.168 INFO:tasks.workunit.client.0.smithi145.stdout:remove divergent objects 2021-01-15T16:25:42.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2021-01-15T16:25:42.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-01-15T16:25:42.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2021-01-15T16:25:42.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2021-01-15T16:25:42.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2021-01-15T16:25:42.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2021-01-15T16:25:42.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2021-01-15T16:25:52.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2021-01-15T16:25:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2021-01-15T16:25:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2021-01-15T16:25:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:25:52.173 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2021-01-15T16:25:52.555 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:25:52.555 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2021-01-15T16:25:12.252161+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:25:52.556 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:25:52.556 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:25:52.557 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:25:52.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2021-01-15T16:25:52.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:25:52.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:25:52.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:25:52.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:25:52.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:25:52.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:25:52.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-01-15T16:25:52.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2021-01-15T16:25:53.213 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T16:25:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-01-15T16:25:53.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2021-01-15T16:25:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T16:25:53.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2021-01-15T16:25:53.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2021-01-15T16:25:54.314 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T16:25:54.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:54.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2021-01-15T16:25:54.874 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T16:25:54.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:54.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2021-01-15T16:25:55.422 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T16:25:55.432 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2021-01-15T16:25:55.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2021-01-15T16:25:55.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:25:55.813 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:25:55.814 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2021-01-15T16:25:12.252161+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:25:55.814 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:25:55.814 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:25:55.815 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:25:55.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:55.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2021-01-15T16:25:55.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:25:55.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:25:55.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:25:55.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:25:55.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:25:55.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:25:55.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:25:55.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:25:55.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:25:55.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:25:55.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:25:55.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:25:55.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:25:55.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:25:55.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:25:55.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:25:55.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:25:55.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:25:55.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:25:55.832 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:25:55.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:55.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:55.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:25:55.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:25:55.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:25:55.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:25:55.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:25:55.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:55.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:25:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:25:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:25:55.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:25:55.835 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:25:55.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:55.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2021-01-15T16:25:55.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:25:55.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:25:55.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:25:55.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:25:55.857 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:55.856+0000 7f55b3224080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:55.859 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:55.859+0000 7f55b3224080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:55.861 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:55.860+0000 7f55b3224080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:56.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:25:56.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:25:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:25:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:25:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:25:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:56.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:25:56.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:56.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:25:56.230 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:25:56.379 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:56.379+0000 7f55b3224080 -1 Falling back to public interface 2021-01-15T16:25:56.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:25:57.603 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:57.603+0000 7f55b3224080 -1 osd.0 29 log_to_monitors {default=true} 2021-01-15T16:25:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:25:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:25:57.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:57.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:25:57.636 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:25:58.043 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/52111,v1:127.0.0.1:6803/52111] [v2:127.0.0.1:6804/52111,v1:127.0.0.1:6805/52111] exists,up c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:25:58.043 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:25:58.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:25:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:25:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:25:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:25:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2021-01-15T16:25:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:25:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:25:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:25:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:25:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:25:58.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:25:58.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:25:58.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:25:58.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:25:58.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:25:58.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:25:58.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:25:58.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:25:58.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:25:58.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:25:58.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:25:58.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:25:58.050 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:25:58.050 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:25:58.050 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:25:58.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:25:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:25:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:25:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:25:58.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:25:58.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:25:58.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:58.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:25:58.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:25:58.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:25:58.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:25:58.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:25:58.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2021-01-15T16:25:58.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:25:58.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:25:58.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:25:58.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:25:58.071 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:58.071+0000 7fe8b05f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:58.073 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:58.073+0000 7fe8b05f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:58.075 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:58.074+0000 7fe8b05f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:25:58.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:25:58.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:25:58.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:25:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:25:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:25:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:25:58.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:58.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:25:58.448 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:25:58.592 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:58.591+0000 7fe8b05f4080 -1 Falling back to public interface 2021-01-15T16:25:58.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:25:59.834 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:25:59.834+0000 7fe8b05f4080 -1 osd.2 29 log_to_monitors {default=true} 2021-01-15T16:25:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:25:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:25:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:25:59.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:25:59.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:25:59.857 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:26:00.312 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/52600,v1:127.0.0.1:6811/52600] [v2:127.0.0.1:6812/52600,v1:127.0.0.1:6813/52600] exists,up 3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:26:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:26:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:26:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:26:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:26:00.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2021-01-15T16:26:00.854 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T16:26:00.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:26:00.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2021-01-15T16:26:01.407 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2021-01-15T16:26:01.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:26:01.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2021-01-15T16:26:01.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:26:01.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:26:01.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:26:01.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:26:01.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:26:01.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:26:01.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:26:01.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:26:01.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:26:01.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:26:01.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:26:01.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:26:01.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:26:01.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:26:01.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:26:01.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:26:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:26:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:26:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:26:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:01.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:26:02.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2021-01-15T16:26:02.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2021-01-15T16:26:02.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855363' 2021-01-15T16:26:02.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:02.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:26:02.122 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:26:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:26:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:26:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:26:02.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:02.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:26:02.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822659 2021-01-15T16:26:02.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822659 2021-01-15T16:26:02.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855363 2-154618822659' 2021-01-15T16:26:02.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:26:02.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855363 2021-01-15T16:26:02.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:26:02.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:26:02.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855363 2021-01-15T16:26:02.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:26:02.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2021-01-15T16:26:02.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2021-01-15T16:26:02.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:26:02.269 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 150323855363 2021-01-15T16:26:02.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 150323855363 2021-01-15T16:26:02.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:26:03.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:26:03.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:26:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855364 -lt 150323855363 2021-01-15T16:26:04.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:26:04.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822659 2021-01-15T16:26:04.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:26:04.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:26:04.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822659 2021-01-15T16:26:04.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:26:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822659 2021-01-15T16:26:04.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2021-01-15T16:26:04.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:26:04.078 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 154618822659 2021-01-15T16:26:04.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822659 -lt 154618822659 2021-01-15T16:26:04.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:26:04.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:04.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:05.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:26:05.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:05.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:05.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:05.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:05.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:05.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:05.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:05.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:05.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:05.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:05.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:26:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:26:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:26:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:26:06.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:06.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:06.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:06.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:06.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:06.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:06.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:06.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:06.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:06.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:06.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:06.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:06.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:06.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:06.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:06.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:06.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:06.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:06.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:06.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:07.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:07.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:07.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:26:07.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:07.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:26:07.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:07.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:07.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:07.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:07.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:07.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:07.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:07.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:08.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:08.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:08.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:08.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:08.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:08.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:08.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:08.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:08.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:08.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:08.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:08.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:08.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:26:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:26:09.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:09.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:09.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:09.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:09.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:09.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:09.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:09.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:09.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:09.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:09.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:10.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:10.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:10.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:10.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:10.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:26:10.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:10.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:26:11.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:11.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:11.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:11.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:11.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:11.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:11.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:11.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:12.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:12.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:12.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:12.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:12.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:13.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:13.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:13.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:26:13.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:26:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:14.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:14.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:14.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:14.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:14.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:14.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:14.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:15.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:15.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:15.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:15.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:15.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:15.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:15.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:15.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:15.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:15.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:15.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:15.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:15.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:16.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:16.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:26:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:26:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:19.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:19.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:19.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:19.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:19.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:19.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:19.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:19.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:19.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:19.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:20.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:20.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:20.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:20.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:20.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:20.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:20.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:20.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:20.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:26:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:20.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:26:27.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:27.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:27.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:27.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:27.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:27.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:27.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:27.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:27.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:27.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:27.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:27.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:28.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:28.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:28.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:28.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:28.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:28.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:26:28.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:28.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:26:35.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:35.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:35.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:35.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:35.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:35.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:35.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:35.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:36.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:36.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:36.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:36.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:36.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:36.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:36.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:36.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:36.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:36.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:36.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:36.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:36.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:37.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:26:37.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:26:37.922 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:26:37.922 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "c816ef7c-8e57-40e8-8653-329194c60552", 2021-01-15T16:26:37.923 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:26:37.923 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:26:37.923 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:26:37.921336+0000", 2021-01-15T16:26:37.923 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:26:37.924 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:26:37.924 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:26:37.924 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:26:37.925 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:26:37.925 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:26:37.925 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:26:37.925 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:26:37.926 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:26:37.926 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.926 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:26:37.926 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:26:37.927 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.927 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:26:37.927 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:26:37.929 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:26:37.929 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:26:37.930 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:26:37.930 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.930 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:26:37.930 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.931 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:26:37.931 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.931 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.931 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:26:37.932 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.932 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:26:37.932 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:26:37.932 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:26:37.932 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2021-01-15T16:26:37.933 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:26:37.933 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.933 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:26:37.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.933 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:26:37.934 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.934 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.934 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:26:37.935 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.935 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:26:37.936 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:26:37.936 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:26:37.937 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:26:37.937 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:26:37.937 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.938 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:26:37.938 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.939 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:26:37.939 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.939 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.940 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:26:37.940 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.941 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.941 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:26:37.941 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.942 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.942 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:26:37.943 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.943 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:26:37.943 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:26:37.944 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:26:37.944 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:26:37.944 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:26:37.946 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:24:44.065303Z", 2021-01-15T16:26:37.947 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:24:44.065303Z", 2021-01-15T16:26:37.947 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:26:37.949 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:26:37.949 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:26:37.949 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:26:37.949 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:26:37.950 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:26:37.950 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:26:37.951 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:26:37.952 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:26:37.952 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:26:37.952 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:26:37.953 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:26:37.953 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:26:37.954 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:26:37.955 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:26:37.955 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.956 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:26:37.956 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.957 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:26:37.957 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.957 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:26:37.958 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:26:37.959 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:26:37.959 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:37.959 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.959 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:37.960 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:26:37.960 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:26:37.961 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.961 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:37.961 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.961 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:26:37.962 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:26:37.962 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:26:37.962 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:26:37.963 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:26:37.963 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.963 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:37.963 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.963 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:26:37.964 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:26:37.964 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.964 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:26:37.964 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 39, 2021-01-15T16:26:37.964 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:26:37.965 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:24:44.820164+0000", 2021-01-15T16:26:37.965 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:26:01.452641+0000", 2021-01-15T16:26:37.965 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:25:59.881925+0000", 2021-01-15T16:26:37.966 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:26:01.201446+0000", 2021-01-15T16:26:37.966 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:26:37.966 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:26:37.966 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:26:37.967 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:26:37.967 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:26:37.967 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:26:37.967 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:26:37.968 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:26:37.968 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:26:37.969 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:26:37.969 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.969 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2021-01-15T16:26:37.970 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:26:37.970 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:26:37.971 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:26:37.971 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:26:37.972 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:26:37.972 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:26:37.972 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:26:37.973 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:26:37.973 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:26:37.974 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:26:37.974 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.975 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:26:37.975 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:26:37.975 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:25:11.336838+0000", 2021-01-15T16:26:37.976 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T16:26:37.976 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T16:26:37.977 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:26:37.977 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:26:37.977 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:26:37.978 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T16:26:37.978 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:26:37.979 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:26:37.979 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:26:37.979 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:26:37.979 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:26:37.980 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T16:26:37.980 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T16:26:37.980 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:26:37.981 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T16:26:37.981 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T16:26:37.982 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T16:26:37.982 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:26:37.982 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:26:37.983 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:26:37.983 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:26:37.984 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:26:37.984 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:26:37.984 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:26:37.985 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.985 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2021-01-15T16:26:37.985 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:26:37.986 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:26:37.986 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T16:26:37.987 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:26:37.987 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T16:26:37.988 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T16:26:37.988 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T16:26:37.988 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T16:26:37.989 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:26:37.989 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:26:37.989 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:26:37.990 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:26:37.990 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:26:37.990 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:26:37.990 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:26:37.991 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:26:37.991 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:26:37.991 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:26:37.991 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:26:37.991 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:26:37.992 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:26:37.992 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:26:37.992 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:26:37.992 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T16:26:37.993 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:26:37.993 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:26:37.993 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:37.993 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:26:37.993 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:26:37.994 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:26:37.994 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:26:37.994 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:26:37.994 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:26:37.995 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:26:37.995 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:26:37.995 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T16:26:37.995 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:26:37.995 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:26:37.996 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:26:37.996 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:26:37.996 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:37.996 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:37.996 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:26:37.997 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:37.997 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:26:37.997 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "c0cb32e2-1310-411a-82ce-6f897da57c9c", 2021-01-15T16:26:37.997 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:26:37.997 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:26:37.998 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:26:37.998 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:26:37.998 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:26:37.998 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:26:37.999 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2021-01-15T16:26:37.999 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 38, 2021-01-15T16:26:37.999 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2021-01-15T16:26:37.999 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:26:37.999 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:26:38.000 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.000 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.000 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.000 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:26:38.000 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.001 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.001 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.001 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.001 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:26:38.002 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.002 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.002 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.002 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.002 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:26:38.003 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.003 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.003 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.003 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:26:38.003 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.004 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.004 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.004 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.004 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:26:38.005 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.005 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.005 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.005 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.005 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:26:38.006 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.006 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.006 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.006 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:26:38.006 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.007 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.007 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.007 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.007 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:26:38.008 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.008 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.008 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.008 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.008 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:26:38.009 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.009 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.009 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.009 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:26:38.010 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.010 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.010 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.010 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.010 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:26:38.011 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:26:38.011 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.011 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.011 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.012 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/52111", 2021-01-15T16:26:38.012 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/52111", 2021-01-15T16:26:38.012 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/52111", 2021-01-15T16:26:38.012 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/52111", 2021-01-15T16:26:38.012 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:26:38.013 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:26:38.013 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:26:38.013 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.013 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.013 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.014 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:26:38.014 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "1cc9002f-f77d-47e8-ab42-303305234f9f", 2021-01-15T16:26:38.014 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:26:38.014 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:26:38.015 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:26:38.015 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:26:38.015 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:26:38.015 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:26:38.015 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:26:38.016 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 23, 2021-01-15T16:26:38.016 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2021-01-15T16:26:38.016 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:26:38.016 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:26:38.016 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.017 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.017 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.017 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:26:38.017 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.018 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.018 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.018 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.018 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:26:38.018 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.019 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.019 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.019 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.019 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:26:38.019 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.020 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.020 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.020 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:26:38.020 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.020 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.021 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.021 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.021 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:26:38.021 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.022 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.022 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.022 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.022 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:26:38.022 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.023 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.023 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.023 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:26:38.023 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.023 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.024 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.024 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.024 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:26:38.024 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.025 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.025 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.025 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.025 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:26:38.025 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.026 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.026 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.026 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:26:38.026 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.026 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.027 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.027 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.027 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:26:38.027 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:26:38.028 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.028 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.028 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.028 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/46821", 2021-01-15T16:26:38.028 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/46821", 2021-01-15T16:26:38.029 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/46821", 2021-01-15T16:26:38.029 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/46821", 2021-01-15T16:26:38.029 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:26:38.030 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:26:38.030 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.030 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.030 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.030 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:26:38.031 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "3648422a-0c5d-4029-85e2-e7056e79a86d", 2021-01-15T16:26:38.031 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:26:38.031 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:26:38.031 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:26:38.032 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:26:38.032 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:26:38.032 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:26:38.032 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2021-01-15T16:26:38.032 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:26:38.033 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2021-01-15T16:26:38.033 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:26:38.033 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:26:38.033 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.033 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.034 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.034 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:26:38.034 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.034 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.035 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.035 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.035 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:26:38.035 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.035 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.036 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.036 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.036 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:26:38.036 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.037 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.037 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:26:38.037 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.037 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.037 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.038 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.038 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:26:38.038 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.038 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.039 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.039 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.039 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:26:38.039 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.039 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.040 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.040 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:26:38.040 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.040 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.040 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.041 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.041 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:26:38.041 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.041 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:26:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:26:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.043 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:26:38.043 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:26:38.043 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.043 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.043 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.044 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:26:38.044 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:26:38.044 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:26:38.044 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.045 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.045 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.045 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/52600", 2021-01-15T16:26:38.045 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/52600", 2021-01-15T16:26:38.045 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/52600", 2021-01-15T16:26:38.046 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/52600", 2021-01-15T16:26:38.046 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:26:38.046 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:26:38.046 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:26:38.046 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.047 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.047 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.047 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:26:38.047 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.048 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:26:38.048 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:53.008501+0000", 2021-01-15T16:26:38.048 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:26:38.048 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:26:38.048 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:26:38.049 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:26:38.049 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:24:53.491260+0000", 2021-01-15T16:26:38.049 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:26:38.049 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.049 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.050 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:26:38.050 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:53.561176+0000", 2021-01-15T16:26:38.050 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:26:38.050 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:26:38.051 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:26:38.051 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:26:38.051 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:25:01.279437+0000", 2021-01-15T16:26:38.051 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:26:38.051 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.052 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.052 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:26:38.052 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:54.112981+0000", 2021-01-15T16:26:38.052 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:26:38.052 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:26:38.053 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:26:38.053 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:26:38.053 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:25:09.017516+0000", 2021-01-15T16:26:38.053 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:26:38.054 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.054 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.054 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:26:38.054 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:26:38.054 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:26:38.055 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:26:38.055 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:26:38.055 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:26:38.055 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:26:38.055 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:26:38.056 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:26:38.056 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:26:38.056 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:26:38.056 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.057 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.057 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:26:38.057 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:26:38.057 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:26:38.057 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:26:38.058 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:26:38.058 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:26:38.058 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:26:38.058 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:26:38.058 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:26:38.059 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:26:38.059 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:26:38.059 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.059 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.060 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:26:38.060 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.060 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:26:38.060 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:26:38.060 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/52111,v1:127.0.0.1:6805/52111]", 2021-01-15T16:26:38.061 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:26:38.061 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:26:38.061 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:26:38.061 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:26:38.061 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:26:38.062 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:26:38.062 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:26:38.062 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:26:38.062 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:26:38.062 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:26:38.063 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:26:38.063 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:26:38.063 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:26:38.063 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:26:38.064 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:26:38.064 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:26:38.064 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:26:38.064 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:26:38.064 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:26:38.065 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:26:38.065 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:26:38.065 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:26:38.065 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:26:38.065 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:26:38.066 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:26:38.066 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/52111,v1:127.0.0.1:6803/52111]", 2021-01-15T16:26:38.066 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:26:38.066 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/52111,v1:127.0.0.1:6809/52111]", 2021-01-15T16:26:38.067 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/52111,v1:127.0.0.1:6807/52111]", 2021-01-15T16:26:38.067 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:26:38.067 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:26:38.067 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:26:38.067 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:26:38.068 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:26:38.068 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:26:38.068 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:26:38.068 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:26:38.068 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:26:38.069 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:26:38.069 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:26:38.069 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:26:38.069 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:26:38.069 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.070 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.070 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:26:38.070 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:26:38.070 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/46821,v1:127.0.0.1:6813/46821]", 2021-01-15T16:26:38.071 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:26:38.071 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:26:38.071 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:26:38.071 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:26:38.071 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:26:38.072 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:26:38.072 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:26:38.072 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:26:38.072 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:26:38.072 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:26:38.073 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:26:38.073 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:26:38.073 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:26:38.073 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:26:38.073 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:26:38.074 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:26:38.074 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:26:38.074 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:26:38.074 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:26:38.075 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:26:38.075 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:26:38.075 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:26:38.075 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:26:38.075 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:26:38.076 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:26:38.076 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/46821,v1:127.0.0.1:6811/46821]", 2021-01-15T16:26:38.076 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:26:38.076 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/46821,v1:127.0.0.1:6817/46821]", 2021-01-15T16:26:38.076 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/46821,v1:127.0.0.1:6815/46821]", 2021-01-15T16:26:38.077 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:26:38.077 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:26:38.077 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:26:38.077 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:26:38.078 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:26:38.078 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:26:38.078 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:26:38.078 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:26:38.078 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:26:38.079 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:26:38.079 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:26:38.079 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:26:38.079 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:26:38.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.080 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:26:38.080 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:26:38.080 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/52600,v1:127.0.0.1:6813/52600]", 2021-01-15T16:26:38.081 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:26:38.081 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:26:38.081 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:26:38.081 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:26:38.081 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:26:38.082 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:26:38.082 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:26:38.082 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:26:38.082 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:26:38.082 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:26:38.083 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:26:38.083 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:26:38.083 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:26:38.083 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:26:38.084 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:26:38.084 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:26:38.084 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:26:38.084 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:26:38.084 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:26:38.085 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:26:38.085 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:26:38.085 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:26:38.085 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:26:38.085 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:26:38.086 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:26:38.086 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/52600,v1:127.0.0.1:6811/52600]", 2021-01-15T16:26:38.086 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:26:38.086 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/52600,v1:127.0.0.1:6817/52600]", 2021-01-15T16:26:38.087 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/52600,v1:127.0.0.1:6815/52600]", 2021-01-15T16:26:38.087 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:26:38.087 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:26:38.087 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:26:38.087 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:26:38.088 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:26:38.088 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:26:38.088 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:26:38.088 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:26:38.088 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:26:38.089 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:26:38.089 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:26:38.089 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:26:38.089 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:26:38.090 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.090 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.090 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:26:38.090 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:26:38.090 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:26:38.091 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:26:38.091 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.091 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:26:38.091 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.092 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:26:38.092 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 35 2021-01-15T16:26:38.092 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.092 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.092 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:26:38.093 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:26:38.093 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.093 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.093 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:26:38.093 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 37 2021-01-15T16:26:38.094 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.094 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.094 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.094 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:26:38.094 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 39, 2021-01-15T16:26:38.095 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:26:38.095 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:26:38.095 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.095 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:26:38.096 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:26:38.096 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:26:38.096 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.096 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.096 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:26:38.097 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:26:38.097 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:26:38.097 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.097 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.097 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:26:38.098 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:26:38.098 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:26:38.098 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.098 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.098 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:26:38.099 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.099 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:26:38.099 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:26:38.099 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.100 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.100 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:26:38.100 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:26:38.100 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.100 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.101 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:26:38.101 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:26:38.101 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.101 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.101 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:26:38.102 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:26:38.102 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.102 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:26:38.102 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:26:38.103 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.103 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.103 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:26:38.103 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:26:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:26:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:26:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.105 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.105 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:26:38.105 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:26:38.105 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.105 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.106 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:26:38.106 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:26:38.106 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.107 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:26:38.107 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:26:38.107 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.107 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.107 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:26:38.108 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:26:38.108 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.108 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.108 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:26:38.108 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:26:38.109 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.109 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.109 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:26:38.109 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.109 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:26:38.110 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:26:38.110 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:26:38.110 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:26:38.110 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.111 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:26:38.111 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:26:38.111 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:26:38.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.111 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:26:38.112 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.112 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:26:38.112 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.112 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.112 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.113 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.113 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:26:38.113 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:26:38.113 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:26:38.113 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:26:38.114 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.114 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:26:38.114 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:26:38.114 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:26:38.115 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.115 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:26:38.115 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.115 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:26:38.115 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.116 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.116 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.116 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:26:38.116 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:26:38.117 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:26:38.117 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:26:38.117 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.117 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:26:38.118 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:26:38.118 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:26:38.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.118 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:26:38.118 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.119 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:26:38.119 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.119 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.119 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:26:38.120 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.120 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:26:38.120 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.120 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.120 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:26:38.121 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.121 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:26:38.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.121 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.121 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.122 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.122 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:26:38.122 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:26:38.122 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:26:38.122 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:26:38.123 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:26:38.123 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:26:38.123 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:26:38.123 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:26:38.124 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.124 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:26:38.124 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.124 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:26:38.124 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.125 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.125 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:26:38.125 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.125 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:26:38.125 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.126 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.126 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:26:38.126 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:26:38.126 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:26:38.127 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.127 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.127 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.127 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.127 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:26:38.128 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.128 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:26:38.128 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:26:38.128 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:26:38.128 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:26:38.129 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:26:38.129 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:26:38.129 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:26:38.129 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.129 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:26:38.130 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:26:38.130 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:26:38.130 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.130 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.131 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:26:38.131 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:26:38.131 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:26:38.131 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.131 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.132 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:26:38.132 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.132 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:26:38.132 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.132 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.133 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:26:38.133 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:26:38.133 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:26:38.133 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:26:38.134 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:26:38.134 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:26:38.134 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:26:38.134 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:26:38.134 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:26:38.135 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:26:38.135 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:26:38.135 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:26:38.135 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:26:38.135 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:26:38.136 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:26:38.136 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T16:26:38.136 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:26:38.136 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:26:38.137 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:26:38.137 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:26:38.137 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:26:38.137 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.137 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:26:38.138 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.138 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:26:38.138 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:26:38.138 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:26:38.138 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:26:38.139 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:26:38.139 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:26:38.139 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:26:38.139 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:26:38.140 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:26:38.140 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:26:38.140 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:26:38.140 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:26:38.140 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:26:38.141 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:26:38.141 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:26:38.141 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:26:38.141 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.141 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.142 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:26:38.142 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:26:38.142 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:26:38.142 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.142 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:26:38.143 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:26:38.143 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.143 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:26:38.143 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:26:38.144 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:26:38.144 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:26:38.144 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:26:38.144 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:26:38.144 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.145 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:26:38.145 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2021-01-15T16:26:38.145 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:26:38.145 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:26:38.145 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:26:38.146 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:26:38.146 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:26:38.146 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:26:38.146 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:26:38.147 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:26:38.147 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:26:38.147 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:26:38.147 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:26:38.147 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:26:38.148 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:26:38.148 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:26:38.148 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:26:38.148 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:26:38.148 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:26:38.149 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:26:38.149 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:26:38.149 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:26:38.149 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:26:38.149 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:26:38.150 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:26:38.150 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:26:38.150 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:26:38.150 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:26:38.151 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:26:38.151 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:26:38.151 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:26:38.151 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:26:38.151 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:26:38.152 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:26:38.152 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:26:38.152 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:26:38.152 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:26:38.152 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:26:38.153 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:26:38.153 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:26:38.153 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:26:38.153 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:26:38.154 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:26:38.154 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:26:38.154 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:26:38.154 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:26:38.154 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.155 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:26:38.155 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:26:38.155 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:26:38.155 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:26:38.155 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:26:38.156 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:26:38.156 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:26:38.156 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:26:38.156 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:26:38.157 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:26:38.157 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:26:38.157 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.157 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2021-01-15T16:26:38.157 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2021-01-15T16:26:38.158 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:26:38.158 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:26:38.158 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:26:38.158 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.158 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:26:38.159 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:26:38.159 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:26:38.159 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:26:38.159 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:26:38.159 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:26:38.160 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:26:38.160 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:26:38.160 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18104, 2021-01-15T16:26:38.160 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2072, 2021-01-15T16:26:38.161 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:26:38.161 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 15998, 2021-01-15T16:26:38.161 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554696, 2021-01-15T16:26:38.161 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:26:38.161 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:26:38.162 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322104008704, 2021-01-15T16:26:38.162 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:26:38.162 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2121728, 2021-01-15T16:26:38.162 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 403228, 2021-01-15T16:26:38.162 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:26:38.163 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:26:38.163 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:26:38.163 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:26:38.163 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16382056 2021-01-15T16:26:38.164 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.164 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:26:38.164 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:26:38.164 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:26:38.164 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:26:38.165 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:26:38.165 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:26:38.165 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:26:38.165 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.165 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:26:38.166 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:26:38.166 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:26:38.166 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:26:38.166 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:26:38.167 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.167 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:26:38.167 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:26:38.167 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.167 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:26:38.168 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:26:38.168 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:26:38.168 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:26:38.168 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:26:38.168 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:26:38.169 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:26:38.169 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:26:38.169 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:26:38.169 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18104, 2021-01-15T16:26:38.169 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2072, 2021-01-15T16:26:38.170 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:26:38.170 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 15998, 2021-01-15T16:26:38.170 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554696, 2021-01-15T16:26:38.170 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:26:38.171 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:26:38.171 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322104008704, 2021-01-15T16:26:38.171 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:26:38.171 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2121728, 2021-01-15T16:26:38.171 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 403228, 2021-01-15T16:26:38.172 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:26:38.172 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:26:38.172 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:26:38.172 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:26:38.172 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16382056 2021-01-15T16:26:38.173 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.173 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:26:38.173 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:26:38.173 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:26:38.173 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:26:38.174 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:26:38.174 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:26:38.174 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:26:38.174 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.175 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:26:38.175 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:26:38.175 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:26:38.175 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:26:38.175 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:26:38.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.176 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:26:38.176 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:26:38.176 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.177 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:26:38.177 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.177 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:26:38.177 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:26:38.178 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:26:38.178 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:26:38.178 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:26:38.178 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:26:38.178 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:26:38.179 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:26:38.179 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:26:38.179 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:26:38.179 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:26:38.179 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:26:38.180 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:26:38.180 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:26:38.180 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:26:38.180 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:26:38.180 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:26:38.181 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:26:38.181 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:26:38.181 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:26:38.181 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:26:38.182 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:26:38.182 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:26:38.182 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:26:38.182 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:26:38.182 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:26:38.183 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:26:38.183 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:26:38.183 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:26:38.183 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:26:38.184 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:26:38.184 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:26:38.184 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:26:38.184 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:26:38.184 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:26:38.185 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:26:38.185 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:26:38.185 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:26:38.185 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:26:38.185 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:26:38.186 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:26:38.186 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:26:38.186 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:26:38.186 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.187 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:26:38.187 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:26:38.187 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:26:38.187 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:26:38.187 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1220608, 2021-01-15T16:26:38.188 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 143228, 2021-01-15T16:26:38.188 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:26:38.188 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:26:38.188 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:26:38.188 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:26:38.189 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:26:38.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.189 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 16, 2021-01-15T16:26:38.189 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 16, 2021-01-15T16:26:38.190 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:26:38.190 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:26:38.190 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:26:38.190 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.190 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.191 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:26:38.191 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.191 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:26:38.191 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855370 2021-01-15T16:26:38.191 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.192 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.192 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:26:38.192 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672977 2021-01-15T16:26:38.192 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.193 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:26:38.193 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822666 2021-01-15T16:26:38.193 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.193 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.194 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:26:38.194 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.194 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:26:38.194 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T16:26:38.194 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.195 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.195 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:26:38.195 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.195 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:26:38.196 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:26:38.196 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:26:38.196 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:26:38.196 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.197 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:26:38.197 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:26:38.197 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:26:38.197 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:26:38.197 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.198 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:26:38.198 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T16:26:38.198 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:26:38.198 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:26:38.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.199 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.199 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:26:38.199 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:26:38.200 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:26:38.200 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:26:38.200 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.200 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:26:38.200 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:26:38.201 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 4, 2021-01-15T16:26:38.201 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:26:03.415952+0000", 2021-01-15T16:26:38.201 INFO:tasks.workunit.client.0.smithi145.stdout: "services": { 2021-01-15T16:26:38.201 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": { 2021-01-15T16:26:38.201 INFO:tasks.workunit.client.0.smithi145.stdout: "daemons": { 2021-01-15T16:26:38.202 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": "", 2021-01-15T16:26:38.202 INFO:tasks.workunit.client.0.smithi145.stdout: "0": { 2021-01-15T16:26:38.202 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:26:38.202 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:26:38.203 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:26:38.203 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:26:38.203 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:26:38.203 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:26:38.203 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.204 INFO:tasks.workunit.client.0.smithi145.stdout: "2": { 2021-01-15T16:26:38.204 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:26:38.204 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:26:38.204 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:26:38.204 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:26:38.205 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:26:38.205 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:26:38.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.206 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.206 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:26:38.206 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:26:38.206 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:26:38.207 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 357, 2021-01-15T16:26:38.207 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:26:38.207 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:26:38.207 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:26:38.207 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:26:38.208 INFO:tasks.workunit.client.0.smithi145.stderr:report 2183844296 2021-01-15T16:26:38.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:26:38.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2021-01-15T16:26:38.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2021-01-15T16:26:38.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2021-01-15T16:26:38.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:26:38.212 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2021-01-15T16:26:38.330 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:26:38.330 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-01-15T16:26:01.755029+0000 29'100 39:121 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:26:38.331 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:26:38.331 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:26:38.332 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:26:38.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2021-01-15T16:26:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:26:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2021-01-15T16:26:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:26:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:26:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:26:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:26:38.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:26:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:26:38.464 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:26:38.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:26:38.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:26:38.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:26:38.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:26:38.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:26:38.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:26:38.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:26:38.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:26:38.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:26:38.914 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:26:38.915 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:26:38.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:26:38.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:38.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:26:39.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855373 2021-01-15T16:26:39.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855373 2021-01-15T16:26:39.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855373' 2021-01-15T16:26:39.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:39.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:26:39.159 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:26:39.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:26:39.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:26:39.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:26:39.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:26:39.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:26:39.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822668 2021-01-15T16:26:39.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822668 2021-01-15T16:26:39.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-150323855373 2-154618822668' 2021-01-15T16:26:39.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:26:39.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855373 2021-01-15T16:26:39.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:26:39.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:26:39.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-150323855373 2021-01-15T16:26:39.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:26:39.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855373 2021-01-15T16:26:39.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 150323855373' 2021-01-15T16:26:39.307 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 150323855373 2021-01-15T16:26:39.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:26:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855373 -lt 150323855373 2021-01-15T16:26:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:26:39.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822668 2021-01-15T16:26:39.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:26:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:26:39.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-154618822668 2021-01-15T16:26:39.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:26:39.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822668 2021-01-15T16:26:39.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2021-01-15T16:26:39.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:26:39.712 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 154618822668 2021-01-15T16:26:40.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822668 -lt 154618822668 2021-01-15T16:26:40.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:26:40.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:40.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:40.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:26:40.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:40.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:40.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:40.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:40.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:40.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:40.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:41.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:41.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:41.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:41.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:41.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:26:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:26:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:26:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:41.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:26:41.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:41.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:41.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:41.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:41.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:41.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:41.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:41.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:42.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:42.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:42.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:42.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:42.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:42.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:42.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:42.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:42.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:42.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:42.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:42.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:42.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:43.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:26:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:43.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:26:43.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:43.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:43.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:43.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:43.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:43.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:43.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:43.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:43.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:43.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:43.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:43.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:44.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:44.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:44.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:44.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:44.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:26:44.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:44.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:26:45.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:45.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:45.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:45.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:45.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:45.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:45.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:45.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:45.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:45.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:45.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:46.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:46.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:46.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:46.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:46.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:26:46.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:46.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:26:47.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:47.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:47.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:47.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:47.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:47.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:48.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:48.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:48.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:48.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:48.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:48.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:48.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:26:48.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:48.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:26:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:50.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:50.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:50.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:50.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:50.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:50.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:51.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:51.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:51.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:51.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:51.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:51.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:51.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:26:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:26:54.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:26:54.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:26:54.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:26:54.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:26:54.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:26:54.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:26:54.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:26:54.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:26:55.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:26:55.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:26:55.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:26:55.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:26:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:26:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:26:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:26:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:26:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:26:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:26:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:26:55.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:26:55.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:26:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:26:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:26:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:26:56.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:26:56.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:27:02.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:27:02.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:27:02.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:27:02.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:27:02.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:27:02.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:27:02.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:27:02.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:27:03.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:27:03.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:27:03.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:27:03.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:27:03.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:27:03.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:27:03.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:27:03.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:27:03.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:27:03.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:27:03.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:27:03.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:27:03.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:27:04.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:27:04.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:27:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:27:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:27:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:27:11.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:27:11.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:27:11.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:27:11.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:27:11.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:27:11.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:27:11.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:27:11.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:27:12.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:27:12.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:27:12.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:27:12.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:27:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:27:12.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:27:12.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:27:12.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:27:12.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:27:12.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:27:12.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:27:12.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:27:12.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:27:13.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:27:13.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:27:13.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:27:13.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:27:13.572 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:27:13.573 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "c816ef7c-8e57-40e8-8653-329194c60552", 2021-01-15T16:27:13.573 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:27:13.573 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:27:13.573 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:27:13.571788+0000", 2021-01-15T16:27:13.573 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:27:13.574 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:27:13.574 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:27:13.574 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:27:13.574 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:27:13.575 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:27:13.575 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:27:13.575 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:27:13.575 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:27:13.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:27:13.576 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:27:13.577 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:27:13.577 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:27:13.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.577 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:27:13.577 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.578 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:27:13.578 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.578 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.578 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:27:13.579 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.579 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:27:13.579 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:27:13.579 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:27:13.579 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2021-01-15T16:27:13.580 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 2 2021-01-15T16:27:13.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.580 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:27:13.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.580 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is stuck undersized for 71s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:27:13.581 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.581 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.581 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:27:13.581 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.581 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:27:13.582 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:27:13.582 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:27:13.582 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:27:13.582 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:27:13.582 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.583 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:27:13.583 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.583 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:27:13.583 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.583 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.584 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:27:13.585 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.585 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.585 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:27:13.585 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.586 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:27:13.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.586 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:27:13.587 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:27:13.587 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:27:13.587 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:27:13.587 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:27:13.587 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:24:44.065303Z", 2021-01-15T16:27:13.588 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:24:44.065303Z", 2021-01-15T16:27:13.588 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:27:13.588 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:27:13.588 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:27:13.588 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:27:13.589 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:27:13.589 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:27:13.589 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:27:13.590 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:27:13.590 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:27:13.590 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:27:13.590 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:27:13.591 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:27:13.591 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:27:13.591 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:27:13.593 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:27:13.593 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.593 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:27:13.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.593 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:27:13.594 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.594 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:27:13.594 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:27:13.594 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:27:13.595 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.595 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.595 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.595 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:27:13.595 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:27:13.596 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.596 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.596 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.596 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:27:13.596 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:27:13.597 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:27:13.597 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:27:13.597 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:27:13.597 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.597 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.598 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:27:13.598 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:27:13.598 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.598 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:27:13.599 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 39, 2021-01-15T16:27:13.599 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:27:13.599 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:24:44.820164+0000", 2021-01-15T16:27:13.599 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:26:01.452641+0000", 2021-01-15T16:27:13.599 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:25:59.881925+0000", 2021-01-15T16:27:13.600 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:26:01.201446+0000", 2021-01-15T16:27:13.600 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:27:13.600 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:27:13.600 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:27:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:27:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:27:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:27:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:27:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:27:13.602 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:27:13.602 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:27:13.602 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.602 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2021-01-15T16:27:13.602 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:27:13.603 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:27:13.603 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:27:13.603 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:27:13.603 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:27:13.604 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:27:13.604 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:27:13.604 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:27:13.604 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:27:13.604 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:27:13.606 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.606 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:27:13.606 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:27:13.606 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:25:11.336838+0000", 2021-01-15T16:27:13.606 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T16:27:13.607 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T16:27:13.607 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:27:13.607 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:27:13.607 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:27:13.608 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T16:27:13.608 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:27:13.608 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:27:13.608 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:27:13.608 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:27:13.609 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:27:13.609 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T16:27:13.609 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T16:27:13.609 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:27:13.610 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T16:27:13.610 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T16:27:13.610 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T16:27:13.610 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:27:13.611 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:27:13.611 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:27:13.611 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:27:13.611 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:27:13.612 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:27:13.612 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:27:13.612 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.612 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "29", 2021-01-15T16:27:13.612 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:27:13.613 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:27:13.613 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T16:27:13.613 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:27:13.613 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T16:27:13.614 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T16:27:13.615 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T16:27:13.615 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T16:27:13.615 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:27:13.616 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:27:13.616 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:27:13.616 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:27:13.616 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:27:13.616 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:27:13.617 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:27:13.617 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:27:13.617 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:27:13.617 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:27:13.617 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:27:13.618 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:27:13.618 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:27:13.618 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:27:13.618 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:27:13.619 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T16:27:13.619 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:27:13.619 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:27:13.619 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.620 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:27:13.620 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:27:13.620 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:27:13.620 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:27:13.620 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:27:13.621 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:27:13.621 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:27:13.621 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:27:13.621 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T16:27:13.621 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:27:13.622 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:27:13.622 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:27:13.622 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:27:13.622 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.623 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.623 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:27:13.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.623 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:27:13.623 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "c0cb32e2-1310-411a-82ce-6f897da57c9c", 2021-01-15T16:27:13.624 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:27:13.624 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:27:13.624 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:27:13.624 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:27:13.625 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:27:13.625 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:27:13.625 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 35, 2021-01-15T16:27:13.626 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 38, 2021-01-15T16:27:13.626 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 30, 2021-01-15T16:27:13.626 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:27:13.626 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:27:13.627 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.627 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.627 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:27:13.627 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.628 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.628 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.628 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:27:13.629 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.629 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.629 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.629 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:27:13.630 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.630 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.630 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:27:13.630 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.631 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.631 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.631 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:27:13.632 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.632 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.632 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.632 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:27:13.633 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.633 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.633 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:27:13.633 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.634 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.634 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.634 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:27:13.635 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.635 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.635 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.636 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:27:13.636 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.637 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.637 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:27:13.637 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.638 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.638 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:27:13.638 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52111 2021-01-15T16:27:13.638 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.639 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.639 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/52111", 2021-01-15T16:27:13.639 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/52111", 2021-01-15T16:27:13.639 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/52111", 2021-01-15T16:27:13.640 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/52111", 2021-01-15T16:27:13.640 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:27:13.640 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:27:13.640 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:27:13.640 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.641 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:27:13.641 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "1cc9002f-f77d-47e8-ab42-303305234f9f", 2021-01-15T16:27:13.641 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:27:13.642 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:27:13.642 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:27:13.642 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:27:13.642 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:27:13.643 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:27:13.643 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:27:13.643 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 23, 2021-01-15T16:27:13.643 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2021-01-15T16:27:13.643 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:27:13.644 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:27:13.644 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.644 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.644 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:27:13.645 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.646 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.646 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:27:13.646 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.647 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.647 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.647 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.647 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:27:13.647 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.648 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.648 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:27:13.648 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.648 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.649 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.649 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:27:13.649 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.650 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.650 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.650 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:27:13.650 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.651 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.651 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:27:13.651 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.652 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.652 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:27:13.652 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.652 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.653 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.653 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:27:13.653 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.654 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.654 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:27:13.654 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.655 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.655 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:27:13.655 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 46821 2021-01-15T16:27:13.656 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.656 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.657 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/46821", 2021-01-15T16:27:13.657 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/46821", 2021-01-15T16:27:13.657 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/46821", 2021-01-15T16:27:13.657 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/46821", 2021-01-15T16:27:13.658 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:27:13.658 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:27:13.658 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.658 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.659 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:27:13.659 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "3648422a-0c5d-4029-85e2-e7056e79a86d", 2021-01-15T16:27:13.659 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:27:13.659 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:27:13.660 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:27:13.660 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:27:13.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:27:13.660 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:27:13.660 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2021-01-15T16:27:13.661 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:27:13.661 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2021-01-15T16:27:13.661 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:27:13.661 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:27:13.662 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.662 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.662 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:27:13.662 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.663 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.663 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.663 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:27:13.663 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.664 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.664 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.664 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:27:13.665 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.665 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.665 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:27:13.665 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.666 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.666 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.667 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:27:13.667 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.667 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.667 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.668 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.668 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:27:13.668 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.669 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.669 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:27:13.669 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.669 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.670 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.670 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:27:13.670 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.670 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.671 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.671 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.671 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:27:13.671 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:27:13.671 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.672 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:27:13.672 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:27:13.672 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.673 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:27:13.673 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:27:13.673 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 52600 2021-01-15T16:27:13.673 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.673 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.674 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.674 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/52600", 2021-01-15T16:27:13.674 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/52600", 2021-01-15T16:27:13.674 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/52600", 2021-01-15T16:27:13.675 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/52600", 2021-01-15T16:27:13.675 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:27:13.675 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:27:13.675 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:27:13.675 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.676 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.676 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.676 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:27:13.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.676 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:27:13.677 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:53.008501+0000", 2021-01-15T16:27:13.677 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:27:13.678 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:27:13.678 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:27:13.678 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:27:13.678 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:24:53.491260+0000", 2021-01-15T16:27:13.679 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:27:13.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.679 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:27:13.679 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:53.561176+0000", 2021-01-15T16:27:13.680 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:27:13.680 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:27:13.680 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:27:13.680 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:27:13.681 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:25:01.279437+0000", 2021-01-15T16:27:13.681 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:27:13.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.681 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:27:13.682 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:25:54.112981+0000", 2021-01-15T16:27:13.682 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:27:13.682 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:27:13.682 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:27:13.683 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:27:13.683 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:25:09.017516+0000", 2021-01-15T16:27:13.683 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:27:13.683 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.683 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.684 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:27:13.684 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:27:13.684 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:27:13.684 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:27:13.684 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:27:13.685 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:27:13.685 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:27:13.685 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:27:13.685 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:27:13.685 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:27:13.686 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:27:13.686 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.686 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:27:13.687 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:27:13.687 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:27:13.687 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:27:13.687 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:27:13.687 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:27:13.688 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:27:13.688 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:27:13.688 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:27:13.688 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:27:13.688 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:27:13.689 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.689 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:27:13.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.690 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:27:13.690 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:27:13.690 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/52111,v1:127.0.0.1:6805/52111]", 2021-01-15T16:27:13.690 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:27:13.690 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:27:13.691 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:27:13.691 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:27:13.691 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:27:13.691 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:27:13.691 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:27:13.692 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:27:13.692 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:27:13.692 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:27:13.692 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:27:13.692 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:27:13.693 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:27:13.693 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:27:13.693 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:27:13.693 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:27:13.694 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:27:13.694 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:27:13.694 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:27:13.694 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:27:13.694 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:27:13.695 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:27:13.695 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:27:13.695 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:27:13.695 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:27:13.695 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/52111,v1:127.0.0.1:6803/52111]", 2021-01-15T16:27:13.696 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:27:13.696 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/52111,v1:127.0.0.1:6809/52111]", 2021-01-15T16:27:13.696 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/52111,v1:127.0.0.1:6807/52111]", 2021-01-15T16:27:13.696 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:27:13.697 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:27:13.697 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:27:13.697 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:27:13.697 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:27:13.697 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:27:13.698 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:27:13.698 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:27:13.698 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:27:13.698 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:27:13.698 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:27:13.699 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:27:13.699 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:27:13.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.699 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.700 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:27:13.700 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:27:13.700 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/46821,v1:127.0.0.1:6813/46821]", 2021-01-15T16:27:13.700 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:27:13.700 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:27:13.701 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:27:13.701 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:27:13.701 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:27:13.701 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:27:13.701 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:27:13.702 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:27:13.702 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:27:13.702 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:27:13.702 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:27:13.703 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:27:13.703 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:27:13.703 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:27:13.703 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:27:13.703 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:27:13.704 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:27:13.704 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:27:13.704 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:27:13.704 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:27:13.704 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:27:13.705 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:27:13.705 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:27:13.705 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:27:13.705 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:27:13.705 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/46821,v1:127.0.0.1:6811/46821]", 2021-01-15T16:27:13.706 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:27:13.706 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/46821,v1:127.0.0.1:6817/46821]", 2021-01-15T16:27:13.706 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/46821,v1:127.0.0.1:6815/46821]", 2021-01-15T16:27:13.706 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:27:13.707 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:27:13.707 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:27:13.707 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:27:13.707 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:27:13.707 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:27:13.708 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:27:13.708 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:27:13.708 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:27:13.708 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:27:13.708 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:27:13.709 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:27:13.709 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:27:13.709 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.709 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:27:13.710 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:27:13.710 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/52600,v1:127.0.0.1:6813/52600]", 2021-01-15T16:27:13.710 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:27:13.710 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:27:13.711 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:27:13.711 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:27:13.711 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:27:13.711 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:27:13.711 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:27:13.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:27:13.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:27:13.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:27:13.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:27:13.712 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:27:13.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:27:13.713 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:27:13.713 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:27:13.713 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:27:13.714 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:27:13.714 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:27:13.714 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:27:13.714 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:27:13.714 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:27:13.715 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:27:13.715 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:27:13.715 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:27:13.715 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:27:13.715 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/52600,v1:127.0.0.1:6811/52600]", 2021-01-15T16:27:13.716 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:27:13.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/52600,v1:127.0.0.1:6817/52600]", 2021-01-15T16:27:13.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/52600,v1:127.0.0.1:6815/52600]", 2021-01-15T16:27:13.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:27:13.716 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:27:13.717 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:27:13.717 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:27:13.717 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:27:13.717 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:27:13.718 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:27:13.718 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:27:13.718 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:27:13.718 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:27:13.718 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:27:13.719 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:27:13.719 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:27:13.719 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.719 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.719 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:27:13.720 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:27:13.720 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:27:13.720 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:27:13.720 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.721 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:27:13.721 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.721 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:27:13.721 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 35 2021-01-15T16:27:13.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.722 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.722 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:27:13.722 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:27:13.722 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.722 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.723 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:27:13.723 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 37 2021-01-15T16:27:13.723 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.723 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.723 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.724 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:27:13.724 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 39, 2021-01-15T16:27:13.724 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:27:13.724 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:27:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:27:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:27:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:27:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:27:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:27:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:27:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:27:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:27:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:27:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:27:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:27:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:27:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:27:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:27:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:27:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:27:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.731 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.731 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:27:13.731 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:27:13.731 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.732 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:27:13.732 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:27:13.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.733 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:27:13.733 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:27:13.733 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.733 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:27:13.734 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:27:13.734 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.734 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.734 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:27:13.734 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:27:13.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.735 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:27:13.735 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:27:13.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.736 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:27:13.736 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:27:13.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:27:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:27:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.738 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:27:13.738 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:27:13.738 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.738 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.738 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:27:13.739 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.739 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:27:13.739 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:27:13.739 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:27:13.740 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:27:13.740 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.740 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:27:13.740 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:27:13.740 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:27:13.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.741 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:27:13.741 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.741 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:27:13.741 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.742 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.742 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:27:13.742 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:27:13.743 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:27:13.743 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:27:13.743 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.743 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:27:13.744 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:27:13.744 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:27:13.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.744 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:27:13.744 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.745 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:27:13.745 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.745 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.745 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.746 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:27:13.746 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:27:13.746 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:27:13.746 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:27:13.746 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.747 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:27:13.747 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:27:13.747 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:27:13.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.748 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:27:13.748 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.748 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:27:13.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.749 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:27:13.749 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.749 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:27:13.749 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.750 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:27:13.750 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.750 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:27:13.750 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.751 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.751 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.751 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.751 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:27:13.751 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:27:13.752 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:27:13.752 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:27:13.752 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:27:13.752 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:27:13.752 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:27:13.753 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:27:13.753 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.753 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:27:13.753 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.753 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:27:13.754 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.754 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.754 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:27:13.754 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.755 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:27:13.755 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.755 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.755 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:27:13.755 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:27:13.756 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:27:13.756 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.756 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.756 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.756 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.757 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:27:13.757 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.757 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:27:13.757 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:27:13.757 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:27:13.758 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:27:13.758 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:27:13.758 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:27:13.758 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:27:13.759 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.759 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:27:13.759 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:27:13.759 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:27:13.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.760 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.760 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:27:13.760 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:27:13.760 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:27:13.760 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.761 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:27:13.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.761 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:27:13.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.762 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.762 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:27:13.762 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:27:13.762 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:27:13.763 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:27:13.763 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:27:13.763 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:27:13.763 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:27:13.763 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:27:13.764 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:27:13.764 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:27:13.764 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:27:13.764 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:27:13.764 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:27:13.765 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:27:13.765 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:27:13.765 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T16:27:13.765 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:27:13.765 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:27:13.766 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:27:13.766 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:27:13.766 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:27:13.766 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.767 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:27:13.767 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.767 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:27:13.767 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:27:13.767 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:27:13.768 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:27:13.768 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:27:13.768 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:27:13.768 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:27:13.768 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:27:13.769 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:27:13.769 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:27:13.769 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:27:13.769 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:27:13.770 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:27:13.770 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:27:13.770 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:27:13.770 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:27:13.770 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.771 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.771 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:27:13.771 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:27:13.771 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:27:13.772 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.772 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:27:13.772 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:27:13.772 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.772 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:27:13.773 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:27:13.773 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:27:13.773 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:27:13.773 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:27:13.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:27:13.774 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:27:13.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2021-01-15T16:27:13.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:27:13.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:27:13.776 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:27:13.776 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:27:13.777 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2021-01-15T16:27:13.777 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:27:13.777 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:27:13.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:27:13.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:27:13.778 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:27:13.779 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:27:13.779 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:27:13.779 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:27:13.780 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:27:13.780 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:27:13.780 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:27:13.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:27:13.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:27:13.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:27:13.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:27:13.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:27:13.782 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:27:13.782 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:27:13.782 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:27:13.782 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:27:13.783 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:27:13.783 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:27:13.783 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:27:13.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:27:13.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:27:13.784 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:27:13.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:27:13.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:27:13.786 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:27:13.786 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:27:13.786 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:27:13.787 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:27:13.787 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:27:13.788 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:27:13.788 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:27:13.788 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:27:13.789 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:27:13.789 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:27:13.790 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:27:13.790 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.790 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:27:13.791 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:27:13.791 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:27:13.792 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:27:13.792 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:27:13.792 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:27:13.793 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:27:13.793 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:27:13.794 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:27:13.794 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:27:13.794 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:27:13.795 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.795 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 17, 2021-01-15T16:27:13.796 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 17, 2021-01-15T16:27:13.796 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:27:13.797 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:27:13.797 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:27:13.797 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.798 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:27:13.798 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:27:13.799 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:27:13.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:27:13.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:27:13.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:27:13.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:27:13.801 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:27:13.801 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18104, 2021-01-15T16:27:13.801 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2072, 2021-01-15T16:27:13.802 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:27:13.802 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 15998, 2021-01-15T16:27:13.803 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554696, 2021-01-15T16:27:13.803 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:27:13.803 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:27:13.804 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322104008704, 2021-01-15T16:27:13.804 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:27:13.805 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2121728, 2021-01-15T16:27:13.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 402522, 2021-01-15T16:27:13.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:27:13.806 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:27:13.806 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:27:13.807 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:27:13.807 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16382056 2021-01-15T16:27:13.807 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.808 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:27:13.808 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:27:13.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:27:13.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:27:13.809 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:27:13.810 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:27:13.810 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:27:13.811 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.811 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:27:13.812 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:27:13.812 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:27:13.812 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:27:13.813 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:27:13.813 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.814 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:27:13.814 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:27:13.814 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.815 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:27:13.815 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:27:13.816 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:27:13.816 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:27:13.816 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:27:13.817 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:27:13.817 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:27:13.818 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:27:13.818 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:27:13.819 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18104, 2021-01-15T16:27:13.819 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2072, 2021-01-15T16:27:13.819 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:27:13.820 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 15998, 2021-01-15T16:27:13.820 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554696, 2021-01-15T16:27:13.821 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:27:13.821 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:27:13.821 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322104008704, 2021-01-15T16:27:13.822 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:27:13.822 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2121728, 2021-01-15T16:27:13.823 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 402522, 2021-01-15T16:27:13.823 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:27:13.823 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:27:13.824 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:27:13.824 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:27:13.825 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16382056 2021-01-15T16:27:13.825 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.825 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:27:13.826 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:27:13.826 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:27:13.827 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:27:13.827 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:27:13.827 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:27:13.828 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:27:13.828 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.828 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:27:13.829 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:27:13.829 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:27:13.830 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:27:13.830 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:27:13.830 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.831 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:27:13.831 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:27:13.832 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.832 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.832 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:27:13.833 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.833 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:27:13.834 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:27:13.834 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:27:13.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2021-01-15T16:27:13.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:27:13.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:27:13.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:27:13.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:27:13.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:27:13.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:27:13.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:27:13.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:27:13.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:27:13.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:27:13.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:27:13.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:27:13.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:27:13.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:27:13.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:27:13.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:27:13.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:27:13.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:27:13.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:27:13.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:27:13.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:27:13.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:27:13.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:27:13.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:27:13.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:27:13.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:27:13.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:27:13.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:27:13.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:27:13.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:27:13.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:27:13.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:27:13.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:27:13.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:27:13.844 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.844 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:27:13.844 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:27:13.844 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:27:13.844 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:27:13.845 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1220608, 2021-01-15T16:27:13.845 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 142522, 2021-01-15T16:27:13.845 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:27:13.845 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:27:13.845 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:27:13.846 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:27:13.846 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:27:13.846 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.846 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 17, 2021-01-15T16:27:13.846 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 17, 2021-01-15T16:27:13.847 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 2, 2021-01-15T16:27:13.847 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 2, 2021-01-15T16:27:13.847 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:27:13.847 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.847 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.848 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:27:13.848 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.848 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:27:13.848 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 150323855380 2021-01-15T16:27:13.848 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.849 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.849 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:27:13.849 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672977 2021-01-15T16:27:13.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.849 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.850 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:27:13.850 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822675 2021-01-15T16:27:13.850 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.850 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.850 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:27:13.851 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.851 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:27:13.851 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T16:27:13.851 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.851 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:27:13.852 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.852 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:27:13.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:27:13.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:27:13.853 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:27:13.853 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.853 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.853 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:27:13.853 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:27:13.854 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:27:13.854 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:27:13.854 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.854 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.854 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:27:13.855 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T16:27:13.855 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:27:13.855 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:27:13.855 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.855 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.856 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:27:13.857 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:27:13.857 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 6, 2021-01-15T16:27:13.857 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:27:01.426875+0000", 2021-01-15T16:27:13.857 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2021-01-15T16:27:13.857 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:27:13.858 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:27:13.858 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:27:13.858 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 446, 2021-01-15T16:27:13.858 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:27:13.858 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:27:13.859 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:27:13.859 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:27:13.860 INFO:tasks.workunit.client.0.smithi145.stderr:report 2931258044 2021-01-15T16:27:13.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:27:13.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2021-01-15T16:27:13.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:13.863 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2021-01-15T16:27:13.957 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:13.958 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-01-15T16:26:01.755029+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:13.958 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:13.958 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:13.959 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:13.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:27:13.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2021-01-15T16:27:13.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:27:13.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:27:13.970 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:27:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:13.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:27:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:27:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok set_recovery_delay 100000 2021-01-15T16:27:14.073 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:27:14.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2021-01-15T16:27:14.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2021-01-15T16:27:14.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:27:14.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:27:14.084 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:27:14.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:14.085 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:14.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:27:14.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:27:14.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok set_recovery_delay 100000 2021-01-15T16:27:14.185 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:27:14.194 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2021-01-15T16:27:14.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2021-01-15T16:27:14.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:14.563 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:14.564 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-01-15T16:26:01.755029+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:14.564 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:14.564 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:14.565 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:14.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2021-01-15T16:27:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2021-01-15T16:27:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-01-15T16:27:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:27:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:27:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:27:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:27:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:27:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:27:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:27:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:27:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:27:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:27:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:27:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:27:15.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:27:15.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:27:15.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:27:15.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:27:15.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:27:15.270 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:27:15.270 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:15.270 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:15.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:27:15.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:27:15.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:27:15.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:27:15.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:27:15.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:27:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:27:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:27:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:27:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:27:15.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:27:15.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:27:15.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:27:15.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:27:15.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:27:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:27:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:27:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:27:15.274 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:27:15.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:15.291+0000 7f4d3c6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:15.293 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:15.293+0000 7f4d3c6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:15.295 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:15.294+0000 7f4d3c6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2021-01-15T16:27:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2021-01-15T16:27:15.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:15.811+0000 7f4d3c6de080 -1 Falling back to public interface 2021-01-15T16:27:17.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:17.005+0000 7f4d3c6de080 -1 osd.1 29 log_to_monitors {default=true} 2021-01-15T16:27:20.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2021-01-15T16:27:20.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:20.683 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2021-01-15T16:27:21.065 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:21.065 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2021-01-15T16:26:01.755029+0000 39'101 40:124 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:21.065 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:21.066 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:21.067 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:21.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2021-01-15T16:27:21.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:27:21.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:27:21.078 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:27:21.078 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:21.078 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:21.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:27:21.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:27:21.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok set_recovery_delay 100000 2021-01-15T16:27:21.180 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:27:21.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2021-01-15T16:27:21.869 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2021-01-15T16:27:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2021-01-15T16:27:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:27:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:27:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:27:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:27:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:27:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:27:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:27:21.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:27:21.881 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:27:22.350 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/52111,v1:127.0.0.1:6803/52111] [v2:127.0.0.1:6804/52111,v1:127.0.0.1:6805/52111] exists,up c0cb32e2-1310-411a-82ce-6f897da57c9c 2021-01-15T16:27:22.351 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:27:22.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:27:22.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:27:22.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:27:22.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2021-01-15T16:27:22.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:27:22.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:27:22.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:27:22.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:27:22.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:27:22.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:27:22.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:27:22.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:27:22.747 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/55197,v1:127.0.0.1:6819/55197] [v2:127.0.0.1:6820/55197,v1:127.0.0.1:6821/55197] exists,up 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:27:22.747 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:27:22.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:27:22.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:27:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:27:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2021-01-15T16:27:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:27:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:27:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:27:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:27:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:27:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:27:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:27:22.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:27:23.159 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/52600,v1:127.0.0.1:6811/52600] [v2:127.0.0.1:6812/52600,v1:127.0.0.1:6813/52600] exists,up 3648422a-0c5d-4029-85e2-e7056e79a86d 2021-01-15T16:27:23.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:27:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:27:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:27:23.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:23.534 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:23.534 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2021-01-15T16:27:21.971580+0000 29'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:23.535 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:23.535 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:23.536 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:23.546 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2021-01-15T16:27:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2021-01-15T16:27:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:23.918 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:23.918 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 peering 2021-01-15T16:27:21.971580+0000 29'106 43:117 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:23.918 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:23.918 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:23.919 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:23.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2021-01-15T16:27:29.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2021-01-15T16:27:29.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:29.844 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2021-01-15T16:27:30.222 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:30.223 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-01-15T16:27:22.462358+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:30.223 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2021-01-15T16:27:30.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:27:30.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:27:30.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:27:30.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:27:30.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:27:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:27:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.33425.out 2021-01-15T16:27:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33425.out 2021-01-15T16:27:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:27:30.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:27:30.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:27:30.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:27:30.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-01-15T16:27:30.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33425.out 2021-01-15T16:27:31.067 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (29'90,44'102] local-lis/les=43/44 n=101 ec=23/23 lis/c=43/23 les/c/f=44/24/0 sis=43 pruub=7296358323ns) 2021-01-15T16:27:31.082 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:023dd3d2:::existing_12:head# 2021-01-15T16:27:31.083 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04c40a0c:::existing_93:head# 2021-01-15T16:27:31.086 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:064ff5fb:::existing_36:head# 2021-01-15T16:27:31.086 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:091f9f1d:::existing_94:head# 2021-01-15T16:27:31.087 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0a865a93:::existing_54:head# 2021-01-15T16:27:31.087 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0fed513c:::existing_41:head# 2021-01-15T16:27:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13b9e0b8:::existing_92:head# 2021-01-15T16:27:31.094 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:169157fa:::existing_81:head# 2021-01-15T16:27:31.094 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17731382:::existing_58:head# 2021-01-15T16:27:31.095 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:18a45a39:::existing_43:head# 2021-01-15T16:27:31.096 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1c160f01:::existing_66:head# 2021-01-15T16:27:31.096 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1dd6c249:::existing_97:head# 2021-01-15T16:27:31.097 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:227e0c77:::existing_7:head# 2021-01-15T16:27:31.097 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:23e1b063:::existing_59:head# 2021-01-15T16:27:31.098 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:28566edb:::existing_78:head# 2021-01-15T16:27:31.098 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2a101182:::existing_20:head# 2021-01-15T16:27:31.099 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2d1519da:::existing_96:head# 2021-01-15T16:27:31.099 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3078b87a:::existing_86:head# 2021-01-15T16:27:31.100 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:30bde01b:::existing_15:head# 2021-01-15T16:27:31.101 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:317a7e19:::existing_56:head# 2021-01-15T16:27:31.101 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:38b9c7fa:::existing_33:head# 2021-01-15T16:27:31.101 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3e16d84b:::existing_48:head# 2021-01-15T16:27:31.102 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:45fe8203:::existing_75:head# 2021-01-15T16:27:31.102 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4e82c9ba:::existing_50:head# 2021-01-15T16:27:31.103 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5298dcdb:::existing_57:head# 2021-01-15T16:27:31.103 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:52ebaa44:::existing_32:head# 2021-01-15T16:27:31.103 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:53588912:::existing_9:head# 2021-01-15T16:27:31.104 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5b7e2de6:::existing_84:head# 2021-01-15T16:27:31.105 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5da0255c:::existing_13:head# 2021-01-15T16:27:31.105 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5da2dab8:::existing_53:head# 2021-01-15T16:27:31.105 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5ee3514b:::existing_30:head# 2021-01-15T16:27:31.105 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:602f83fe:::foo:head# 2021-01-15T16:27:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:60426cc7:::existing_62:head# 2021-01-15T16:27:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:628918ca:::existing_79:head# 2021-01-15T16:27:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:63f8ab2d:::existing_77:head# 2021-01-15T16:27:31.108 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:67e1780f:::existing_27:head# 2021-01-15T16:27:31.108 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:68143ca1:::existing_71:head# 2021-01-15T16:27:31.109 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6bac9509:::existing_29:head# 2021-01-15T16:27:31.109 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6bbec798:::existing_45:head# 2021-01-15T16:27:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6cb23a40:::existing_65:head# 2021-01-15T16:27:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7055ec5e:::existing_83:head# 2021-01-15T16:27:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:70a54e2b:::existing_87:head# 2021-01-15T16:27:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:750ace21:::existing_19:head# 2021-01-15T16:27:31.112 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:78623b1d:::existing_6:head# 2021-01-15T16:27:31.112 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:79881152:::existing_63:head# 2021-01-15T16:27:31.112 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7a80c0d5:::existing_99:head# 2021-01-15T16:27:31.112 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:81490677:::existing_76:head# 2021-01-15T16:27:31.113 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:85ec8e5d:::existing_85:head# 2021-01-15T16:27:31.113 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8e2abbf0:::existing_49:head# 2021-01-15T16:27:31.114 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8ee0fe0f:::existing_52:head# 2021-01-15T16:27:31.114 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:923c5d4d:::existing_5:head# 2021-01-15T16:27:31.115 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:92731e71:::existing_89:head# 2021-01-15T16:27:31.115 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:928205cf:::existing_82:head# 2021-01-15T16:27:31.115 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:93be2267:::existing_24:head# 2021-01-15T16:27:31.115 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:99ceda2c:::existing_44:head# 2021-01-15T16:27:31.116 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9dd917ee:::existing_26:head# 2021-01-15T16:27:31.116 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a28805a6:::existing_72:head# 2021-01-15T16:27:31.117 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a294a8f6:::existing_16:head# 2021-01-15T16:27:31.117 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a2f5473a:::existing_42:head# 2021-01-15T16:27:31.118 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a504a533:::existing_90:head# 2021-01-15T16:27:31.127 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a691337e:::existing_61:head# 2021-01-15T16:27:31.132 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a6e8d7ac:::existing_28:head# 2021-01-15T16:27:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a9fde86e:::existing_18:head# 2021-01-15T16:27:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:aa309faa:::existing_68:head# 2021-01-15T16:27:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ae40b0ed:::existing_31:head# 2021-01-15T16:27:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b279b1fe:::existing_10:head# 2021-01-15T16:27:31.135 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b343e277:::existing_88:head# 2021-01-15T16:27:31.135 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b7130e90:::existing_80:head# 2021-01-15T16:27:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bc6f7309:::existing_70:head# 2021-01-15T16:27:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:be8489b7:::existing_38:head# 2021-01-15T16:27:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:beb4e5e3:::existing_47:head# 2021-01-15T16:27:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bebec877:::existing_74:head# 2021-01-15T16:27:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c04347d8:::existing_60:head# 2021-01-15T16:27:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c156033e:::existing_100:head# 2021-01-15T16:27:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c19f5a8e:::existing_8:head# 2021-01-15T16:27:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c3e6ebe4:::existing_17:head# 2021-01-15T16:27:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c6c5a7c2:::existing_39:head# 2021-01-15T16:27:31.140 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cc1d0abf:::existing_95:head# 2021-01-15T16:27:31.141 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ccb3865b:::existing_73:head# 2021-01-15T16:27:31.141 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cd84e19e:::existing_40:head# 2021-01-15T16:27:31.141 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d2cd64ce:::existing_37:head# 2021-01-15T16:27:31.142 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d41a25c9:::existing_22:head# 2021-01-15T16:27:31.142 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d7f088da:::existing_91:head# 2021-01-15T16:27:31.143 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d9cbd1d6:::existing_64:head# 2021-01-15T16:27:31.143 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dcc4ff82:::existing_98:head# 2021-01-15T16:27:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dd677bd3:::existing_35:head# 2021-01-15T16:27:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dd6d90e7:::existing_46:head# 2021-01-15T16:27:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e0e22077:::existing_21:head# 2021-01-15T16:27:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e255b710:::existing_55:head# 2021-01-15T16:27:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:eff1d5db:::existing_14:head# 2021-01-15T16:27:31.146 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f1b156bf:::existing_23:head# 2021-01-15T16:27:31.146 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5ce4199:::existing_34:head# 2021-01-15T16:27:31.147 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f6809456:::existing_11:head# 2021-01-15T16:27:31.147 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f699a2a9:::existing_67:head# 2021-01-15T16:27:31.148 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f9baac43:::existing_25:head# 2021-01-15T16:27:31.148 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fb76688e:::existing_51:head# 2021-01-15T16:27:31.149 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fbb92eec:::existing_69:head# 2021-01-15T16:27:31.149 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2021-01-15T16:27:31.149 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2021-01-15T16:27:31.149 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2021-01-15T16:27:31.149 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-01-15T16:27:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:Remove successful 2021-01-15T16:27:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33425.out 2021-01-15T16:27:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:27:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:27:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:27:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:27:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-01-15T16:27:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33425.out 2021-01-15T16:27:32.352 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={23=1}} deleted_pools ) 2021-01-15T16:27:32.352 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 1.0 2021-01-15T16:27:32.353 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:023dd3d2:::existing_12:head# 2021-01-15T16:27:32.353 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.444 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04c40a0c:::existing_93:head# 2021-01-15T16:27:32.444 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.486 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:064ff5fb:::existing_36:head# 2021-01-15T16:27:32.486 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.519 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:091f9f1d:::existing_94:head# 2021-01-15T16:27:32.520 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.553 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0a865a93:::existing_54:head# 2021-01-15T16:27:32.553 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.586 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0fed513c:::existing_41:head# 2021-01-15T16:27:32.587 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.620 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13b9e0b8:::existing_92:head# 2021-01-15T16:27:32.620 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.653 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:169157fa:::existing_81:head# 2021-01-15T16:27:32.654 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.686 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17731382:::existing_58:head# 2021-01-15T16:27:32.687 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.720 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:18a45a39:::existing_43:head# 2021-01-15T16:27:32.720 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.753 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1c160f01:::existing_66:head# 2021-01-15T16:27:32.753 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.804 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1dd6c249:::existing_97:head# 2021-01-15T16:27:32.804 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.837 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:227e0c77:::existing_7:head# 2021-01-15T16:27:32.837 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.871 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:23e1b063:::existing_59:head# 2021-01-15T16:27:32.871 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.904 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:28566edb:::existing_78:head# 2021-01-15T16:27:32.904 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.937 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2a101182:::existing_20:head# 2021-01-15T16:27:32.938 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:32.971 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2d1519da:::existing_96:head# 2021-01-15T16:27:32.971 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.005 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3078b87a:::existing_86:head# 2021-01-15T16:27:33.005 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.038 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:30bde01b:::existing_15:head# 2021-01-15T16:27:33.039 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.105 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:317a7e19:::existing_56:head# 2021-01-15T16:27:33.105 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.138 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:38b9c7fa:::existing_33:head# 2021-01-15T16:27:33.138 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.172 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3e16d84b:::existing_48:head# 2021-01-15T16:27:33.173 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.205 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:45fe8203:::existing_75:head# 2021-01-15T16:27:33.206 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.248 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4e82c9ba:::existing_50:head# 2021-01-15T16:27:33.248 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5298dcdb:::existing_57:head# 2021-01-15T16:27:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:52ebaa44:::existing_32:head# 2021-01-15T16:27:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:53588912:::existing_9:head# 2021-01-15T16:27:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5b7e2de6:::existing_84:head# 2021-01-15T16:27:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.415 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5da0255c:::existing_13:head# 2021-01-15T16:27:33.415 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.448 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5da2dab8:::existing_53:head# 2021-01-15T16:27:33.449 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.482 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5ee3514b:::existing_30:head# 2021-01-15T16:27:33.482 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.540 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:602f83fe:::foo:head# 2021-01-15T16:27:33.541 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.574 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:60426cc7:::existing_62:head# 2021-01-15T16:27:33.575 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.599 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:628918ca:::existing_79:head# 2021-01-15T16:27:33.599 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.624 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:63f8ab2d:::existing_77:head# 2021-01-15T16:27:33.624 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.649 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:67e1780f:::existing_27:head# 2021-01-15T16:27:33.649 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.674 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:68143ca1:::existing_71:head# 2021-01-15T16:27:33.674 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.749 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6bac9509:::existing_29:head# 2021-01-15T16:27:33.750 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.816 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6bbec798:::existing_45:head# 2021-01-15T16:27:33.817 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.841 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6cb23a40:::existing_65:head# 2021-01-15T16:27:33.842 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.875 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7055ec5e:::existing_83:head# 2021-01-15T16:27:33.875 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.908 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:70a54e2b:::existing_87:head# 2021-01-15T16:27:33.909 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.942 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:750ace21:::existing_19:head# 2021-01-15T16:27:33.942 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:33.976 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:78623b1d:::existing_6:head# 2021-01-15T16:27:33.976 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.009 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:79881152:::existing_63:head# 2021-01-15T16:27:34.009 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.042 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7a80c0d5:::existing_99:head# 2021-01-15T16:27:34.042 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.076 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:81490677:::existing_76:head# 2021-01-15T16:27:34.076 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.109 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:85ec8e5d:::existing_85:head# 2021-01-15T16:27:34.110 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.142 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8e2abbf0:::existing_49:head# 2021-01-15T16:27:34.143 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.176 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8ee0fe0f:::existing_52:head# 2021-01-15T16:27:34.176 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.218 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:923c5d4d:::existing_5:head# 2021-01-15T16:27:34.218 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.251 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:92731e71:::existing_89:head# 2021-01-15T16:27:34.252 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.285 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:928205cf:::existing_82:head# 2021-01-15T16:27:34.285 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.318 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:93be2267:::existing_24:head# 2021-01-15T16:27:34.318 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.377 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:99ceda2c:::existing_44:head# 2021-01-15T16:27:34.377 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.410 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9dd917ee:::existing_26:head# 2021-01-15T16:27:34.411 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.443 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a28805a6:::existing_72:head# 2021-01-15T16:27:34.444 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.477 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a294a8f6:::existing_16:head# 2021-01-15T16:27:34.477 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.510 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a2f5473a:::existing_42:head# 2021-01-15T16:27:34.511 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.544 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a504a533:::existing_90:head# 2021-01-15T16:27:34.544 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.577 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a691337e:::existing_61:head# 2021-01-15T16:27:34.578 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.611 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a6e8d7ac:::existing_28:head# 2021-01-15T16:27:34.611 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.653 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a9fde86e:::existing_18:head# 2021-01-15T16:27:34.653 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.695 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:aa309faa:::existing_68:head# 2021-01-15T16:27:34.695 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.728 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ae40b0ed:::existing_31:head# 2021-01-15T16:27:34.728 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.770 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b279b1fe:::existing_10:head# 2021-01-15T16:27:34.770 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.803 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b343e277:::existing_88:head# 2021-01-15T16:27:34.804 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.837 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b7130e90:::existing_80:head# 2021-01-15T16:27:34.837 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.871 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bc6f7309:::existing_70:head# 2021-01-15T16:27:34.871 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.903 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:be8489b7:::existing_38:head# 2021-01-15T16:27:34.904 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.937 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:beb4e5e3:::existing_47:head# 2021-01-15T16:27:34.937 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:34.971 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bebec877:::existing_74:head# 2021-01-15T16:27:34.971 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.037 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c04347d8:::existing_60:head# 2021-01-15T16:27:35.038 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.071 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c156033e:::existing_100:head# 2021-01-15T16:27:35.071 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.104 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c19f5a8e:::existing_8:head# 2021-01-15T16:27:35.105 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.139 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c3e6ebe4:::existing_17:head# 2021-01-15T16:27:35.139 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.171 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c6c5a7c2:::existing_39:head# 2021-01-15T16:27:35.171 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.205 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cc1d0abf:::existing_95:head# 2021-01-15T16:27:35.205 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.238 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ccb3865b:::existing_73:head# 2021-01-15T16:27:35.238 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.271 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cd84e19e:::existing_40:head# 2021-01-15T16:27:35.272 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.305 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d2cd64ce:::existing_37:head# 2021-01-15T16:27:35.305 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.338 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d41a25c9:::existing_22:head# 2021-01-15T16:27:35.339 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.366 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d7f088da:::existing_91:head# 2021-01-15T16:27:35.366 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.399 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d9cbd1d6:::existing_64:head# 2021-01-15T16:27:35.400 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.424 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dcc4ff82:::existing_98:head# 2021-01-15T16:27:35.424 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.463 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dd677bd3:::existing_35:head# 2021-01-15T16:27:35.464 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.542 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dd6d90e7:::existing_46:head# 2021-01-15T16:27:35.542 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.637 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e0e22077:::existing_21:head# 2021-01-15T16:27:35.637 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.684 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e255b710:::existing_55:head# 2021-01-15T16:27:35.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.751 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:eff1d5db:::existing_14:head# 2021-01-15T16:27:35.751 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.818 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f1b156bf:::existing_23:head# 2021-01-15T16:27:35.818 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.868 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5ce4199:::existing_34:head# 2021-01-15T16:27:35.868 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.918 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f6809456:::existing_11:head# 2021-01-15T16:27:35.919 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:35.968 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f699a2a9:::existing_67:head# 2021-01-15T16:27:35.969 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:36.019 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f9baac43:::existing_25:head# 2021-01-15T16:27:36.019 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:36.129 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fb76688e:::existing_51:head# 2021-01-15T16:27:36.129 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:36.186 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fbb92eec:::existing_69:head# 2021-01-15T16:27:36.187 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:27:36.228 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (29'90,44'102] local-lis/les=43/44 n=101 ec=23/23 lis/c=43/23 les/c/f=44/24/0 sis=43 pruub=7296358323ns) 2021-01-15T16:27:36.311 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2021-01-15T16:27:36.831 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2021-01-15T16:27:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2021-01-15T16:27:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:37.214 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:37.215 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2021-01-15T16:27:22.462358+0000 39'101 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:37.215 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:37.215 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2021-01-15T16:27:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:27:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:27:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:27:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:27:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:27:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:27:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:27:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:27:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:27:37.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:27:37.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:27:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:27:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:27:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:27:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:27:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:27:37.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:27:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:27:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:37.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:27:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:27:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:27:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:27:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:27:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:27:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:27:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:27:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:27:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:27:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:27:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:27:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:27:37.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:27:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:27:37.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:27:37.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:27:37.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:27:37.237 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:27:37.257 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:37.256+0000 7ff166cac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:37.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:37.258+0000 7ff166cac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:37.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:37.259+0000 7ff166cac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:27:37.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:27:37.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:27:37.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:27:37.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:27:37.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:27:37.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:27:37.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:27:37.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:27:37.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:27:37.632 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:27:37.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:37.775+0000 7ff166cac080 -1 Falling back to public interface 2021-01-15T16:27:38.034 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/55197,v1:127.0.0.1:6819/55197] [v2:127.0.0.1:6820/55197,v1:127.0.0.1:6821/55197] exists,up 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:27:38.034 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:27:38.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:27:38.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:27:38.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:27:38.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2021-01-15T16:27:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:27:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:27:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:27:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:27:38.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:27:38.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:27:38.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:27:38.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:27:38.435 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/55197,v1:127.0.0.1:6819/55197] [v2:127.0.0.1:6820/55197,v1:127.0.0.1:6821/55197] exists,up 1cc9002f-f77d-47e8-ab42-303305234f9f 2021-01-15T16:27:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:27:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:27:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:27:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2021-01-15T16:27:39.043 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:27:39.043+0000 7ff166cac080 -1 osd.1 44 log_to_monitors {default=true} 2021-01-15T16:27:58.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2021-01-15T16:27:58.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:27:58.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:27:58.438 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:27:58.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:27:58.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:27:58.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:27:58.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2021-01-15T16:27:58.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok dump_ops_in_flight 2021-01-15T16:27:58.548 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:27:58.548 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [], 2021-01-15T16:27:58.548 INFO:tasks.workunit.client.0.smithi145.stdout: "num_ops": 0 2021-01-15T16:27:58.548 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:27:58.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2021-01-15T16:27:58.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:58.558 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2021-01-15T16:27:58.927 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:58.927 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2021-01-15T16:27:40.792112+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:58.928 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:58.928 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:58.929 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:58.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-01-15T16:27:58.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T16:27:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:27:59.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-01-15T16:27:59.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T16:27:59.213 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:27:59.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2021-01-15T16:27:59.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2021-01-15T16:27:59.356 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:27:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2021-01-15T16:27:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2021-01-15T16:27:59.365 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2021-01-15T16:27:59.739 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:27:59.739 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2021-01-15T16:27:40.792112+0000 44'102 47:149 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:25:11.380825+0000 0'0 2021-01-15T16:25:11.380825+0000 0 2021-01-15T16:27:59.740 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:27:59.740 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:27:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:27:59.752 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2021-01-15T16:27:59.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2021-01-15T16:27:59.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:27:59.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2021-01-15T16:27:59.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:27:59.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2021-01-15T16:27:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:27:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2021-01-15T16:27:59.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:27:59.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2021-01-15T16:27:59.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2021-01-15T16:27:59.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:27:59.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2021-01-15T16:27:59.899 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_1: (2) No such file or directory 2021-01-15T16:27:59.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:27:59.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2021-01-15T16:27:59.929 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_2: (2) No such file or directory 2021-01-15T16:27:59.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2021-01-15T16:27:59.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-01-15T16:27:59.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:27:59.935 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2021-01-15T16:25:42.212244+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:27:59.935 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2021-01-15T16:25:42.212244+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:27:59.935 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:27:59.935 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2021-01-15T16:27:59.935 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2021-01-15T16:26:38.368313+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-01-15T16:27:59.936 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4536.0:1 2021-01-15T16:25:42.208075+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.936 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4536.0:1 2021-01-15T16:25:42.208075+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.936 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:27:59.936 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'104 last_divergent_update: 29'104 2021-01-15T16:27:59.937 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-01-15T16:27:59.937 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-01-15T16:27:59.937 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2021-01-15T16:25:42.211899+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:27:59.937 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2021-01-15T16:25:42.211899+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:27:59.937 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-01-15T16:27:59.938 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2021-01-15T16:27:59.938 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-01-15T16:27:59.938 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:27:59.939 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2021-01-15T16:27:59.939 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4540.0:1 2021-01-15T16:25:42.211899+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:27:59.939 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-01-15T16:27:59.939 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2021-01-15T16:27:59.939 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2021-01-15T16:25:42.197784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.940 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2021-01-15T16:25:42.197784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.940 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:27:59.940 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2021-01-15T16:27:59.940 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-01-15T16:27:59.940 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:27:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2021-01-15T16:27:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4529.0:1 2021-01-15T16:25:42.197784+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-01-15T16:27:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2021-01-15T16:27:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2021-01-15T16:25:42.193234+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.942 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2021-01-15T16:25:42.193234+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.942 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:27:59.943 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'101 last_divergent_update: 29'101 2021-01-15T16:27:59.943 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-01-15T16:27:59.943 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:27:59.943 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2021-01-15T16:27:59.944 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4527.0:1 2021-01-15T16:25:42.193234+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.944 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-01-15T16:27:59.944 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2021-01-15T16:27:59.944 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4534.0:1 2021-01-15T16:25:42.207309+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.944 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4534.0:1 2021-01-15T16:25:42.207309+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:27:59.945 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:27:59.945 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2021-01-15T16:27:59.945 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-01-15T16:27:59.945 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:27:22.058+0000 7f4d19bb2700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-01-15T16:27:59.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-01-15T16:27:59.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:27:59.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2021-01-15T16:27:59.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:27:59.947 INFO:tasks.workunit.client.0.smithi145.stdout:success 2021-01-15T16:27:59.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2021-01-15T16:27:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.33425.out 2021-01-15T16:27:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33425.out': No such file or directory 2021-01-15T16:27:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2021-01-15T16:27:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T16:27:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T16:28:00.436 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T16:28:00.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2021-01-15T16:28:00.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:28:00.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:28:00.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:28:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:28:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:28:00.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:28:00.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-01-15T16:28:00.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:28:00.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:28:00.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:28:00.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:28:00.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:28:00.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:28:00.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:28:00.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:28:00.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:28:00.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:28:00.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:28:00.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:28:00.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:28:00.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:28:00.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:28:00.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:28:00.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:28:00.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:28:00.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:28:00.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:28:00.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:28:00.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:28:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:28:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:00.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:28:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:28:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:28:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-01-15T16:28:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-01-15T16:28:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-01-15T16:28:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-01-15T16:28:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:28:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:28:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:28:00.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:28:00.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:28:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:28:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:28:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:28:00.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:28:00.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:28:00.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:28:00.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:28:00.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:28:00.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:28:00.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:28:00.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:28:00.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:28:00.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:28:00.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:28:00.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:28:00.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:28:00.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:28:00.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:28:00.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:00.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:00.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:28:00.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:28:00.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:28:00.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-01-15T16:28:00.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:28:00.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:00.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:00.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.33425 2021-01-15T16:28:00.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:28:00.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:28:00.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:28:00.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-01-15T16:28:00.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2021-01-15T16:28:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2021-01-15T16:28:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2021-01-15T16:28:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2021-01-15T16:28:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2021-01-15T16:28:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2021-01-15T16:28:00.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2021-01-15T16:28:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2021-01-15T16:28:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:28:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:28:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2021-01-15T16:28:00.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-01-15T16:28:00.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:28:00.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:28:00.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:28:00.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-01-15T16:28:00.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-01-15T16:28:00.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:28:00.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:28:00.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:28:00.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:28:00.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:00.824 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:00.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:00.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:28:00.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:28:00.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:28:00.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:28:00.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:28:00.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:28:00.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:28:00.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:28:00.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:28:00.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:28:00.989 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:28:00.989 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:00.989 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:28:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:28:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get fsid 2021-01-15T16:28:01.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:28:01.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:28:01.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:28:01.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:28:01.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:28:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:28:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:28:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:28:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:28:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:28:01.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:28:01.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get mon_host 2021-01-15T16:28:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2021-01-15T16:28:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2021-01-15T16:28:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:28:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:28:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:28:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2021-01-15T16:28:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:28:01.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:28:01.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:28:01.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:28:01.681 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:28:01.681 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:01.681 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:01.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:01.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:28:01.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:28:01.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-01-15T16:28:01.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2021-01-15T16:28:01.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:28:01.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:28:01.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:28:01.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:28:01.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:28:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:28:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:28:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:28:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:28:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:28:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:28:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:28:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:28:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:28:01.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:28:01.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:28:01.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:28:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:28:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:01.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:01.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:01.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:28:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:28:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:28:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:28:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:28:01.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:28:01.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:28:01.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:01.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:28:01.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:28:01.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:28:01.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:28:01.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:28:01.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 0b3ad672-7cb8-48b8-8f80-595c5ea20d86' 2021-01-15T16:28:01.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:28:01.719 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:28:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCRwgFgH758KxAApA+37F8ckclPmh1zF4dmMQ== 2021-01-15T16:28:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCRwgFgH758KxAApA+37F8ckclPmh1zF4dmMQ=="}' 2021-01-15T16:28:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 -i td/divergent-priors/0/new.json 2021-01-15T16:28:02.080 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:28:02.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2021-01-15T16:28:02.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCRwgFgH758KxAApA+37F8ckclPmh1zF4dmMQ== --osd-uuid 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:28:02.122 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.121+0000 7fc6eb79a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.123+0000 7fc6eb79a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:02.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.124+0000 7fc6eb79a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:02.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.125+0000 7fc6eb79a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:28:02.126 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.125+0000 7fc6eb79a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:28:02.126 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.125+0000 7fc6eb79a080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:28:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:02.178+0000 7fc6eb79a080 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-01-15T16:28:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-01-15T16:28:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:28:06.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:28:06.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:28:06.658 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:28:07.186 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:28:07.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:28:07.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:07.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:28:07.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:28:07.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:28:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:07.214+0000 7f66b3603080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:07.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:07.234+0000 7f66b3603080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:07.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:07.235+0000 7f66b3603080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:28:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:28:07.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:28:07.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:28:07.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:28:07.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:07.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:28:07.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:07.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:28:07.591 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:28:07.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:07.750+0000 7f66b3603080 -1 Falling back to public interface 2021-01-15T16:28:07.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:28:08.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:08.543+0000 7f66b3603080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:28:08.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:28:08.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:08.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:28:08.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:08.997 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:28:08.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:28:09.424 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/58452,v1:127.0.0.1:6803/58452] [v2:127.0.0.1:6804/58452,v1:127.0.0.1:6805/58452] exists,up 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:28:09.425 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:28:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:28:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:28:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:28:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-01-15T16:28:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2021-01-15T16:28:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:28:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:28:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:28:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:28:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:28:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:28:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:28:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:28:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:28:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:28:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:28:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:28:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:28:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:28:09.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:28:09.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:28:09.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:28:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:28:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:09.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:28:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:28:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:28:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:28:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:28:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:28:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:28:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:28:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:28:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:28:09.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:28:09.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:28:09.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 63348b66-8664-42fb-9057-333e23903fe5' 2021-01-15T16:28:09.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:28:09.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCZwgFgflNxGhAAhYPQzgxlAtYSwU6uMfWkUQ== 2021-01-15T16:28:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCZwgFgflNxGhAAhYPQzgxlAtYSwU6uMfWkUQ=="}' 2021-01-15T16:28:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 63348b66-8664-42fb-9057-333e23903fe5 -i td/divergent-priors/1/new.json 2021-01-15T16:28:09.935 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:28:09.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2021-01-15T16:28:09.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCZwgFgflNxGhAAhYPQzgxlAtYSwU6uMfWkUQ== --osd-uuid 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:28:09.977 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.976+0000 7ff6c7dcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.978+0000 7ff6c7dcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:09.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.979+0000 7ff6c7dcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:09.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.979+0000 7ff6c7dcf080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:28:09.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.980+0000 7ff6c7dcf080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:28:09.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:09.980+0000 7ff6c7dcf080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:28:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:10.040+0000 7ff6c7dcf080 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-01-15T16:28:14.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-01-15T16:28:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:28:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:28:14.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:28:14.609 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:28:15.143 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:28:15.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:28:15.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:15.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:28:15.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:28:15.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:28:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:15.171+0000 7fef0cffc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:15.189 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:15.188+0000 7fef0cffc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:15.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:15.191+0000 7fef0cffc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:15.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:28:15.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:28:15.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:28:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:28:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:28:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:28:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:15.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:28:15.570 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:28:15.706 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:15.706+0000 7fef0cffc080 -1 Falling back to public interface 2021-01-15T16:28:15.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:28:16.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:16.499+0000 7fef0cffc080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:28:16.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:28:16.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:16.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:28:16.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:16.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:28:16.995 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:28:17.394 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/59716,v1:127.0.0.1:6811/59716] [v2:127.0.0.1:6812/59716,v1:127.0.0.1:6813/59716] exists,up 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:28:17.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:28:17.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:28:17.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:28:17.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2021-01-15T16:28:17.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2021-01-15T16:28:17.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:28:17.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:28:17.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:28:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:28:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:28:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:28:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:28:17.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:28:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:28:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:28:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:28:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:28:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:28:17.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:28:17.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:28:17.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:28:17.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:28:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:28:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:17.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:17.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:28:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:28:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:28:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:28:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:28:17.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:28:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:28:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:28:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:28:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:28:17.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:28:17.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:28:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:28:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 3b18e4eb-df97-4d4b-a727-c5d699863dd4' 2021-01-15T16:28:17.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:28:17.404 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:28:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQChwgFgEjitGBAAMt9rKnPqgOL27HI4NpSVIg== 2021-01-15T16:28:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQChwgFgEjitGBAAMt9rKnPqgOL27HI4NpSVIg=="}' 2021-01-15T16:28:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3b18e4eb-df97-4d4b-a727-c5d699863dd4 -i td/divergent-priors/2/new.json 2021-01-15T16:28:17.959 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:28:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2021-01-15T16:28:17.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQChwgFgEjitGBAAMt9rKnPqgOL27HI4NpSVIg== --osd-uuid 3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:28:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.000+0000 7f61546fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:18.002 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.002+0000 7f61546fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.003+0000 7f61546fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.003+0000 7f61546fc080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:28:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.003+0000 7f61546fc080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:28:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.003+0000 7f61546fc080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:28:18.038 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:18.038+0000 7f61546fc080 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-01-15T16:28:22.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-01-15T16:28:22.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:28:22.606 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:28:22.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:28:22.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:28:23.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:28:23.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:28:23.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:28:23.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:28:23.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:28:23.136 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:28:23.163 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:23.163+0000 7f9d14bbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:23.179 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:23.178+0000 7f9d14bbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:23.180 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:23.180+0000 7f9d14bbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:28:23.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:28:23.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:28:23.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:28:23.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:28:23.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:28:23.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:23.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:28:23.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:23.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:28:23.552 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:28:23.717 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:23.714+0000 7f9d14bbb080 -1 Falling back to public interface 2021-01-15T16:28:23.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:28:24.516 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:28:24.513+0000 7f9d14bbb080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:28:24.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:28:24.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:28:24.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:28:24.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:28:24.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:28:24.986 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:28:25.404 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/60980,v1:127.0.0.1:6819/60980] [v2:127.0.0.1:6820/60980,v1:127.0.0.1:6821/60980] exists,up 3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:28:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:28:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:28:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:28:25.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2021-01-15T16:28:26.231 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:28:26.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2021-01-15T16:28:26.900 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2021-01-15T16:28:26.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2021-01-15T16:28:27.661 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2021-01-15T16:28:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2021-01-15T16:28:27.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:28:28.372 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:28:28.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:28:29.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2021-01-15T16:28:30.142 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T16:28:30.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2021-01-15T16:28:30.786 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T16:28:30.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2021-01-15T16:28:31.438 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 pg_autoscale_mode to on 2021-01-15T16:28:31.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2021-01-15T16:28:31.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:28:31.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:28:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:28:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:28:31.901 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:28:31.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:28:31.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:31.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:28:32.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:28:32.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:28:32.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:28:32.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:32.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:28:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T16:28:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T16:28:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T16:28:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:28:32.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:28:32.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:28:32.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2021-01-15T16:28:32.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:32.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:28:32.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:32.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:28:32.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:28:32.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:32.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:28:32.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:28:32.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:32.328 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:28:32.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2021-01-15T16:28:32.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:28:33.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:28:33.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:34.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T16:28:34.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:34.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T16:28:34.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:34.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:28:34.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T16:28:34.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:34.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T16:28:34.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T16:28:34.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:28:34.127 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T16:28:34.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2021-01-15T16:28:34.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:34.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:28:34.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:28:34.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:28:34.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:34.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:28:34.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:28:34.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:28:34.525 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:28:34.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T16:28:34.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2021-01-15T16:28:34.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:28:34.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:28:34.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:28:34.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:28:34.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:28:34.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:28:34.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:28:34.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:28:34.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:28:34.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:28:35.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:28:35.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:28:35.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:28:35.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:28:35.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:28:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:28:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:28:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:28:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:28:35.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:35.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:28:35.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T16:28:35.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T16:28:35.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T16:28:35.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:35.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:28:35.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T16:28:35.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T16:28:35.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-01-15T16:28:35.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:35.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:28:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509447 2021-01-15T16:28:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509447 2021-01-15T16:28:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2021-01-15T16:28:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:35.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T16:28:35.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T16:28:35.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:35.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T16:28:35.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T16:28:35.869 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T16:28:35.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:36.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2021-01-15T16:28:36.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:28:37.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:28:37.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:37.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2021-01-15T16:28:37.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:37.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T16:28:37.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:28:37.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T16:28:37.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:37.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T16:28:37.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T16:28:37.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:28:37.726 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T16:28:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T16:28:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:38.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509447 2021-01-15T16:28:38.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:38.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:28:38.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-01-15T16:28:38.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:38.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509447 2021-01-15T16:28:38.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-01-15T16:28:38.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:28:38.146 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509447 2021-01-15T16:28:38.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509447 2021-01-15T16:28:38.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:28:38.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:28:38.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:28:39.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:28:39.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:28:39.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:28:39.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:28:39.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:28:39.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:28:39.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:28:39.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:28:39.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:28:39.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:28:39.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:28:39.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:28:39.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:28:39.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:28:39.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:28:39.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2021-01-15T16:28:39.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2021-01-15T16:28:40.356 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:28:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2021-01-15T16:28:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2021-01-15T16:28:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:28:40.366 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2021-01-15T16:28:40.741 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:28:40.742 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:28:29.580215+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:28:40.742 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:28:40.742 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:28:40.743 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:28:40.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2021-01-15T16:28:40.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-01-15T16:28:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2021-01-15T16:28:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2021-01-15T16:28:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-01-15T16:28:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2021-01-15T16:28:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2021-01-15T16:28:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2021-01-15T16:28:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2021-01-15T16:28:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2021-01-15T16:28:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2021-01-15T16:28:40.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2021-01-15T16:28:40.756 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2021-01-15T16:28:40.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:40.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2021-01-15T16:28:40.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:40.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2021-01-15T16:28:41.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2021-01-15T16:28:41.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2021-01-15T16:28:41.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2021-01-15T16:28:41.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2021-01-15T16:28:41.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2021-01-15T16:28:41.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2021-01-15T16:28:41.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2021-01-15T16:28:41.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2021-01-15T16:28:41.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:41.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2021-01-15T16:28:42.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2021-01-15T16:28:42.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2021-01-15T16:28:42.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2021-01-15T16:28:42.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2021-01-15T16:28:42.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2021-01-15T16:28:42.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2021-01-15T16:28:42.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2021-01-15T16:28:42.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:42.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2021-01-15T16:28:43.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2021-01-15T16:28:43.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2021-01-15T16:28:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2021-01-15T16:28:43.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2021-01-15T16:28:43.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2021-01-15T16:28:43.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2021-01-15T16:28:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:43.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2021-01-15T16:28:44.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2021-01-15T16:28:44.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2021-01-15T16:28:44.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2021-01-15T16:28:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2021-01-15T16:28:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2021-01-15T16:28:44.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2021-01-15T16:28:44.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2021-01-15T16:28:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2021-01-15T16:28:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2021-01-15T16:28:45.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2021-01-15T16:28:45.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2021-01-15T16:28:45.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2021-01-15T16:28:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2021-01-15T16:28:45.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2021-01-15T16:28:45.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2021-01-15T16:28:45.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:45.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2021-01-15T16:28:46.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2021-01-15T16:28:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2021-01-15T16:28:46.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2021-01-15T16:28:46.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2021-01-15T16:28:46.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2021-01-15T16:28:46.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2021-01-15T16:28:46.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2021-01-15T16:28:46.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:46.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2021-01-15T16:28:47.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2021-01-15T16:28:47.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2021-01-15T16:28:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2021-01-15T16:28:47.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2021-01-15T16:28:47.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2021-01-15T16:28:47.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2021-01-15T16:28:47.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:47.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2021-01-15T16:28:48.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2021-01-15T16:28:48.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2021-01-15T16:28:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2021-01-15T16:28:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2021-01-15T16:28:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2021-01-15T16:28:48.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2021-01-15T16:28:48.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2021-01-15T16:28:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2021-01-15T16:28:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2021-01-15T16:28:49.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2021-01-15T16:28:49.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2021-01-15T16:28:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2021-01-15T16:28:49.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2021-01-15T16:28:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2021-01-15T16:28:49.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2021-01-15T16:28:49.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:49.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2021-01-15T16:28:50.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2021-01-15T16:28:50.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2021-01-15T16:28:50.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2021-01-15T16:28:50.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2021-01-15T16:28:50.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2021-01-15T16:28:50.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2021-01-15T16:28:50.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2021-01-15T16:28:50.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:50.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2021-01-15T16:28:51.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2021-01-15T16:28:51.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2021-01-15T16:28:51.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2021-01-15T16:28:51.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2021-01-15T16:28:51.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2021-01-15T16:28:51.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2021-01-15T16:28:51.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2021-01-15T16:28:51.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:51.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2021-01-15T16:28:52.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2021-01-15T16:28:52.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2021-01-15T16:28:52.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2021-01-15T16:28:52.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2021-01-15T16:28:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2021-01-15T16:28:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2021-01-15T16:28:52.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2021-01-15T16:28:52.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:52.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2021-01-15T16:28:53.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:53.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2021-01-15T16:28:53.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:53.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2021-01-15T16:28:53.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2021-01-15T16:28:53.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2021-01-15T16:28:53.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:28:53.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2021-01-15T16:28:53.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:28:53.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:28:53.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:28:53.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:28:53.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:28:53.401 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:28:53.401 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:28:53.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:28:53.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:28:53.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:28:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:28:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:28:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:28:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:28:53.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:28:53.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:28:53.851 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:28:53.851 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:28:53.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:28:53.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:53.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:28:53.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T16:28:53.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T16:28:53.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T16:28:53.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:53.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:28:54.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:28:54.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:28:54.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T16:28:54.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:28:54.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:28:54.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T16:28:54.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T16:28:54.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T16:28:54.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:54.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T16:28:54.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:54.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:28:54.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T16:28:54.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:54.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T16:28:54.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T16:28:54.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:54.280 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T16:28:54.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2021-01-15T16:28:54.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:28:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:28:55.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:28:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836495 2021-01-15T16:28:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:56.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:28:56.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:28:56.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:28:56.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:56.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:28:56.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:28:56.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:28:56.094 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:28:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T16:28:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:28:56.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T16:28:56.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:28:56.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:28:56.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T16:28:56.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:28:56.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T16:28:56.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T16:28:56.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:28:56.511 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T16:28:56.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T16:28:56.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:28:56.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:28:56.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:28:57.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:28:57.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:28:57.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:28:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:28:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:28:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:28:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:28:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:28:57.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:28:57.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:28:57.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:28:57.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:28:58.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:28:58.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:28:58.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:28:58.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2021-01-15T16:28:58.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T16:28:58.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2021-01-15T16:28:58.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2021-01-15T16:28:58.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T16:28:58.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2021-01-15T16:28:58.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2021-01-15T16:28:58.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:28:58.801 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2021-01-15T16:28:59.180 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:28:59.180 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-01-15T16:28:29.580215+0000 30'100 30:112 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:28:59.181 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:28:59.181 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:28:59.182 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:28:59.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:28:59.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2021-01-15T16:28:59.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:28:59.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:28:59.194 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:28:59.195 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:59.195 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:59.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:28:59.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:28:59.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok config set objectstore_blackhole 1 2021-01-15T16:28:59.290 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:28:59.291 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:28:59.291 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:28:59.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:28:59.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2021-01-15T16:28:59.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:28:59.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:28:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:28:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:28:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:28:59.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:28:59.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:28:59.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok config set objectstore_blackhole 1 2021-01-15T16:28:59.398 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:28:59.398 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:28:59.398 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:28:59.406 INFO:tasks.workunit.client.0.smithi145.stdout:create new divergent objects 2021-01-15T16:28:59.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2021-01-15T16:28:59.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2021-01-15T16:28:59.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:28:59.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:28:59.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2021-01-15T16:28:59.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2021-01-15T16:28:59.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2021-01-15T16:28:59.408 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent objects 2021-01-15T16:28:59.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2021-01-15T16:28:59.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-01-15T16:28:59.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2021-01-15T16:28:59.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2021-01-15T16:28:59.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2021-01-15T16:28:59.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2021-01-15T16:28:59.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2021-01-15T16:28:59.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-01-15T16:28:59.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2021-01-15T16:28:59.412 INFO:tasks.workunit.client.0.smithi145.stdout:remove divergent objects 2021-01-15T16:28:59.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2021-01-15T16:28:59.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2021-01-15T16:28:59.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2021-01-15T16:28:59.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2021-01-15T16:28:59.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2021-01-15T16:28:59.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2021-01-15T16:28:59.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2021-01-15T16:29:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2021-01-15T16:29:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2021-01-15T16:29:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2021-01-15T16:29:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:29:09.418 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2021-01-15T16:29:09.816 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:29:09.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.821 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.821 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 peering 2021-01-15T16:29:07.717935+0000 30'106 32:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:09.821 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.821 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.821 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.822 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.822 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.822 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.822 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.823 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.823 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.823 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.823 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:09.823 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:29:09.824 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:29:09.824 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:29:09.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2021-01-15T16:29:09.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:29:09.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:29:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:29:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:29:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:29:09.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:29:09.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-01-15T16:29:09.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2021-01-15T16:29:10.451 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T16:29:10.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-01-15T16:29:10.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2021-01-15T16:29:10.991 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T16:29:11.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2021-01-15T16:29:11.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2021-01-15T16:29:11.518 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T16:29:11.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:11.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2021-01-15T16:29:12.088 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T16:29:12.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:12.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2021-01-15T16:29:12.639 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T16:29:12.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2021-01-15T16:29:12.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:29:12.648 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2021-01-15T16:29:13.019 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:29:13.020 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.020 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.020 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.020 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.021 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.021 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.021 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.021 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.021 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.022 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.022 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.022 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.022 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.022 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1457 0 0 16 16 stale+peering 2021-01-15T16:29:07.717935+0000 30'106 32:122 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:13.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.024 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.025 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.026 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T16:29:07.546683+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T16:29:07.546683+0000 0'0 2021-01-15T16:29:07.546683+0000 0 2021-01-15T16:29:13.026 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:29:13.027 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:29:13.027 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:29:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2021-01-15T16:29:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:29:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:29:13.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:29:13.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:29:13.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:29:13.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:29:13.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:29:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:29:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:29:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:29:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:29:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:29:13.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:29:13.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:29:13.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:29:13.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:29:13.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:29:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:29:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:29:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:29:13.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:29:13.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:29:13.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:29:13.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:29:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:29:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:29:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:29:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:29:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:29:13.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:29:13.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:29:13.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:29:13.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:29:13.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2021-01-15T16:29:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:29:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:29:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:29:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:29:13.042 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:29:13.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:13.061+0000 7f80253e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:13.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:13.063+0000 7f80253e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:13.065 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:13.065+0000 7f80253e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:13.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:29:13.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:29:13.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:29:13.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:29:13.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:29:13.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:29:13.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:29:13.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:29:13.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:29:13.433 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:29:13.580 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:13.580+0000 7f80253e6080 -1 Falling back to public interface 2021-01-15T16:29:13.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:29:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:14.750+0000 7f80253e6080 -1 osd.0 30 log_to_monitors {default=true} 2021-01-15T16:29:14.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:29:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:29:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:29:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:29:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:29:14.830 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:29:15.245 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 39 up_thru 0 down_at 34 last_clean_interval [5,30) [v2:127.0.0.1:6802/65028,v1:127.0.0.1:6803/65028] [v2:127.0.0.1:6804/65028,v1:127.0.0.1:6805/65028] exists,up 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:29:15.246 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:29:15.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:29:15.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:29:15.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:29:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2021-01-15T16:29:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:29:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:29:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:29:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:29:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:29:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:29:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:29:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:29:15.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:29:15.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:29:15.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:29:15.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:29:15.250 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:29:15.250 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:29:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:29:15.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:29:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:29:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:29:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:29:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:29:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:29:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:29:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:29:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:29:15.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:29:15.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:29:15.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:29:15.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:29:15.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2021-01-15T16:29:15.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:29:15.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:29:15.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:29:15.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:29:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:15.274+0000 7fa460ffb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:15.276 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:15.275+0000 7fa460ffb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:15.276 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:15.276+0000 7fa460ffb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:29:15.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:29:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:29:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:29:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:29:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:29:15.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:29:15.651 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:29:15.794 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:15.794+0000 7fa460ffb080 -1 Falling back to public interface 2021-01-15T16:29:16.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:29:17.015 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:29:17.014+0000 7fa460ffb080 -1 osd.2 30 log_to_monitors {default=true} 2021-01-15T16:29:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:29:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:29:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:29:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:29:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:29:17.051 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:29:17.476 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 40 up_thru 0 down_at 36 last_clean_interval [15,30) [v2:127.0.0.1:6810/65517,v1:127.0.0.1:6811/65517] [v2:127.0.0.1:6812/65517,v1:127.0.0.1:6813/65517] exists,up 3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:29:17.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:29:17.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:29:17.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:29:17.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:17.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2021-01-15T16:29:18.022 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T16:29:18.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:29:18.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2021-01-15T16:29:18.978 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2021-01-15T16:29:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:29:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2021-01-15T16:29:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:29:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:29:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:29:18.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:29:18.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:29:18.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:29:18.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:29:18.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:29:18.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:29:18.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:29:19.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:29:19.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:29:19.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:29:19.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:29:19.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:29:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:29:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:29:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:29:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:29:19.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:19.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:29:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724547 2021-01-15T16:29:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724547 2021-01-15T16:29:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547' 2021-01-15T16:29:19.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:19.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:29:19.751 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:29:19.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:29:19.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:29:19.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:29:19.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:19.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:29:19.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691843 2021-01-15T16:29:19.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691843 2021-01-15T16:29:19.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724547 2-171798691843' 2021-01-15T16:29:19.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:29:19.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-167503724547 2021-01-15T16:29:19.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:29:19.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:29:19.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724547 2021-01-15T16:29:19.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:29:19.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724547 2021-01-15T16:29:19.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2021-01-15T16:29:19.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:29:19.899 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 167503724547 2021-01-15T16:29:20.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 167503724547 2021-01-15T16:29:20.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:29:21.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:29:21.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:29:21.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724548 -lt 167503724547 2021-01-15T16:29:21.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:29:21.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-171798691843 2021-01-15T16:29:21.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:29:21.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:29:21.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-171798691843 2021-01-15T16:29:21.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:29:21.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691843 2021-01-15T16:29:21.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 171798691843' 2021-01-15T16:29:21.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:29:21.707 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 171798691843 2021-01-15T16:29:22.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691843 -lt 171798691843 2021-01-15T16:29:22.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:29:22.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:22.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:22.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 32 == 0 2021-01-15T16:29:22.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:22.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:22.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:22.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:22.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:22.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:22.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:23.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:23.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:23.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:23.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:29:23.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:29:23.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:29:23.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:23.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:29:23.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:23.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:23.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:23.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:23.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:23.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:23.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:23.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:24.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:24.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:24.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:24.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:24.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:24.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:24.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:24.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:24.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:25.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:25.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:25.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:29:25.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:25.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:29:25.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:25.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:25.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:25.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:25.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:25.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:25.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:25.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:25.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:25.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:25.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:25.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:26.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:26.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:26.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:26.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:26.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:29:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:29:27.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:27.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:27.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:27.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:27.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:27.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:27.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:27.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:27.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:27.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:27.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:27.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:28.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:28.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:29:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:28.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:29:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:29.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:29.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:29.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:29.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:29.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:29.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:29.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:29.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:29.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:29.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:30.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:30.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:30.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:30.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:29:30.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:30.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:29:32.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:32.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:32.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:32.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:32.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:32.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:32.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:32.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:32.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:32.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:32.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:32.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:33.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:33.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:33.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:33.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:33.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:33.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:33.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:33.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:33.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:33.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:33.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:33.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:29:33.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:33.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:29:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:37.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:37.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:37.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:37.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:37.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:37.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:37.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:37.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:37.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:37.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:37.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:29:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:29:44.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:44.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:44.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:44.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:44.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:44.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:44.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:44.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:45.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:45.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:45.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:45.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:45.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:45.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:46.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:46.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:46.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:29:46.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:29:46.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:29:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:29:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:53.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:29:54.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:29:54.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:29:54.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:54.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:54.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:29:54.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:29:54.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:29:54.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:29:54.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:29:54.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:29:54.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:29:54.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:29:54.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:29:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:29:55.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:29:55.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:29:55.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:29:55.546 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:29:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "21126fff-feba-4bfc-abd7-72ef9c949fda", 2021-01-15T16:29:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:29:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:29:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:29:55.545889+0000", 2021-01-15T16:29:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:29:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:29:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:29:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:29:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:29:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:29:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:29:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:29:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:29:55.550 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.550 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:29:55.550 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:29:55.550 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.551 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:29:55.551 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:29:55.551 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:29:55.551 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:29:55.551 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:29:55.552 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.552 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:29:55.552 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.552 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:29:55.553 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.553 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.553 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:29:55.553 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.553 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:29:55.554 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:29:55.554 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:29:55.554 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2021-01-15T16:29:55.554 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 32 2021-01-15T16:29:55.555 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.555 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:29:55.555 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.555 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.555 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.556 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.556 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.557 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.557 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.557 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.557 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.557 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.558 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.558 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.558 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.558 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.559 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.559 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.559 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.560 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.560 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.560 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.561 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.561 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.561 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.561 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.562 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.562 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.563 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.563 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.563 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.563 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.564 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.564 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.564 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.565 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.565 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.566 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.566 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.567 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.567 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.567 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.567 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.567 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.568 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.568 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.568 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.568 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.568 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.569 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.569 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.569 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.569 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.569 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.570 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.570 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.570 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.571 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.571 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.571 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.571 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.572 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.572 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.572 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.572 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.573 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.573 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.573 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.574 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.574 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.574 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.574 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.575 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.575 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.576 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.576 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2021-01-15T16:29:55.576 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.576 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.576 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:29:55.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.577 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:29:55.577 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:29:55.577 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:29:55.577 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:29:55.578 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:29:55.578 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.578 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:29:55.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.579 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:29:55.579 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.579 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.579 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:29:55.579 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.580 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.580 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:29:55.580 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.581 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:29:55.581 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.581 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:29:55.581 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:29:55.581 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:29:55.582 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:29:55.582 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:29:55.582 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:28:00.614842Z", 2021-01-15T16:29:55.582 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:28:00.614842Z", 2021-01-15T16:29:55.582 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:29:55.583 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:29:55.583 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:29:55.583 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:29:55.583 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:29:55.584 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:29:55.584 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:29:55.584 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:29:55.584 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:29:55.584 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:29:55.585 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:29:55.585 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:29:55.585 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:29:55.585 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:29:55.585 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:29:55.586 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.586 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:29:55.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.586 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:29:55.586 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.587 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:29:55.587 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:29:55.587 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:29:55.587 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.588 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.588 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:29:55.588 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:29:55.588 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.589 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.589 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.589 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:29:55.589 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:29:55.589 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:29:55.590 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:29:55.590 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:29:55.590 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.590 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.591 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:29:55.591 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:29:55.591 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.591 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:29:55.592 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 46, 2021-01-15T16:29:55.592 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:29:55.592 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:28:01.192050+0000", 2021-01-15T16:29:55.592 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:29:19.021391+0000", 2021-01-15T16:29:55.592 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:29:17.089890+0000", 2021-01-15T16:29:55.593 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:29:18.485078+0000", 2021-01-15T16:29:55.593 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:29:55.593 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:29:55.593 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:29:55.594 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:29:55.594 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:29:55.594 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:29:55.594 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:29:55.594 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:29:55.595 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:29:55.595 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:29:55.595 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.595 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2021-01-15T16:29:55.595 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:29:55.596 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:29:55.596 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:29:55.596 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:29:55.596 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:29:55.597 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:29:55.597 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:29:55.597 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:29:55.597 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:29:55.597 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:29:55.598 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.598 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:29:55.598 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:29:55.598 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:28:27.965281+0000", 2021-01-15T16:29:55.598 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T16:29:55.599 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T16:29:55.599 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:29:55.599 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:29:55.599 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:29:55.600 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T16:29:55.600 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:29:55.600 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:29:55.600 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:29:55.600 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:29:55.601 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:29:55.601 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "on", 2021-01-15T16:29:55.601 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 32, 2021-01-15T16:29:55.601 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:29:55.602 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 32, 2021-01-15T16:29:55.602 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 32, 2021-01-15T16:29:55.602 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 32, 2021-01-15T16:29:55.602 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:29:55.602 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:29:55.603 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:29:55.603 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:29:55.603 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:29:55.603 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:29:55.603 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:29:55.604 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.604 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "32", 2021-01-15T16:29:55.604 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:29:55.604 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:29:55.605 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "32", 2021-01-15T16:29:55.605 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:29:55.605 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T16:29:55.605 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T16:29:55.605 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T16:29:55.606 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T16:29:55.606 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:29:55.606 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:29:55.606 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:29:55.606 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:29:55.607 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:29:55.607 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:29:55.607 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:29:55.607 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:29:55.608 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:29:55.608 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:29:55.608 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:29:55.608 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:29:55.608 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:29:55.609 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:29:55.609 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:29:55.609 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T16:29:55.609 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:29:55.609 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:29:55.610 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.610 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:29:55.610 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:29:55.610 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:29:55.611 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:29:55.611 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:29:55.611 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:29:55.611 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:29:55.611 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:29:55.612 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T16:29:55.612 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:29:55.612 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:29:55.612 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:29:55.612 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:29:55.613 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.613 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.613 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:29:55.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.613 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:29:55.614 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "0b3ad672-7cb8-48b8-8f80-595c5ea20d86", 2021-01-15T16:29:55.614 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:29:55.614 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:29:55.614 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:29:55.615 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:29:55.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:29:55.615 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2021-01-15T16:29:55.615 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 39, 2021-01-15T16:29:55.615 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 44, 2021-01-15T16:29:55.616 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 34, 2021-01-15T16:29:55.616 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:29:55.616 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:29:55.616 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.617 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.617 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:29:55.617 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.617 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.618 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.618 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.618 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:29:55.618 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.618 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.619 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.619 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.619 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:29:55.619 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.620 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.620 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:29:55.620 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.621 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.621 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.621 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:29:55.621 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.621 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.622 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.622 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.622 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:29:55.622 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.623 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.623 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:29:55.623 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.623 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.624 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.624 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.624 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:29:55.624 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.624 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.625 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.625 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.625 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:29:55.625 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.625 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.626 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.626 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:29:55.626 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.626 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.627 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.627 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:29:55.627 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:29:55.627 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.628 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.628 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/65028", 2021-01-15T16:29:55.628 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/65028", 2021-01-15T16:29:55.628 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/65028", 2021-01-15T16:29:55.629 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/65028", 2021-01-15T16:29:55.629 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:29:55.629 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:29:55.629 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:29:55.629 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.630 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:29:55.630 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "63348b66-8664-42fb-9057-333e23903fe5", 2021-01-15T16:29:55.631 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:29:55.631 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:29:55.631 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:29:55.631 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:29:55.631 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:29:55.632 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:29:55.632 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:29:55.632 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 32, 2021-01-15T16:29:55.632 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 35, 2021-01-15T16:29:55.633 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:29:55.633 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:29:55.633 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.633 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.634 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:29:55.634 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.634 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.634 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.635 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:29:55.635 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.635 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.635 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.636 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:29:55.636 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.636 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.637 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:29:55.637 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.637 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.638 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:29:55.638 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.638 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.638 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.639 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:29:55.639 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.639 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.639 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.640 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:29:55.640 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.640 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.640 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.641 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:29:55.641 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.641 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.641 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.642 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:29:55.642 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.642 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.643 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:29:55.643 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.643 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.643 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.644 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:29:55.644 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:29:55.644 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.644 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.645 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/59716", 2021-01-15T16:29:55.645 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/59716", 2021-01-15T16:29:55.645 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/59716", 2021-01-15T16:29:55.645 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/59716", 2021-01-15T16:29:55.646 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:29:55.646 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:29:55.646 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.646 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.647 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:29:55.647 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "3b18e4eb-df97-4d4b-a727-c5d699863dd4", 2021-01-15T16:29:55.647 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:29:55.647 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:29:55.648 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:29:55.648 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:29:55.648 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:29:55.648 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2021-01-15T16:29:55.648 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 40, 2021-01-15T16:29:55.649 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:29:55.649 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 36, 2021-01-15T16:29:55.649 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:29:55.649 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:29:55.649 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.650 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.650 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:29:55.650 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.651 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.651 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:29:55.651 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.652 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.652 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.652 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:29:55.652 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.653 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.653 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:29:55.653 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.654 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.654 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:29:55.654 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.654 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.655 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.655 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:29:55.655 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.656 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.656 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:29:55.656 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.657 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.657 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:29:55.657 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.658 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.658 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.658 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:29:55.658 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:29:55.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.659 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:29:55.659 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:29:55.659 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.660 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:29:55.660 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:29:55.660 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:29:55.660 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.661 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.661 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/65517", 2021-01-15T16:29:55.661 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/65517", 2021-01-15T16:29:55.661 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/65517", 2021-01-15T16:29:55.662 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/65517", 2021-01-15T16:29:55.662 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:29:55.662 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:29:55.662 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:29:55.663 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.663 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.663 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.663 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:29:55.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.664 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:29:55.664 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:10.269445+0000", 2021-01-15T16:29:55.664 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:29:55.664 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:29:55.665 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:29:55.665 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:29:55.665 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:09.527491+0000", 2021-01-15T16:29:55.665 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:29:55.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.666 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:29:55.666 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:10.795129+0000", 2021-01-15T16:29:55.666 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:29:55.666 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:29:55.667 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:29:55.667 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:29:55.667 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:17.519770+0000", 2021-01-15T16:29:55.667 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:29:55.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.668 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:29:55.668 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:11.337040+0000", 2021-01-15T16:29:55.668 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:29:55.669 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:29:55.669 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:29:55.669 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:29:55.669 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:25.473895+0000", 2021-01-15T16:29:55.669 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:29:55.670 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.670 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.670 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:29:55.670 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:29:55.670 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:29:55.671 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:29:55.671 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:29:55.671 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:29:55.671 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:29:55.672 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:29:55.672 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:29:55.672 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:29:55.672 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:29:55.672 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.673 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.673 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:29:55.673 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:29:55.673 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:29:55.673 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:29:55.674 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:29:55.674 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:29:55.674 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:29:55.674 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:29:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:29:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:29:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:29:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.675 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.676 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:29:55.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.676 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:29:55.676 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:29:55.676 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/65028,v1:127.0.0.1:6805/65028]", 2021-01-15T16:29:55.677 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:29:55.677 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:29:55.677 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:29:55.677 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:29:55.677 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:29:55.678 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:29:55.678 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:29:55.678 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:29:55.678 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:29:55.679 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:29:55.679 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:29:55.679 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:29:55.679 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:29:55.679 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:29:55.680 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:29:55.680 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:29:55.680 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:29:55.680 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:29:55.680 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:29:55.681 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:29:55.681 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:29:55.681 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:29:55.681 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:29:55.681 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:29:55.682 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:29:55.682 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/65028,v1:127.0.0.1:6803/65028]", 2021-01-15T16:29:55.682 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:29:55.682 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/65028,v1:127.0.0.1:6809/65028]", 2021-01-15T16:29:55.683 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/65028,v1:127.0.0.1:6807/65028]", 2021-01-15T16:29:55.683 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:29:55.683 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:29:55.683 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:29:55.683 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:29:55.684 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:29:55.684 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:29:55.684 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:29:55.684 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:29:55.684 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:29:55.685 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:29:55.685 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:29:55.685 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:29:55.685 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:29:55.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.686 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:29:55.686 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:29:55.686 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/59716,v1:127.0.0.1:6813/59716]", 2021-01-15T16:29:55.687 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:29:55.687 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:29:55.687 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:29:55.687 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:29:55.687 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:29:55.688 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:29:55.688 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:29:55.688 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:29:55.688 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:29:55.688 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:29:55.689 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:29:55.689 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:29:55.689 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:29:55.689 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:29:55.690 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:29:55.690 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:29:55.690 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:29:55.690 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:29:55.690 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:29:55.691 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:29:55.691 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:29:55.691 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:29:55.691 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:29:55.691 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:29:55.692 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:29:55.692 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/59716,v1:127.0.0.1:6811/59716]", 2021-01-15T16:29:55.692 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:29:55.692 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/59716,v1:127.0.0.1:6817/59716]", 2021-01-15T16:29:55.693 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/59716,v1:127.0.0.1:6815/59716]", 2021-01-15T16:29:55.693 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:29:55.693 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:29:55.693 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:29:55.693 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:29:55.694 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:29:55.694 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:29:55.694 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:29:55.694 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:29:55.695 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:29:55.695 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:29:55.695 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:29:55.695 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:29:55.695 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:29:55.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.696 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.696 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:29:55.696 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:29:55.696 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/65517,v1:127.0.0.1:6813/65517]", 2021-01-15T16:29:55.697 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:29:55.697 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:29:55.697 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:29:55.697 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:29:55.697 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:29:55.698 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:29:55.698 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:29:55.698 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:29:55.698 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:29:55.699 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:29:55.699 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:29:55.699 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:29:55.699 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:29:55.699 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:29:55.700 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:29:55.700 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:29:55.700 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:29:55.700 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:29:55.700 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:29:55.701 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:29:55.701 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:29:55.701 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:29:55.701 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:29:55.702 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:29:55.702 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:29:55.702 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/65517,v1:127.0.0.1:6811/65517]", 2021-01-15T16:29:55.702 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:29:55.702 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/65517,v1:127.0.0.1:6817/65517]", 2021-01-15T16:29:55.703 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/65517,v1:127.0.0.1:6815/65517]", 2021-01-15T16:29:55.703 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:29:55.703 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:29:55.703 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:29:55.703 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:29:55.704 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:29:55.704 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:29:55.704 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:29:55.704 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:29:55.704 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:29:55.705 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:29:55.705 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:29:55.705 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:29:55.705 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:29:55.706 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.706 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.706 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:29:55.706 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:29:55.706 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:29:55.707 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:29:55.707 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.707 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:29:55.707 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.707 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:29:55.708 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 39 2021-01-15T16:29:55.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.708 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.708 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:29:55.709 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:29:55.709 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.709 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:29:55.709 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40 2021-01-15T16:29:55.710 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.710 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.710 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:29:55.710 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 46, 2021-01-15T16:29:55.711 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:29:55.711 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:29:55.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.711 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:29:55.711 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:29:55.712 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:29:55.712 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.712 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.712 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:29:55.713 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:29:55.713 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:29:55.713 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.713 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.713 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:29:55.714 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:29:55.714 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:29:55.714 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.714 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.714 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:29:55.715 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.715 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:29:55.715 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:29:55.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.715 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.716 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:29:55.716 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:29:55.716 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.716 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.717 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:29:55.717 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:29:55.717 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.717 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.717 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:29:55.718 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:29:55.718 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.718 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.718 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:29:55.718 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:29:55.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.719 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.719 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:29:55.719 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:29:55.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.720 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.720 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:29:55.720 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:29:55.720 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.720 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.721 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:29:55.721 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:29:55.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.721 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.722 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:29:55.722 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:29:55.722 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.722 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.722 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:29:55.723 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:29:55.723 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.723 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.723 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:29:55.723 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:29:55.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.724 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.724 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:29:55.724 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:29:55.724 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.725 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.725 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:29:55.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.725 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:29:55.726 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:29:55.726 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:29:55.726 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:29:55.726 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.726 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:29:55.727 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:29:55.727 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:29:55.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.727 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:29:55.727 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.728 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:29:55.728 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.728 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.728 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.729 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:29:55.729 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:29:55.729 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:29:55.729 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:29:55.729 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.730 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:29:55.730 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:29:55.730 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:29:55.730 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.731 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:29:55.731 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.731 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:29:55.731 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.731 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.732 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:29:55.732 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:29:55.732 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:29:55.733 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:29:55.733 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.733 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:29:55.733 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:29:55.733 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:29:55.734 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.734 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:29:55.734 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.734 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:29:55.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.735 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:29:55.735 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.735 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:29:55.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.736 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:29:55.736 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.736 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:29:55.737 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.737 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.738 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:29:55.738 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:29:55.738 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:29:55.738 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:29:55.738 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:29:55.739 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:29:55.739 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:29:55.739 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:29:55.739 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.740 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:29:55.740 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.740 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:29:55.740 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.741 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:29:55.741 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.741 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:29:55.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.742 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:29:55.742 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:29:55.742 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:29:55.743 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.743 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.743 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.743 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.744 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:29:55.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.744 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:29:55.744 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:29:55.744 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:29:55.745 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:29:55.745 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:29:55.745 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:29:55.745 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:29:55.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:29:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:29:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:29:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:29:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:29:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:29:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:29:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:29:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:29:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:29:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:29:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:29:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:29:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:29:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:29:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:29:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:29:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:29:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:29:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:29:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:29:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:29:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:29:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T16:29:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:29:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:29:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:29:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:29:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:29:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:29:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:29:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:29:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:29:55.755 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:29:55.755 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:29:55.755 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:29:55.755 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:29:55.756 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:29:55.756 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:29:55.756 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:29:55.756 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:29:55.756 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:29:55.757 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:29:55.757 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:29:55.757 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:29:55.757 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:29:55.757 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.758 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.758 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:29:55.758 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:29:55.758 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:29:55.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.759 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:29:55.759 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:29:55.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.759 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:29:55.760 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:29:55.760 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:29:55.760 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:29:55.760 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:29:55.760 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:29:55.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.761 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2021-01-15T16:29:55.761 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 32, 2021-01-15T16:29:55.761 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:29:55.761 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:29:55.762 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:29:55.762 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:29:55.762 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:29:55.762 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:29:55.763 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:29:55.763 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:29:55.763 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:29:55.763 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:29:55.763 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:29:55.764 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:29:55.764 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:29:55.764 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:29:55.764 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:29:55.764 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:29:55.765 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:29:55.765 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:29:55.765 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:29:55.765 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:29:55.766 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:29:55.766 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:29:55.766 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:29:55.766 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:29:55.766 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:29:55.767 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:29:55.767 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:29:55.767 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:29:55.767 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:29:55.767 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:29:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:29:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:29:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:29:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:29:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:29:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:29:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:29:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:29:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:29:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:29:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:29:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:29:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:29:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:29:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:29:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:29:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:29:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:29:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:29:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:29:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:29:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:29:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:29:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:29:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:29:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 512, 2021-01-15T16:29:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 512, 2021-01-15T16:29:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2021-01-15T16:29:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2021-01-15T16:29:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:29:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.775 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:29:55.775 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:29:55.775 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:29:55.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2021-01-15T16:29:55.775 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:29:55.776 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:29:55.776 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:29:55.776 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:29:55.776 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19572, 2021-01-15T16:29:55.776 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2196, 2021-01-15T16:29:55.777 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:29:55.777 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 17342, 2021-01-15T16:29:55.777 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553228, 2021-01-15T16:29:55.777 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:29:55.778 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:29:55.778 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322102505472, 2021-01-15T16:29:55.778 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:29:55.778 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2248704, 2021-01-15T16:29:55.778 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 456490, 2021-01-15T16:29:55.779 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:29:55.779 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:29:55.779 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:29:55.780 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:29:55.780 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17758312 2021-01-15T16:29:55.780 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.780 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:29:55.781 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:29:55.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:29:55.781 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:29:55.781 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:29:55.781 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:29:55.782 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:29:55.782 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.782 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:29:55.782 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 54, 2021-01-15T16:29:55.782 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 54, 2021-01-15T16:29:55.783 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 54000000, 2021-01-15T16:29:55.783 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 54000000 2021-01-15T16:29:55.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.783 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:29:55.784 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:29:55.784 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.784 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:29:55.784 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:29:55.784 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:29:55.785 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:29:55.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2021-01-15T16:29:55.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:29:55.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:29:55.785 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:29:55.786 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:29:55.786 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19572, 2021-01-15T16:29:55.786 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2196, 2021-01-15T16:29:55.786 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:29:55.787 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 17342, 2021-01-15T16:29:55.787 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553228, 2021-01-15T16:29:55.787 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:29:55.787 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:29:55.787 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322102505472, 2021-01-15T16:29:55.788 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:29:55.788 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2248704, 2021-01-15T16:29:55.788 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 456490, 2021-01-15T16:29:55.788 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:29:55.788 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:29:55.789 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:29:55.789 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:29:55.789 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17758312 2021-01-15T16:29:55.789 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.789 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:29:55.790 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:29:55.790 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:29:55.790 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:29:55.790 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:29:55.791 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:29:55.791 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:29:55.791 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.791 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:29:55.791 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 54, 2021-01-15T16:29:55.792 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 54, 2021-01-15T16:29:55.792 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 54000000, 2021-01-15T16:29:55.792 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 54000000 2021-01-15T16:29:55.792 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.792 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:29:55.793 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:29:55.793 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.793 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.793 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:29:55.793 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.794 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:29:55.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2021-01-15T16:29:55.794 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:29:55.794 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48300, 2021-01-15T16:29:55.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:29:55.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:29:55.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:29:55.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:29:55.795 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:29:55.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:29:55.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:29:55.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:29:55.796 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:29:55.797 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:29:55.797 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:29:55.797 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:29:55.797 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 100, 2021-01-15T16:29:55.798 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 100, 2021-01-15T16:29:55.798 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:29:55.798 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:29:55.798 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:29:55.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:29:55.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:29:55.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:29:55.799 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:29:55.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:29:55.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:29:55.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:29:55.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:29:55.800 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:29:55.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:29:55.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:29:55.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:29:55.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:29:55.801 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:29:55.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:29:55.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:29:55.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:29:55.802 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:29:55.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:29:55.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:29:55.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:29:55.803 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:29:55.803 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.804 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:29:55.804 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:29:55.804 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:29:55.804 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:29:55.804 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1220608, 2021-01-15T16:29:55.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 143228, 2021-01-15T16:29:55.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:29:55.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:29:55.805 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:29:55.806 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:29:55.806 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:29:55.806 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.807 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 512, 2021-01-15T16:29:55.807 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 512, 2021-01-15T16:29:55.807 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2021-01-15T16:29:55.807 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2021-01-15T16:29:55.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:29:55.808 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.808 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.808 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:29:55.808 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.808 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:29:55.809 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 167503724554 2021-01-15T16:29:55.809 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.809 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.809 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:29:55.810 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672977 2021-01-15T16:29:55.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.810 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.810 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:29:55.810 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 171798691850 2021-01-15T16:29:55.811 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.811 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.811 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:29:55.811 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.811 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:29:55.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 32 2021-01-15T16:29:55.812 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.812 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:29:55.812 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.813 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:29:55.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:29:55.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2021-01-15T16:29:55.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:29:55.814 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.814 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.814 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:29:55.814 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:29:55.815 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:29:55.815 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:29:55.815 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.815 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.816 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:29:55.816 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 32, 2021-01-15T16:29:55.816 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2021-01-15T16:29:55.816 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:29:55.817 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.817 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.817 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:29:55.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:29:55.817 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:29:55.818 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:29:55.818 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.818 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:29:55.818 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:29:55.818 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 4, 2021-01-15T16:29:55.819 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:29:19.469361+0000", 2021-01-15T16:29:55.819 INFO:tasks.workunit.client.0.smithi145.stdout: "services": { 2021-01-15T16:29:55.819 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": { 2021-01-15T16:29:55.819 INFO:tasks.workunit.client.0.smithi145.stdout: "daemons": { 2021-01-15T16:29:55.820 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": "", 2021-01-15T16:29:55.820 INFO:tasks.workunit.client.0.smithi145.stdout: "0": { 2021-01-15T16:29:55.820 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:29:55.820 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:29:55.820 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:29:55.821 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:29:55.821 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:29:55.821 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:29:55.821 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.821 INFO:tasks.workunit.client.0.smithi145.stdout: "2": { 2021-01-15T16:29:55.822 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:29:55.822 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:29:55.822 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:29:55.822 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:29:55.822 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:29:55.823 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:29:55.823 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.823 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.823 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.824 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:29:55.824 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:29:55.825 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:29:55.825 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 360, 2021-01-15T16:29:55.825 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:29:55.825 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:29:55.825 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:29:55.826 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:29:55.827 INFO:tasks.workunit.client.0.smithi145.stderr:report 334738367 2021-01-15T16:29:55.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:29:55.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2021-01-15T16:29:55.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2021-01-15T16:29:55.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2021-01-15T16:29:55.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:29:55.828 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2021-01-15T16:29:55.937 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:29:55.937 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.938 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.938 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.938 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.938 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.940 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.940 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.940 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.940 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.940 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.942 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.942 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.942 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.942 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.943 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.943 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.943 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.943 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.943 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:29:55.944 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:29:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:29:55.945 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:29:55.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2021-01-15T16:29:56.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:29:56.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2021-01-15T16:29:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:29:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:29:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:29:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:29:56.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:29:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:29:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:29:56.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:29:56.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:29:56.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:29:56.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:29:56.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:29:56.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:29:56.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:29:56.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:29:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:29:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:29:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:29:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:29:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:56.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:29:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724557 2021-01-15T16:29:56.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724557 2021-01-15T16:29:56.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724557' 2021-01-15T16:29:56.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:56.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:29:56.793 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:29:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:29:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:29:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:29:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:29:56.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:29:56.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691852 2021-01-15T16:29:56.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691852 2021-01-15T16:29:56.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-167503724557 2-171798691852' 2021-01-15T16:29:56.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:29:56.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-167503724557 2021-01-15T16:29:56.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:29:56.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:29:56.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-167503724557 2021-01-15T16:29:56.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:29:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724557 2021-01-15T16:29:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 167503724557' 2021-01-15T16:29:56.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:29:56.942 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 167503724557 2021-01-15T16:29:57.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724555 -lt 167503724557 2021-01-15T16:29:57.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:29:58.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:29:58.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:29:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724557 -lt 167503724557 2021-01-15T16:29:58.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:29:58.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-171798691852 2021-01-15T16:29:58.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:29:58.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:29:58.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-171798691852 2021-01-15T16:29:58.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:29:58.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691852 2021-01-15T16:29:58.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 171798691852' 2021-01-15T16:29:58.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:29:58.746 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 171798691852 2021-01-15T16:29:59.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691853 -lt 171798691852 2021-01-15T16:29:59.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:29:59.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:29:59.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:29:59.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 32 == 0 2021-01-15T16:29:59.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:29:59.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:29:59.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:29:59.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:29:59.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:29:59.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:29:59.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:00.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:00.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:00.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:00.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:00.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:00.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:30:00.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:30:00.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:30:00.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:30:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:00.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:00.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:01.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:01.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:01.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:01.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:01.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:01.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:01.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:01.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:01.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:01.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:01.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:01.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:01.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:02.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:02.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:02.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:30:02.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:30:02.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:02.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:02.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:02.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:02.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:02.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:02.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:02.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:02.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:02.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:02.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:02.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:03.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:03.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:03.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:03.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:03.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:03.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:03.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:30:03.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:03.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:30:04.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:04.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:04.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:04.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:04.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:04.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:04.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:04.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:04.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:04.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:04.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:04.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:05.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:05.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:05.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:05.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:05.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:05.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:05.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:05.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:05.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:05.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:05.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:05.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:30:05.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:05.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:30:06.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:06.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:06.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:06.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:06.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:06.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:06.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:06.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:06.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:06.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:06.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:06.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:07.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:07.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:07.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:07.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:30:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:07.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:30:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:09.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:09.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:09.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:09.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:09.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:09.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:09.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:09.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:09.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:09.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:10.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:10.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:10.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:10.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:10.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:10.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:10.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:10.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:10.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:10.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:30:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:30:14.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:14.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:14.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:14.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:14.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:14.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:14.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:14.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:14.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:14.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:14.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:14.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:15.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:15.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:15.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:15.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:15.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:15.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:15.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:15.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:30:15.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:15.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:30:21.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:21.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:21.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:21.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:22.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:22.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:22.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:22.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:22.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:22.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:22.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:22.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:22.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:22.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:22.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:22.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:22.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:23.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:23.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:23.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:30:23.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:30:23.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:30:30.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:30:30.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:30:30.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:30:30.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:30:30.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:30:30.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:30:30.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:30:30.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:30:31.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:30:31.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:30:31.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:30:31.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:30:31.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 32 2021-01-15T16:30:31.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:30:31.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:30:31.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:30:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:30:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:30:31.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:30:31.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:30:31.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:30:32.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:30:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:30:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:30:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:30:32.624 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:30:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "21126fff-feba-4bfc-abd7-72ef9c949fda", 2021-01-15T16:30:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:30:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:30:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:30:32.623380+0000", 2021-01-15T16:30:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:30:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:30:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:30:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:30:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:30:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:30:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:30:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:30:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:30:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:30:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:30:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:30:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:30:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:30:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:30:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:30:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:30:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:30:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:30:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:30:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:30:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:30:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2021-01-15T16:30:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 64 2021-01-15T16:30:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:30:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.634 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.2 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.4 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.5 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.6 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.9 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.a is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.b is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.c is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.d is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.e is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.f is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.10 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.11 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.12 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.13 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.15 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.16 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.17 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.18 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.19 is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1a is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1b is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1c is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1d is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1e is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.1f is stuck undersized for 72s, current state active+undersized+degraded, last acting [0,2]" 2021-01-15T16:30:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:30:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:30:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:30:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:30:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:30:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:30:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:30:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:30:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:30:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:30:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:30:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:30:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:30:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:30:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:30:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:30:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:28:00.614842Z", 2021-01-15T16:30:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:28:00.614842Z", 2021-01-15T16:30:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:30:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:30:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:30:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:30:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:30:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:30:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:30:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:30:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:30:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:30:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:30:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:30:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:30:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:30:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:30:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:30:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:30:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:30:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:30:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:30:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:30:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:30:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:30:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:30:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:30:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:30:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:30:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:30:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:30:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:30:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 46, 2021-01-15T16:30:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:30:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:28:01.192050+0000", 2021-01-15T16:30:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:29:19.021391+0000", 2021-01-15T16:30:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:29:17.089890+0000", 2021-01-15T16:30:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:29:18.485078+0000", 2021-01-15T16:30:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:30:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:30:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:30:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:30:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:30:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:30:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:30:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:30:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:30:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:30:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 7, 2021-01-15T16:30:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:30:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:30:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:30:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:30:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:30:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:30:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:30:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:30:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:30:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:30:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:30:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:30:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:28:27.965281+0000", 2021-01-15T16:30:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T16:30:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T16:30:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:30:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:30:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:30:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T16:30:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:30:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:30:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:30:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:30:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:30:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "on", 2021-01-15T16:30:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 32, 2021-01-15T16:30:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:30:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 32, 2021-01-15T16:30:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 32, 2021-01-15T16:30:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 32, 2021-01-15T16:30:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:30:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:30:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:30:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:30:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:30:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:30:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:30:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "32", 2021-01-15T16:30:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:30:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:30:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "32", 2021-01-15T16:30:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:30:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T16:30:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T16:30:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T16:30:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T16:30:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:30:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:30:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:30:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:30:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:30:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:30:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:30:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:30:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:30:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:30:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T16:30:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:30:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:30:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:30:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:30:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:30:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:30:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:30:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:30:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:30:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:30:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T16:30:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:30:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:30:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:30:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:30:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:30:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:30:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "0b3ad672-7cb8-48b8-8f80-595c5ea20d86", 2021-01-15T16:30:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:30:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:30:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:30:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:30:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:30:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2021-01-15T16:30:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 39, 2021-01-15T16:30:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 44, 2021-01-15T16:30:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 34, 2021-01-15T16:30:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:30:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:30:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:30:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:30:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:30:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:30:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:30:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:30:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:30:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:30:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:30:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:30:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:30:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65028 2021-01-15T16:30:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/65028", 2021-01-15T16:30:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/65028", 2021-01-15T16:30:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/65028", 2021-01-15T16:30:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/65028", 2021-01-15T16:30:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:30:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:30:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:30:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:30:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "63348b66-8664-42fb-9057-333e23903fe5", 2021-01-15T16:30:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:30:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:30:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:30:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:30:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:30:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:30:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:30:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 32, 2021-01-15T16:30:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 35, 2021-01-15T16:30:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:30:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:30:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:30:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:30:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:30:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:30:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:30:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:30:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:30:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:30:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:30:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:30:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:30:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 59716 2021-01-15T16:30:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/59716", 2021-01-15T16:30:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/59716", 2021-01-15T16:30:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/59716", 2021-01-15T16:30:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/59716", 2021-01-15T16:30:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:30:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:30:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:30:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "3b18e4eb-df97-4d4b-a727-c5d699863dd4", 2021-01-15T16:30:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:30:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:30:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:30:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:30:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:30:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 30, 2021-01-15T16:30:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 40, 2021-01-15T16:30:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:30:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 36, 2021-01-15T16:30:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:30:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:30:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:30:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:30:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:30:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:30:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:30:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:30:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:30:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:30:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:30:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:30:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:30:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:30:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:30:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:30:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 65517 2021-01-15T16:30:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/65517", 2021-01-15T16:30:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/65517", 2021-01-15T16:30:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/65517", 2021-01-15T16:30:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/65517", 2021-01-15T16:30:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:30:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:30:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:30:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:30:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:30:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:10.269445+0000", 2021-01-15T16:30:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:30:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:30:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:30:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:30:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:09.527491+0000", 2021-01-15T16:30:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:30:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:30:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:10.795129+0000", 2021-01-15T16:30:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:30:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:30:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:30:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:30:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:17.519770+0000", 2021-01-15T16:30:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:30:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:30:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:29:11.337040+0000", 2021-01-15T16:30:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:30:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:30:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:30:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:30:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:28:25.473895+0000", 2021-01-15T16:30:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:30:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:30:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:30:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:30:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:30:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:30:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:30:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:30:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:30:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:30:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:30:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:30:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:30:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:30:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:30:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:30:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:30:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:30:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:30:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:30:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:30:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:30:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:30:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:30:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:30:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:30:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/65028,v1:127.0.0.1:6805/65028]", 2021-01-15T16:30:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:30:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:30:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:30:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:30:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:30:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:30:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:30:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:30:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:30:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:30:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:30:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:30:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:30:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:30:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:30:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:30:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:30:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:30:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:30:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:30:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:30:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:30:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:30:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:30:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:30:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/65028,v1:127.0.0.1:6803/65028]", 2021-01-15T16:30:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:30:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/65028,v1:127.0.0.1:6809/65028]", 2021-01-15T16:30:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/65028,v1:127.0.0.1:6807/65028]", 2021-01-15T16:30:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:30:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:30:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:30:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:30:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:30:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:30:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:30:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:30:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:30:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:30:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:30:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:30:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:30:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:30:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:30:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/59716,v1:127.0.0.1:6813/59716]", 2021-01-15T16:30:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:30:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:30:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:30:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:30:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:30:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:30:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:30:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:30:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:30:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:30:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:30:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:30:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:30:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:30:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:30:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:30:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:30:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:30:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:30:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:30:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:30:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:30:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:30:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:30:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:30:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/59716,v1:127.0.0.1:6811/59716]", 2021-01-15T16:30:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:30:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/59716,v1:127.0.0.1:6817/59716]", 2021-01-15T16:30:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/59716,v1:127.0.0.1:6815/59716]", 2021-01-15T16:30:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:30:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:30:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:30:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:30:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:30:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:30:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:30:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:30:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:30:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:30:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:30:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:30:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:30:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:30:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:30:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/65517,v1:127.0.0.1:6813/65517]", 2021-01-15T16:30:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:30:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:30:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:30:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:30:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:30:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:30:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:30:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:30:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:30:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:30:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:30:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:30:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:30:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:30:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:30:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:30:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:30:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:30:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:30:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:30:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:30:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:30:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:30:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:30:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:30:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/65517,v1:127.0.0.1:6811/65517]", 2021-01-15T16:30:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:30:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/65517,v1:127.0.0.1:6817/65517]", 2021-01-15T16:30:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/65517,v1:127.0.0.1:6815/65517]", 2021-01-15T16:30:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:30:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:30:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:30:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:30:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:30:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:30:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:30:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:30:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:30:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:30:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:30:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:30:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:30:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:30:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:30:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:30:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:30:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:30:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:30:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 39 2021-01-15T16:30:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:30:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:30:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:30:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40 2021-01-15T16:30:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:30:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 46, 2021-01-15T16:30:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:30:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:30:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:30:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:30:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:30:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:30:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:30:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:30:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:30:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:30:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:30:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:30:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:30:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:30:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:30:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:30:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:30:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:30:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:30:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:30:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:30:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:30:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:30:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:30:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:30:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:30:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:30:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:30:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:30:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:30:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:30:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:30:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:30:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:30:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:30:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:30:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:30:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:30:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:30:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:30:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:30:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:30:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:30:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:30:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:30:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:30:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:30:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:30:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:30:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:30:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:30:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:30:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:30:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:30:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:30:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:30:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:30:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:30:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:30:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:30:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:30:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:30:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:30:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:30:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:30:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:30:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:30:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:30:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:30:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:30:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:30:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:30:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:30:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:30:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:30:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:30:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:30:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:30:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:30:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:30:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:30:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:30:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:30:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:30:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:30:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:30:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:30:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:30:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:30:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:30:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:30:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:30:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:30:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:30:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:30:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:30:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:30:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:30:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:30:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:30:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:30:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:30:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:30:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:30:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:30:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:30:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:30:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:30:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:30:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T16:30:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:30:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:30:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:30:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:30:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:30:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:30:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:30:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:30:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:30:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:30:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:30:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:30:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:30:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:30:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:30:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:30:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:30:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:30:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:30:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:30:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:30:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:30:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:30:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:30:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:30:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:30:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:30:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:30:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:30:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:30:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:30:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:30:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:30:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2021-01-15T16:30:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 32, 2021-01-15T16:30:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:30:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:30:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:30:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:30:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2021-01-15T16:30:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:30:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:30:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:30:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:30:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:30:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:30:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:30:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:30:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:30:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:30:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:30:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:30:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:30:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:30:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:30:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:30:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:30:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:30:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:30:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:30:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:30:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:30:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:30:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:30:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:30:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:30:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:30:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:30:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:30:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:30:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:30:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:30:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:30:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:30:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:30:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:30:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:30:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:30:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:30:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:30:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:30:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:30:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:30:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:30:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:30:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:30:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:30:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:30:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:30:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:30:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 513, 2021-01-15T16:30:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 513, 2021-01-15T16:30:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2021-01-15T16:30:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2021-01-15T16:30:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:30:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:30:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:30:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:30:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2021-01-15T16:30:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:30:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:30:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:30:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:30:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19572, 2021-01-15T16:30:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2196, 2021-01-15T16:30:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:30:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 17342, 2021-01-15T16:30:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553228, 2021-01-15T16:30:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:30:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:30:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322102505472, 2021-01-15T16:30:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:30:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2248704, 2021-01-15T16:30:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 455784, 2021-01-15T16:30:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:30:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:30:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:30:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:30:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17758312 2021-01-15T16:30:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:30:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:30:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:30:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:30:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:30:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:30:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:30:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:30:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 54, 2021-01-15T16:30:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 54, 2021-01-15T16:30:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 54000000, 2021-01-15T16:30:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 54000000 2021-01-15T16:30:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:30:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:30:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:30:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:30:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:30:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:30:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 65, 2021-01-15T16:30:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:30:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:30:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:30:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:30:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19572, 2021-01-15T16:30:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2196, 2021-01-15T16:30:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:30:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 17342, 2021-01-15T16:30:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553228, 2021-01-15T16:30:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:30:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:30:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322102505472, 2021-01-15T16:30:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:30:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2248704, 2021-01-15T16:30:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 455784, 2021-01-15T16:30:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:30:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:30:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:30:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:30:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17758312 2021-01-15T16:30:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:30:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:30:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:30:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:30:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:30:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:30:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:30:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:30:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 54, 2021-01-15T16:30:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 54, 2021-01-15T16:30:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 54000000, 2021-01-15T16:30:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 54000000 2021-01-15T16:30:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:30:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:30:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:30:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:30:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 32, 2021-01-15T16:30:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:30:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 47947, 2021-01-15T16:30:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 100, 2021-01-15T16:30:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T16:30:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 300, 2021-01-15T16:30:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:30:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:30:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 100, 2021-01-15T16:30:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:30:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:30:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 100, 2021-01-15T16:30:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:30:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:30:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:30:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:30:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:30:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:30:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:30:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:30:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:30:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:30:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:30:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:30:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:30:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:30:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:30:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:30:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:30:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:30:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:30:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:30:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:30:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:30:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:30:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:30:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:30:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:30:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:30:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:30:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:30:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:30:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:30:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:30:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:30:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:30:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1220608, 2021-01-15T16:30:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 142522, 2021-01-15T16:30:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:30:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:30:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:30:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:30:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:30:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 513, 2021-01-15T16:30:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 513, 2021-01-15T16:30:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 64, 2021-01-15T16:30:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 64, 2021-01-15T16:30:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:30:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:30:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:30:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 167503724564 2021-01-15T16:30:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672977 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:30:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 171798691859 2021-01-15T16:30:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:30:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:30:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 32 2021-01-15T16:30:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:30:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:30:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:30:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2021-01-15T16:30:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:30:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:30:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:30:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:30:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:30:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:30:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 32, 2021-01-15T16:30:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 32, 2021-01-15T16:30:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:30:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:30:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:30:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:30:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:30:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:30:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:30:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 6, 2021-01-15T16:30:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:30:17.483944+0000", 2021-01-15T16:30:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2021-01-15T16:30:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:30:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:30:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:30:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 460, 2021-01-15T16:30:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:30:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:30:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:30:32.894 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:30:32.894 INFO:tasks.workunit.client.0.smithi145.stderr:report 357699504 2021-01-15T16:30:32.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:30:32.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2021-01-15T16:30:32.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:32.895 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2021-01-15T16:30:33.011 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:33.011 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.011 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.011 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.012 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.012 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.012 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.012 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.012 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.013 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.013 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.013 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.013 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.013 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.014 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.014 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.014 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.014 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.015 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.015 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.015 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.015 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.015 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.016 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.016 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.016 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.016 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.016 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.017 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.017 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.017 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.017 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 46'101 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.017 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:33.018 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:33.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:30:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2021-01-15T16:30:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:30:33.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:30:33.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:30:33.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:30:33.024 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:30:33.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:30:33.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:30:33.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok set_recovery_delay 100000 2021-01-15T16:30:33.126 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:30:33.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2021-01-15T16:30:33.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2021-01-15T16:30:33.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:30:33.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:30:33.136 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:30:33.136 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:30:33.136 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:30:33.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:30:33.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:30:33.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok set_recovery_delay 100000 2021-01-15T16:30:33.238 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:30:33.246 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2021-01-15T16:30:33.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2021-01-15T16:30:33.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:33.622 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:33.623 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.623 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.623 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.624 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.624 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.624 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.625 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.625 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.625 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.625 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.626 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.626 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.626 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.627 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.627 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.627 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.628 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.628 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.628 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.629 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.629 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.629 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.630 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.630 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.630 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.631 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.631 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.632 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.632 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.632 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.633 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 46'101 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.633 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:127 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:33.633 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:33.634 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:33.635 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:33.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2021-01-15T16:30:34.400 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2021-01-15T16:30:34.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-01-15T16:30:34.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:30:34.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:30:34.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:30:34.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:30:34.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:30:34.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:30:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:30:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:30:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:30:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:30:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:30:34.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:30:34.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:30:34.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:30:34.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:30:34.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:30:34.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:30:34.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:30:34.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:30:34.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:30:34.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:30:34.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:30:34.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:30:34.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:30:34.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:30:34.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:30:34.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:30:34.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:30:34.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:30:34.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:30:34.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:30:34.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:30:34.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:30:34.420 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:30:34.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:30:34.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:30:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:30:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:30:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:30:34.441 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:34.441+0000 7efe6e48c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:34.442 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:34.442+0000 7efe6e48c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:34.444 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:34.444+0000 7efe6e48c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:34.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2021-01-15T16:30:34.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2021-01-15T16:30:34.961 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:34.959+0000 7efe6e48c080 -1 Falling back to public interface 2021-01-15T16:30:36.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:36.849+0000 7efe6e48c080 -1 osd.1 33 log_to_monitors {default=true} 2021-01-15T16:30:39.814 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2021-01-15T16:30:39.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2021-01-15T16:30:39.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:40.196 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:40.196 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 46'101 46:129 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:40.204 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:40.204 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:40.204 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:40.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2021-01-15T16:30:40.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:30:40.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:30:40.209 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:30:40.209 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:30:40.209 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:30:40.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:30:40.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:30:40.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok set_recovery_delay 100000 2021-01-15T16:30:40.311 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:30:40.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2021-01-15T16:30:41.179 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2021-01-15T16:30:41.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2021-01-15T16:30:41.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:30:41.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:30:41.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:30:41.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:30:41.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:30:41.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:30:41.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:30:41.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:30:41.191 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:30:41.597 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 39 up_thru 44 down_at 34 last_clean_interval [5,30) [v2:127.0.0.1:6802/65028,v1:127.0.0.1:6803/65028] [v2:127.0.0.1:6804/65028,v1:127.0.0.1:6805/65028] exists,up 0b3ad672-7cb8-48b8-8f80-595c5ea20d86 2021-01-15T16:30:41.598 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:30:41.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:30:41.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:30:41.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:30:41.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2021-01-15T16:30:41.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:30:41.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:30:41.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:30:41.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:30:41.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:30:41.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:30:41.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:30:41.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:30:42.017 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/68147,v1:127.0.0.1:6819/68147] [v2:127.0.0.1:6820/68147,v1:127.0.0.1:6821/68147] exists,up 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:30:42.018 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:30:42.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:30:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:30:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:30:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2021-01-15T16:30:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:30:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:30:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:30:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:30:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:30:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:30:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:30:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:30:42.411 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 40 up_thru 0 down_at 36 last_clean_interval [15,30) [v2:127.0.0.1:6810/65517,v1:127.0.0.1:6811/65517] [v2:127.0.0.1:6812/65517,v1:127.0.0.1:6813/65517] exists,up 3b18e4eb-df97-4d4b-a727-c5d699863dd4 2021-01-15T16:30:42.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:30:42.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:30:42.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:30:42.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:42.811 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:42.811 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.812 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.812 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.813 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.813 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.813 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.814 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.814 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.815 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.815 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.815 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.815 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.816 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.816 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.816 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.816 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.817 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.818 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.819 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 46'101 46:129 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.820 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:42.820 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:42.820 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:42.821 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:42.822 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2021-01-15T16:30:42.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2021-01-15T16:30:42.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:43.197 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:43.197 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454329+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453294+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453608+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453635+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.198 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453607+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451769+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427507+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453220+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.199 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453214+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452796+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451771+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452814+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452802+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452825+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453171+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452556+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452359+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451742+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.201 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452458+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453663+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453548+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.453943+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339875+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.202 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.427554+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.339871+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.454046+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452414+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.451829+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452393+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.204 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2021-01-15T16:29:19.451765+0000 46'101 46:129 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.204 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2021-01-15T16:29:19.452924+0000 30'100 46:128 [0,2] 0 [0,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:43.204 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:43.204 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:43.205 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:43.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2021-01-15T16:30:55.553 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2021-01-15T16:30:55.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2021-01-15T16:30:55.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:55.942 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830025+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830672+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.944 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831085+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.945 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831618+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.946 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.831277+0000 30'100 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.946 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.826441+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.947 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828715+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.949 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830765+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.950 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831278+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.950 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831076+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.952 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.826991+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.953 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.829672+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.953 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830525+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.954 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.829101+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.955 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.829202+0000 30'100 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.956 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.827319+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.957 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.829953+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.958 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830242+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.959 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.828422+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.959 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828663+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.960 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830987+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.960 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2021-01-15T16:30:41.828906+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.961 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.829292+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.962 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828034+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.962 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.828141+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.963 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2021-01-15T16:30:41.832510+0000 30'100 52:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.964 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831303+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.964 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.826697+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.964 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2021-01-15T16:30:41.828380+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.964 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2021-01-15T16:30:41.828847+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.965 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.829905+0000 46'101 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.965 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830210+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:55.965 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:55.965 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:55.966 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:55.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2021-01-15T16:30:55.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:30:55.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:30:55.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:30:55.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:30:55.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:30:56.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:30:56.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.33425.out 2021-01-15T16:30:56.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33425.out 2021-01-15T16:30:56.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:30:56.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:30:56.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:30:56.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:30:56.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-01-15T16:30:56.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.33425.out 2021-01-15T16:30:56.803 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=50/52 n=3 ec=23/23 lis/c=50/50 les/c/f=52/52/0 sis=50 pruub=13625554253ns) 2021-01-15T16:30:56.836 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:023dd3d2:::existing_12:head# 2021-01-15T16:30:56.837 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04c40a0c:::existing_93:head# 2021-01-15T16:30:56.837 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:064ff5fb:::existing_36:head# 2021-01-15T16:30:56.838 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2021-01-15T16:30:56.838 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2021-01-15T16:30:56.838 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2021-01-15T16:30:56.839 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-01-15T16:30:56.953 INFO:tasks.workunit.client.0.smithi145.stderr:Remove successful 2021-01-15T16:30:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.33425.out 2021-01-15T16:30:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:30:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:30:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:30:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:30:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-01-15T16:30:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.33425.out 2021-01-15T16:30:57.980 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e52 pg_nums {1={23=1,32=32}} deleted_pools ) 2021-01-15T16:30:57.981 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 1.0 2021-01-15T16:30:57.981 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:023dd3d2:::existing_12:head# 2021-01-15T16:30:57.981 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:30:58.047 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04c40a0c:::existing_93:head# 2021-01-15T16:30:58.047 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:30:58.080 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:064ff5fb:::existing_36:head# 2021-01-15T16:30:58.080 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:30:58.113 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 52 info 1.0( v 30'100 (30'90,30'100] local-lis/les=50/52 n=3 ec=23/23 lis/c=50/50 les/c/f=52/52/0 sis=50 pruub=13625554253ns) 2021-01-15T16:30:58.147 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2021-01-15T16:30:58.668 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2021-01-15T16:30:58.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2021-01-15T16:30:58.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:30:59.051 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:30:59.051 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830025+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.051 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830672+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.052 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831085+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.052 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831618+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.052 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.831277+0000 30'100 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.052 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.826441+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.053 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828715+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.053 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830765+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.053 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831278+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.053 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831076+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.053 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.826991+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.054 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.829672+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.054 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830525+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.054 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.829101+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.054 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.829202+0000 30'100 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.054 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.827319+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.055 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.829953+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.055 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830242+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.055 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.828422+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.055 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828663+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.055 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.830987+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.056 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2021-01-15T16:30:41.828906+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.056 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.829292+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.056 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.828034+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.056 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.828141+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.056 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovering+degraded+wait 2021-01-15T16:30:41.832510+0000 30'100 52:135 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.057 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.831303+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.057 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2021-01-15T16:30:41.826697+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.057 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2021-01-15T16:30:41.828380+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.057 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2021-01-15T16:30:41.828847+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.057 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2021-01-15T16:30:41.829905+0000 46'101 52:134 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.058 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2021-01-15T16:30:41.830210+0000 30'100 52:136 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:30:59.058 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:30:59.058 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:30:59.058 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:30:59.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2021-01-15T16:30:59.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:30:59.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:30:59.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:30:59.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:30:59.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:30:59.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:30:59.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:30:59.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:30:59.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:30:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:30:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:30:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:30:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:30:59.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:30:59.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:30:59.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:30:59.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:30:59.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:30:59.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:30:59.069 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:30:59.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:30:59.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:30:59.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:30:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:30:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:30:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:30:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:30:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:30:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:30:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:30:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:30:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:30:59.072 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:30:59.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:30:59.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:30:59.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:30:59.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:30:59.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:30:59.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:30:59.093 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:59.092+0000 7f094fc9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:59.094 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:59.094+0000 7f094fc9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:59.095 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:59.095+0000 7f094fc9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:30:59.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:30:59.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:30:59.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:30:59.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:30:59.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:30:59.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:30:59.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:30:59.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:30:59.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:30:59.465 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:30:59.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:30:59.612+0000 7f094fc9d080 -1 Falling back to public interface 2021-01-15T16:30:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/68147,v1:127.0.0.1:6819/68147] [v2:127.0.0.1:6820/68147,v1:127.0.0.1:6821/68147] exists,up 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:30:59.941 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:30:59.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:30:59.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:30:59.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:30:59.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2021-01-15T16:30:59.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:30:59.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:30:59.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:30:59.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:30:59.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:30:59.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:30:59.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:30:59.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:31:00.342 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 35 last_clean_interval [10,33) [v2:127.0.0.1:6818/68147,v1:127.0.0.1:6819/68147] [v2:127.0.0.1:6820/68147,v1:127.0.0.1:6821/68147] exists,up 63348b66-8664-42fb-9057-333e23903fe5 2021-01-15T16:31:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:31:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:31:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:31:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2021-01-15T16:31:01.254 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:01.253+0000 7f094fc9d080 -1 osd.1 52 log_to_monitors {default=true} 2021-01-15T16:31:20.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2021-01-15T16:31:20.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:31:20.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:31:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:31:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:20.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:31:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2021-01-15T16:31:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok dump_ops_in_flight 2021-01-15T16:31:20.455 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:31:20.455 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [], 2021-01-15T16:31:20.455 INFO:tasks.workunit.client.0.smithi145.stdout: "num_ops": 0 2021-01-15T16:31:20.455 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:31:20.463 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2021-01-15T16:31:20.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2021-01-15T16:31:20.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:31:20.836 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:31:20.836 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.664698+0000 30'100 63:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.837 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-01-15T16:31:11.084732+0000 30'100 62:149 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:11.084670+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.837 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:08.436394+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.837 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:13.178227+0000 30'100 63:159 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:13.178163+0000 30'100 2021-01-15T16:31:13.178163+0000 0 2021-01-15T16:31:20.837 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:12.663834+0000 30'100 63:164 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.838 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:10.364398+0000 30'100 62:145 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:07.183251+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.838 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:09.979193+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:09.979128+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.838 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.586730+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.838 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.664266+0000 30'100 63:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.838 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:10.398646+0000 30'100 62:145 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.839 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:08.435138+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.839 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.583396+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.839 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:12.633657+0000 30'100 63:162 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.839 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:10.398062+0000 30'100 62:145 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.839 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-01-15T16:31:14.055902+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:14.055831+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.840 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.587547+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.840 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2021-01-15T16:31:14.991588+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:14.991520+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.841 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:11.171111+0000 30'100 63:159 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:11.171049+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.841 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2021-01-15T16:31:12.663823+0000 30'100 63:162 [1,2,0] 1 [1,2,0] 1 30'100 2021-01-15T16:31:10.943823+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.841 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:10.339576+0000 30'100 62:145 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:08.193537+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.841 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.586512+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.841 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:08.435473+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2021-01-15T16:31:10.398250+0000 52'101 62:153 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:12.663406+0000 30'100 63:160 [1,2,0] 1 [1,2,0] 1 30'100 2021-01-15T16:31:08.992686+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2021-01-15T16:31:02.586689+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2021-01-15T16:31:12.038457+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:12.038388+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:10.398679+0000 30'100 62:145 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.843 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2021-01-15T16:31:12.664383+0000 30'100 63:162 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean 2021-01-15T16:31:02.585763+0000 30'100 63:156 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-01-15T16:31:16.117541+0000 30'100 64:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:16.117474+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 4 0 0 0 0 1579 0 0 17 17 active+clean 2021-01-15T16:31:12.322236+0000 46'101 62:150 [0,1,2] 0 [0,1,2] 0 46'101 2021-01-15T16:31:12.322166+0000 46'101 2021-01-15T16:31:12.322166+0000 0 2021-01-15T16:31:20.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.663810+0000 30'100 63:160 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:20.845 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:31:20.845 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:31:20.845 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:31:20.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-01-15T16:31:20.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T16:31:20.980 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:31:20.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-01-15T16:31:20.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T16:31:21.149 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:31:21.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2021-01-15T16:31:21.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2021-01-15T16:31:21.291 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:31:21.300 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2021-01-15T16:31:21.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2021-01-15T16:31:21.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2021-01-15T16:31:21.677 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:31:21.677 INFO:tasks.workunit.client.0.smithi145.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.664698+0000 30'100 66:163 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.677 INFO:tasks.workunit.client.0.smithi145.stdout:1.1e 3 0 0 0 0 1449 0 0 10 10 active+clean 2021-01-15T16:31:18.563990+0000 30'100 66:163 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:11.084670+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.677 INFO:tasks.workunit.client.0.smithi145.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:08.436394+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1.1c 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:13.178227+0000 30'100 66:162 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:13.178163+0000 30'100 2021-01-15T16:31:13.178163+0000 0 2021-01-15T16:31:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:12.663834+0000 30'100 66:167 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:10.364398+0000 30'100 65:148 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:07.183251+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:09.979193+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:09.979128+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.586730+0000 30'100 66:159 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.679 INFO:tasks.workunit.client.0.smithi145.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.664266+0000 30'100 66:163 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.679 INFO:tasks.workunit.client.0.smithi145.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:18.656578+0000 30'100 66:161 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.679 INFO:tasks.workunit.client.0.smithi145.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:08.435138+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.679 INFO:tasks.workunit.client.0.smithi145.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.583396+0000 30'100 66:159 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.679 INFO:tasks.workunit.client.0.smithi145.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:12.633657+0000 30'100 66:165 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.12 1 0 0 0 0 483 0 0 16 16 active+clean 2021-01-15T16:31:19.148296+0000 30'100 66:152 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:19.148218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.11 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-01-15T16:31:14.055902+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:14.055831+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.587547+0000 30'100 66:159 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2021-01-15T16:31:14.991588+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:14.991520+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.680 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2021-01-15T16:31:11.171111+0000 30'100 66:162 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:11.171049+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.681 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2021-01-15T16:31:12.663823+0000 30'100 66:165 [1,2,0] 1 [1,2,0] 1 30'100 2021-01-15T16:31:10.943823+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.681 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 1 0 0 0 0 483 0 0 16 16 active+clean 2021-01-15T16:31:18.189690+0000 30'100 65:154 [0,1,2] 0 [0,1,2] 0 30'100 2021-01-15T16:31:18.189630+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.681 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2021-01-15T16:31:02.586512+0000 30'100 66:159 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.681 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:08.435473+0000 30'100 63:129 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.681 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2021-01-15T16:31:18.656829+0000 52'101 66:163 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.682 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:12.663406+0000 30'100 66:163 [1,2,0] 1 [1,2,0] 1 30'100 2021-01-15T16:31:08.992686+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.682 INFO:tasks.workunit.client.0.smithi145.stdout:1.8 3 0 0 0 0 1449 0 0 10 10 active+clean 2021-01-15T16:31:19.054830+0000 30'100 66:163 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:19.054764+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.682 INFO:tasks.workunit.client.0.smithi145.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2021-01-15T16:31:12.038457+0000 30'100 63:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:12.038388+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.682 INFO:tasks.workunit.client.0.smithi145.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2021-01-15T16:31:10.398679+0000 30'100 65:148 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.683 INFO:tasks.workunit.client.0.smithi145.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2021-01-15T16:31:12.664383+0000 30'100 66:165 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.683 INFO:tasks.workunit.client.0.smithi145.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2021-01-15T16:31:18.091560+0000 30'100 66:163 [1,0,2] 1 [1,0,2] 1 30'100 2021-01-15T16:31:18.091496+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.683 INFO:tasks.workunit.client.0.smithi145.stdout:1.d 3 0 0 0 0 1449 0 0 16 16 active+clean 2021-01-15T16:31:16.117541+0000 30'100 64:133 [2,0,1] 2 [2,0,1] 2 30'100 2021-01-15T16:31:16.117474+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.683 INFO:tasks.workunit.client.0.smithi145.stdout:1.e 4 0 0 0 0 1579 0 0 14 14 active+clean 2021-01-15T16:31:18.656690+0000 46'101 66:165 [1,0,2] 1 [1,0,2] 1 46'101 2021-01-15T16:31:12.322166+0000 46'101 2021-01-15T16:31:12.322166+0000 0 2021-01-15T16:31:21.684 INFO:tasks.workunit.client.0.smithi145.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2021-01-15T16:31:12.663810+0000 30'100 66:163 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T16:28:28.042218+0000 0'0 2021-01-15T16:28:28.042218+0000 0 2021-01-15T16:31:21.684 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:31:21.685 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:31:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:31:21.688 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2021-01-15T16:31:21.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2021-01-15T16:31:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:31:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2021-01-15T16:31:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:31:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2021-01-15T16:31:21.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:31:21.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2021-01-15T16:31:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:31:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2021-01-15T16:31:21.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2021-01-15T16:31:21.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:31:21.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2021-01-15T16:31:22.091 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_1: (2) No such file or directory 2021-01-15T16:31:22.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2021-01-15T16:31:22.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2021-01-15T16:31:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test/newobject_2: (2) No such file or directory 2021-01-15T16:31:22.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2021-01-15T16:31:22.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-01-15T16:31:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:31:22.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'101 (30'2) modify 1:97db2a0f:::existing_2:head by client.4530.0:1 2021-01-15T16:28:59.440140+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 20 _merge_object_divergent_entries: keeping 30'101 (30'2) modify 1:97db2a0f:::existing_2:head by client.4530.0:1 2021-01-15T16:28:59.440140+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:31:22.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'101 last_divergent_update: 30'101 2021-01-15T16:31:22.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2021-01-15T16:31:22.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:31:22.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2021-01-15T16:31:22.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'101 (30'2) modify 1:97db2a0f:::existing_2:head by client.4530.0:1 2021-01-15T16:28:59.440140+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2021-01-15T16:31:22.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.175+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2021-01-15T16:31:22.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'104 (30'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2021-01-15T16:28:59.452828+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:31:22.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 20 _merge_object_divergent_entries: keeping 30'104 (30'4) delete 1:70b3bf12:::existing_4:head by client.4540.0:1 2021-01-15T16:28:59.452828+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:31:22.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:31:22.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'104 last_divergent_update: 30'104 2021-01-15T16:31:22.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: more recent entry found: 46'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2021-01-15T16:29:55.976738+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2021-01-15T16:31:22.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4534.0:1 2021-01-15T16:28:59.447628+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 20 _merge_object_divergent_entries: keeping 30'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4534.0:1 2021-01-15T16:28:59.447628+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:31:22.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'103 last_divergent_update: 30'103 2021-01-15T16:31:22.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2021-01-15T16:31:22.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4995c700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2021-01-15T16:31:22.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2021-01-15T16:28:59.457053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:31:22.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 20 _merge_object_divergent_entries: keeping 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2021-01-15T16:28:59.457053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:31:22.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2021-01-15T16:31:22.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'106 last_divergent_update: 30'106 2021-01-15T16:31:22.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2021-01-15T16:31:22.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:31:22.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2021-01-15T16:31:22.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2021-01-15T16:28:59.457053+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2021-01-15T16:31:22.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2021-01-15T16:31:22.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.245+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2021-01-15T16:31:22.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2021-01-15T16:28:59.440666+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 20 _merge_object_divergent_entries: keeping 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2021-01-15T16:28:59.440666+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:31:22.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'102 last_divergent_update: 30'102 2021-01-15T16:31:22.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2021-01-15T16:31:22.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:31:22.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2021-01-15T16:31:22.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4531.0:1 2021-01-15T16:28:59.440666+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2021-01-15T16:31:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2021-01-15T16:31:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'105 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4541.0:1 2021-01-15T16:28:59.453757+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 20 _merge_object_divergent_entries: keeping 30'105 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4541.0:1 2021-01-15T16:28:59.453757+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:31:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:31:22.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'105 last_divergent_update: 30'105 2021-01-15T16:31:22.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2021-01-15T16:31:22.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:30:41.246+0000 7efe4a95e700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2021-01-15T16:31:22.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-01-15T16:31:22.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:31:22.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2021-01-15T16:31:22.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:31:22.180 INFO:tasks.workunit.client.0.smithi145.stdout:success 2021-01-15T16:31:22.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2021-01-15T16:31:22.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.33425.out 2021-01-15T16:31:22.182 INFO:tasks.workunit.client.0.smithi145.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.33425.out': No such file or directory 2021-01-15T16:31:22.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2021-01-15T16:31:22.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T16:31:22.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T16:31:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T16:31:22.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2021-01-15T16:31:22.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:31:22.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:31:22.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:31:22.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:31:22.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:31:22.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:31:22.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-01-15T16:31:22.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:31:22.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:31:22.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:31:22.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:31:22.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:31:22.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:31:22.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:31:22.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:31:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:31:22.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:31:22.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:31:22.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:31:22.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:31:22.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:31:22.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:31:22.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:31:22.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:31:22.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:31:22.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:31:22.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:31:22.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:31:22.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:31:22.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:31:22.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:22.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:22.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:31:22.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:31:22.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:31:22.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2021-01-15T16:31:22.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2021-01-15T16:31:22.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2021-01-15T16:31:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2021-01-15T16:31:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:31:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:31:22.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:31:22.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:31:22.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:31:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:31:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:31:22.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:31:22.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:31:22.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:31:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:31:22.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:31:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:31:22.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:31:22.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:31:22.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:31:22.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:31:22.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:31:22.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:31:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:31:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:31:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:31:22.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:31:22.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:22.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:22.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:31:22.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:31:22.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:31:22.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2021-01-15T16:31:22.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:31:22.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:22.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:22.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.33425 2021-01-15T16:31:22.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:31:22.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:31:22.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:31:22.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2021-01-15T16:31:22.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2021-01-15T16:31:22.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2021-01-15T16:31:22.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2021-01-15T16:31:22.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2021-01-15T16:31:22.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2021-01-15T16:31:22.952 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2021-01-15T16:31:22.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2021-01-15T16:31:22.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2021-01-15T16:31:22.954 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:31:22.954 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:31:22.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2021-01-15T16:31:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2021-01-15T16:31:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:31:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:31:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:31:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2021-01-15T16:31:22.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2021-01-15T16:31:23.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:31:23.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:31:23.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:31:23.252 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:31:23.252 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:23.252 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:23.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:23.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:31:23.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:31:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:31:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:31:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:31:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:31:23.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:31:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:31:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:31:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:31:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:31:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:31:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:31:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get fsid 2021-01-15T16:31:23.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:31:23.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:31:23.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:31:23.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:31:23.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:31:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:31:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:31:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:31:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:31:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-mon.a.asok 2021-01-15T16:31:23.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:31:23.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.33425/ceph-mon.a.asok config get mon_host 2021-01-15T16:31:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2021-01-15T16:31:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/divergent-priors 2021-01-15T16:31:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:31:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:31:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:31:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/divergent-priors/x 2021-01-15T16:31:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:31:24.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:31:24.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:31:24.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:31:24.539 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:31:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:24.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:24.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:24.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:31:24.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:31:24.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-01-15T16:31:24.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2021-01-15T16:31:24.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:31:24.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:31:24.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:31:24.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:31:24.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:31:24.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:31:24.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:31:24.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:31:24.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:31:24.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:31:24.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:31:24.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:31:24.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:31:24.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:31:24.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:31:24.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:31:24.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:31:24.572 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:31:24.572 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:24.572 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:24.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:24.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:24.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:31:24.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:31:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:31:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:31:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:31:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:31:24.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:31:24.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:24.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:31:24.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:31:24.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:31:24.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:31:24.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:31:24.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 6412922c-b560-401d-81b8-a8bbe5ef1e10' 2021-01-15T16:31:24.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:31:24.577 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:31:24.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBcwwFgLlYMIxAA6I1H4b2LDcgd1pqxXo0AMA== 2021-01-15T16:31:24.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBcwwFgLlYMIxAA6I1H4b2LDcgd1pqxXo0AMA=="}' 2021-01-15T16:31:24.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6412922c-b560-401d-81b8-a8bbe5ef1e10 -i td/divergent-priors/0/new.json 2021-01-15T16:31:24.937 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:31:24.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/0/new.json 2021-01-15T16:31:24.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBcwwFgLlYMIxAA6I1H4b2LDcgd1pqxXo0AMA== --osd-uuid 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:31:24.979 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.978+0000 7f6458f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:24.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.980+0000 7f6458f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:24.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.981+0000 7f6458f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:24.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.982+0000 7f6458f97080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:31:24.983 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.982+0000 7f6458f97080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:31:24.983 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:24.982+0000 7f6458f97080 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2021-01-15T16:31:25.032 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:25.032+0000 7f6458f97080 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2021-01-15T16:31:29.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/0/keyring 2021-01-15T16:31:29.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:31:29.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:31:29.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:31:29.505 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:31:30.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:31:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:31:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:31:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:31:30.037 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:31:30.065 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:30.065+0000 7f62f5a32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:30.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:30.084+0000 7f62f5a32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:30.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:30.085+0000 7f62f5a32080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:30.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:31:30.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:31:30.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:31:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:31:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:31:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:31:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:30.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:31:30.439 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:31:30.603 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:30.603+0000 7f62f5a32080 -1 Falling back to public interface 2021-01-15T16:31:30.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:31:31.353 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:31.353+0000 7f62f5a32080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:31:31.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:31:31.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:31.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:31:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:31:31.859 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:31:32.269 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/71400,v1:127.0.0.1:6803/71400] [v2:127.0.0.1:6804/71400,v1:127.0.0.1:6805/71400] exists,up 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:31:32.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:31:32.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:31:32.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:31:32.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-01-15T16:31:32.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2021-01-15T16:31:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:31:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:31:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:31:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:31:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:31:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:31:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:31:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:31:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:31:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:31:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:31:32.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:31:32.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:31:32.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:31:32.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:31:32.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:31:32.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:31:32.275 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:31:32.275 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:32.275 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:32.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:32.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:32.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:31:32.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:31:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:31:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:31:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:31:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:31:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:31:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:31:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:31:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:31:32.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:31:32.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:31:32.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 833bc698-267b-4dab-a554-ed9c97578364' 2021-01-15T16:31:32.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:31:32.280 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:31:32.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBkwwFg9ucwERAAiXXNcJMyefHaw/BQno2IUA== 2021-01-15T16:31:32.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBkwwFg9ucwERAAiXXNcJMyefHaw/BQno2IUA=="}' 2021-01-15T16:31:32.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 833bc698-267b-4dab-a554-ed9c97578364 -i td/divergent-priors/1/new.json 2021-01-15T16:31:32.786 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:31:32.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/1/new.json 2021-01-15T16:31:32.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBkwwFg9ucwERAAiXXNcJMyefHaw/BQno2IUA== --osd-uuid 833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:31:32.828 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.828+0000 7f8444250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:32.830 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.829+0000 7f8444250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:32.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.831+0000 7f8444250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:32.833 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.832+0000 7f8444250080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:31:32.833 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.832+0000 7f8444250080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:31:32.833 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.832+0000 7f8444250080 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2021-01-15T16:31:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:32.883+0000 7f8444250080 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2021-01-15T16:31:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/1/keyring 2021-01-15T16:31:37.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:31:37.392 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:31:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:31:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:31:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:31:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:31:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:31:37.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:31:37.928 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:31:37.958 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:37.953+0000 7f76f0f7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:37.960 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:37.955+0000 7f76f0f7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:37.961 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:37.956+0000 7f76f0f7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:31:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:31:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:31:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:31:38.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:31:38.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:38.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:31:38.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:38.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:31:38.330 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:31:38.479 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:38.478+0000 7f76f0f7b080 -1 Falling back to public interface 2021-01-15T16:31:38.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:31:39.284 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:39.283+0000 7f76f0f7b080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:31:39.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:31:39.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:39.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:31:39.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:39.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:31:39.742 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:31:40.192 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/72672,v1:127.0.0.1:6811/72672] [v2:127.0.0.1:6812/72672,v1:127.0.0.1:6813/72672] exists,up 833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:31:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:31:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:31:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:31:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2021-01-15T16:31:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2021-01-15T16:31:40.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/divergent-priors 2021-01-15T16:31:40.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:31:40.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:31:40.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:31:40.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:31:40.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:31:40.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:31:40.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:31:40.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:31:40.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:31:40.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:31:40.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:31:40.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:31:40.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:31:40.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:31:40.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:31:40.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:31:40.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:31:40.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:31:40.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:31:40.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:40.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:31:40.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:31:40.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:31:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:31:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:31:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:31:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:31:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:31:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:31:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:31:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:31:40.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:31:40.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:31:40.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 baffd605-38c3-450d-87ae-a21bfb354b1f' 2021-01-15T16:31:40.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:31:40.202 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:31:40.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBswwFgT1mfDBAAieoA70KpL4blO4QApltePA== 2021-01-15T16:31:40.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBswwFgT1mfDBAAieoA70KpL4blO4QApltePA=="}' 2021-01-15T16:31:40.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new baffd605-38c3-450d-87ae-a21bfb354b1f -i td/divergent-priors/2/new.json 2021-01-15T16:31:40.783 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:31:40.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/divergent-priors/2/new.json 2021-01-15T16:31:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBswwFgT1mfDBAAieoA70KpL4blO4QApltePA== --osd-uuid baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:31:40.827 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.827+0000 7f05a0764080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:40.828 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.828+0000 7f05a0764080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:40.830 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.829+0000 7f05a0764080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:40.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.831+0000 7f05a0764080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:31:40.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.831+0000 7f05a0764080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:31:40.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.831+0000 7f05a0764080 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2021-01-15T16:31:40.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:40.871+0000 7f05a0764080 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2021-01-15T16:31:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/divergent-priors/2/keyring 2021-01-15T16:31:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:31:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:31:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:31:45.485 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:31:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:31:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:31:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:31:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:31:46.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:31:46.022 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:31:46.047 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:46.046+0000 7f842ea00080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:46.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:46.062+0000 7f842ea00080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:46.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:46.064+0000 7f842ea00080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:31:46.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:31:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:31:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:31:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:31:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:31:46.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:46.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:31:46.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:46.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:31:46.420 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:31:46.579 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:46.578+0000 7f842ea00080 -1 Falling back to public interface 2021-01-15T16:31:46.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:31:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:31:47.347+0000 7f842ea00080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:31:47.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:31:47.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:31:47.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:31:47.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:31:47.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:31:47.861 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:31:48.284 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/73936,v1:127.0.0.1:6819/73936] [v2:127.0.0.1:6820/73936,v1:127.0.0.1:6821/73936] exists,up baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:31:48.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:31:48.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:31:48.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:31:48.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2021-01-15T16:31:48.983 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:31:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2021-01-15T16:31:49.777 INFO:tasks.workunit.client.0.smithi145.stderr:noin is set 2021-01-15T16:31:49.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2021-01-15T16:31:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:nodown is set 2021-01-15T16:31:50.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2021-01-15T16:31:50.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: local pool_name=test 2021-01-15T16:31:50.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: shift 2021-01-15T16:31:50.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: local allow_overwrites=true 2021-01-15T16:31:50.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2275: create_ec_pool: shift 2021-01-15T16:31:50.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2021-01-15T16:31:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: create_ec_pool: create_pool test 1 1 erasure myprofile 2021-01-15T16:31:51.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T16:31:51.695 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:31:51.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:31:52.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: '[' true = true ']' 2021-01-15T16:31:52.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2021-01-15T16:31:53.472 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 allow_ec_overwrites to true 2021-01-15T16:31:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: wait_for_clean 2021-01-15T16:31:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:31:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:31:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:31:53.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:31:53.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:31:53.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:31:53.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:31:53.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:31:53.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:31:53.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:31:53.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:31:53.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:31:53.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:31:53.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:31:53.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:31:53.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:31:53.973 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:31:53.973 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:31:53.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:31:53.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:53.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:31:54.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:31:54.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:31:54.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:31:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:54.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:31:54.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T16:31:54.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T16:31:54.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T16:31:54.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:54.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:31:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:31:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:31:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2021-01-15T16:31:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:31:54.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:31:54.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:31:54.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:31:54.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:31:54.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:31:54.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:31:54.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:31:54.402 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:31:54.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:31:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T16:31:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:31:54.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T16:31:54.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:31:54.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:31:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T16:31:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:31:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T16:31:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T16:31:54.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:31:54.811 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T16:31:55.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2021-01-15T16:31:55.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:31:55.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:31:55.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:31:55.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:31:55.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:31:55.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:31:55.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:31:55.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:31:55.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:31:55.210 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:31:55.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509442 -lt 64424509444 2021-01-15T16:31:55.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:31:56.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:31:56.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:31:57.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T16:31:57.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:31:57.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:31:57.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:31:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:31:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:31:57.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:31:57.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:31:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:31:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:31:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:31:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:31:57.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:31:57.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:31:57.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:31:57.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:31:58.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:31:58.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:31:58.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:31:58.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: create_ec_pool: return 0 2021-01-15T16:31:58.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2021-01-15T16:31:58.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:31:58.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:31:58.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:31:58.845 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:31:58.846 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:31:58.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:31:58.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:58.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:31:58.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T16:31:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T16:31:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T16:31:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:58.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:31:59.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T16:31:59.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T16:31:59.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-01-15T16:31:59.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:31:59.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:31:59.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509447 2021-01-15T16:31:59.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509447 2021-01-15T16:31:59.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509447' 2021-01-15T16:31:59.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:31:59.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T16:31:59.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:31:59.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:31:59.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T16:31:59.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:31:59.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T16:31:59.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T16:31:59.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:31:59.274 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T16:31:59.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836490 2021-01-15T16:31:59.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:32:00.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:32:00.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:32:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2021-01-15T16:32:01.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:01.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T16:32:01.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:01.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:32:01.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T16:32:01.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:01.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T16:32:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T16:32:01.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:32:01.070 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T16:32:01.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T16:32:01.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:01.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509447 2021-01-15T16:32:01.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:01.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:32:01.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509447 2021-01-15T16:32:01.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:01.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509447 2021-01-15T16:32:01.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2021-01-15T16:32:01.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:32:01.471 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509447 2021-01-15T16:32:01.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509447 2021-01-15T16:32:01.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2021-01-15T16:32:01.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:32:01.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:32:01.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:32:01.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:32:01.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:32:01.866 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:32:01.867 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:32:01.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:32:01.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:32:01.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:32:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:32:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:32:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:32:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:32:01.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:32:02.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:32:02.350 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:32:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:32:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:32:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:02.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:32:02.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2021-01-15T16:32:02.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2021-01-15T16:32:02.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2021-01-15T16:32:02.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:02.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:32:02.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T16:32:02.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T16:32:02.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2021-01-15T16:32:02.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:02.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:32:02.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2021-01-15T16:32:02.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2021-01-15T16:32:02.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2021-01-15T16:32:02.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:02.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2021-01-15T16:32:02.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:02.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:32:02.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2021-01-15T16:32:02.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:02.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2021-01-15T16:32:02.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-01-15T16:32:02.782 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836492 2021-01-15T16:32:02.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:32:03.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836492 2021-01-15T16:32:03.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:32:04.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:32:04.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:32:04.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836492 2021-01-15T16:32:04.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:04.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T16:32:04.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:04.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:32:04.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T16:32:04.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:04.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T16:32:04.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T16:32:04.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:32:04.587 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T16:32:04.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T16:32:04.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:04.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2021-01-15T16:32:04.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:04.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:32:04.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-01-15T16:32:04.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:04.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2021-01-15T16:32:04.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-01-15T16:32:04.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:32:04.985 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2021-01-15T16:32:05.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509449 2021-01-15T16:32:05.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:32:05.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:32:05.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:32:05.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:32:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:32:05.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:32:05.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:32:05.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:32:05.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:32:05.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:32:05.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:32:06.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:32:06.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:32:06.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:32:06.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:32:06.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:32:06.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:32:06.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:32:06.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2021-01-15T16:32:06.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2021-01-15T16:32:07.230 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:07.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2021-01-15T16:32:07.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2021-01-15T16:32:07.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:32:07.241 INFO:tasks.workunit.client.0.smithi145.stdout:primary and soon to be divergent is 1 2021-01-15T16:32:07.610 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:32:07.611 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:31:52.658917+0000 0'0 28:12 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:32:07.611 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:32:07.611 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:32:07.612 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:07.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2021-01-15T16:32:07.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:07.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2021-01-15T16:32:07.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2021-01-15T16:32:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2021-01-15T16:32:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2021-01-15T16:32:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2021-01-15T16:32:07.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2021-01-15T16:32:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2021-01-15T16:32:07.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2021-01-15T16:32:07.626 INFO:tasks.workunit.client.0.smithi145.stdout:writing initial objects 2021-01-15T16:32:07.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:07.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2021-01-15T16:32:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2021-01-15T16:32:07.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:07.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2021-01-15T16:32:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2021-01-15T16:32:08.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:08.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2021-01-15T16:32:08.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:08.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2021-01-15T16:32:08.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:08.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2021-01-15T16:32:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2021-01-15T16:32:09.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:09.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2021-01-15T16:32:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:09.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2021-01-15T16:32:09.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:09.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2021-01-15T16:32:09.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:09.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2021-01-15T16:32:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:09.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2021-01-15T16:32:10.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:10.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2021-01-15T16:32:10.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:10.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2021-01-15T16:32:10.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:10.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2021-01-15T16:32:10.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:10.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2021-01-15T16:32:10.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:10.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2021-01-15T16:32:11.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:11.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2021-01-15T16:32:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2021-01-15T16:32:11.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:11.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2021-01-15T16:32:11.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:11.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2021-01-15T16:32:11.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:11.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2021-01-15T16:32:12.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:12.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2021-01-15T16:32:12.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:12.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2021-01-15T16:32:12.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:12.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2021-01-15T16:32:12.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:12.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2021-01-15T16:32:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2021-01-15T16:32:13.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:13.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2021-01-15T16:32:13.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:13.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2021-01-15T16:32:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2021-01-15T16:32:13.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:13.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2021-01-15T16:32:13.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:13.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2021-01-15T16:32:14.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:14.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2021-01-15T16:32:14.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:14.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2021-01-15T16:32:14.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:14.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2021-01-15T16:32:14.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:14.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2021-01-15T16:32:14.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:14.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2021-01-15T16:32:15.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:15.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2021-01-15T16:32:15.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:15.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2021-01-15T16:32:15.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:15.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2021-01-15T16:32:15.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:15.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2021-01-15T16:32:15.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:15.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2021-01-15T16:32:16.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:16.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2021-01-15T16:32:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2021-01-15T16:32:16.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:16.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2021-01-15T16:32:16.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:16.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2021-01-15T16:32:16.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:16.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2021-01-15T16:32:17.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:17.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2021-01-15T16:32:17.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:17.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2021-01-15T16:32:17.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:17.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2021-01-15T16:32:17.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:17.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2021-01-15T16:32:17.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:17.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2021-01-15T16:32:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2021-01-15T16:32:18.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:18.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2021-01-15T16:32:18.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:18.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2021-01-15T16:32:18.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:18.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2021-01-15T16:32:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2021-01-15T16:32:19.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:19.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2021-01-15T16:32:19.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:19.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2021-01-15T16:32:19.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:19.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2021-01-15T16:32:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2021-01-15T16:32:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2021-01-15T16:32:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2021-01-15T16:32:20.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:20.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2021-01-15T16:32:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2021-01-15T16:32:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2021-01-15T16:32:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2021-01-15T16:32:21.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:21.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2021-01-15T16:32:21.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:21.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2021-01-15T16:32:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:21.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2021-01-15T16:32:21.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:21.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2021-01-15T16:32:21.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:21.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2021-01-15T16:32:22.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:22.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2021-01-15T16:32:22.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:22.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2021-01-15T16:32:22.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:22.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2021-01-15T16:32:22.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:22.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2021-01-15T16:32:23.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2021-01-15T16:32:23.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2021-01-15T16:32:23.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2021-01-15T16:32:23.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2021-01-15T16:32:23.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2021-01-15T16:32:23.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:23.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2021-01-15T16:32:24.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:24.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2021-01-15T16:32:24.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:24.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2021-01-15T16:32:24.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:24.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2021-01-15T16:32:24.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:24.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2021-01-15T16:32:25.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2021-01-15T16:32:25.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2021-01-15T16:32:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2021-01-15T16:32:25.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2021-01-15T16:32:25.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2021-01-15T16:32:25.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:25.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2021-01-15T16:32:26.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:26.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2021-01-15T16:32:26.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:26.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2021-01-15T16:32:26.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:26.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2021-01-15T16:32:26.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:26.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2021-01-15T16:32:26.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:26.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2021-01-15T16:32:27.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:27.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-01-15T16:32:27.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2021-01-15T16:32:27.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-01-15T16:32:27.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:32:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2021-01-15T16:32:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:32:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:32:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:32:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:32:27.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:32:27.499 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:32:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:32:27.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:32:27.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:32:27.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:32:27.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:32:27.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:32:27.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:32:27.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:32:27.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:32:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:32:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:32:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:32:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:32:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:27.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:32:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2021-01-15T16:32:28.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2021-01-15T16:32:28.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2021-01-15T16:32:28.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:28.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:32:28.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2021-01-15T16:32:28.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2021-01-15T16:32:28.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2021-01-15T16:32:28.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:28.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:32:28.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509456 2021-01-15T16:32:28.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509456 2021-01-15T16:32:28.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509456' 2021-01-15T16:32:28.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:28.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2021-01-15T16:32:28.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:28.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:32:28.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-01-15T16:32:28.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:28.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2021-01-15T16:32:28.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-01-15T16:32:28.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:32:28.393 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836500 2021-01-15T16:32:28.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2021-01-15T16:32:28.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:28.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2021-01-15T16:32:28.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:28.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:32:28.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2021-01-15T16:32:28.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:28.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2021-01-15T16:32:28.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-01-15T16:32:28.806 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672978 2021-01-15T16:32:28.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:32:29.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2021-01-15T16:32:29.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:32:29.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509456 2021-01-15T16:32:29.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:32:29.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:32:29.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-01-15T16:32:29.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:32:29.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509456 2021-01-15T16:32:29.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-01-15T16:32:29.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:32:29.208 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509456 2021-01-15T16:32:29.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509454 -lt 64424509456 2021-01-15T16:32:29.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:32:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:32:30.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:32:31.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509456 2021-01-15T16:32:31.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:32:31.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:32:31.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:32:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:32:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:32:31.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:32:31.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:32:31.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:32:31.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:32:31.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:32:31.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:32:31.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:32:31.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:32:31.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:32:31.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:32:32.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2021-01-15T16:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T16:32:32.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=existing_1 2021-01-15T16:32:32.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test existing_1 2021-01-15T16:32:32.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T16:32:32.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2021-01-15T16:32:32.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2021-01-15T16:32:32.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:32:32.866 INFO:tasks.workunit.client.0.smithi145.stdout:blackholing osds 0 2 2021-01-15T16:32:33.240 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:32:33.240 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2021-01-15T16:31:52.658917+0000 29'100 29:112 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:32:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:32:33.240 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:32:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:33.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2021-01-15T16:32:33.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:32:33.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:32:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:32:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:32:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:32:33.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:32:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-01-15T16:32:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok config set objectstore_blackhole 1 2021-01-15T16:32:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:32:33.350 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:32:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:32:33.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:33.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2021-01-15T16:32:33.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:32:33.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:32:33.359 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:32:33.359 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:32:33.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:32:33.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:32:33.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2021-01-15T16:32:33.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok config set objectstore_blackhole 1 2021-01-15T16:32:33.457 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:32:33.457 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2021-01-15T16:32:33.458 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:32:33.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2021-01-15T16:32:33.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2021-01-15T16:32:33.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2021-01-15T16:32:33.466 INFO:tasks.workunit.client.0.smithi145.stdout:writing divergent object 2021-01-15T16:32:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2021-01-15T16:32:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2021-01-15T16:32:34.718 INFO:tasks.workunit.client.0.smithi145.stdout:created pool test snap snap1 2021-01-15T16:32:34.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2021-01-15T16:32:34.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2021-01-15T16:32:34.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2021-01-15T16:32:44.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2021-01-15T16:32:44.724 INFO:tasks.workunit.client.0.smithi145.stderr:rados: no process found 2021-01-15T16:32:44.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2021-01-15T16:32:44.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:32:44.725 INFO:tasks.workunit.client.0.smithi145.stdout:killing all the osds 2021-01-15T16:32:45.107 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:32:45.107 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2021-01-15T16:31:52.658917+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:32:45.107 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:32:45.108 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:32:45.109 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:45.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2021-01-15T16:32:45.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:32:45.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:32:45.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:32:45.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:32:45.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:32:45.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:32:45.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:45.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2021-01-15T16:32:45.765 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T16:32:45.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:45.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2021-01-15T16:32:46.299 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T16:32:46.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2021-01-15T16:32:46.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2021-01-15T16:32:46.850 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T16:32:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2021-01-15T16:32:47.374 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T16:32:47.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:47.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2021-01-15T16:32:47.908 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T16:32:47.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2021-01-15T16:32:47.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:32:47.918 INFO:tasks.workunit.client.0.smithi145.stdout:bringing up non_divergent 0 2 2021-01-15T16:32:48.294 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:32:48.294 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2021-01-15T16:31:52.658917+0000 30'104 30:116 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:32:48.295 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:32:48.295 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:32:48.296 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:48.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:48.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-01-15T16:32:48.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:32:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:32:48.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:32:48.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:32:48.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:32:48.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:32:48.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:32:48.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:32:48.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:32:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:32:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:32:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:32:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:32:48.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:32:48.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:32:48.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:32:48.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:32:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:32:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:32:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:32:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:32:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:32:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:32:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:32:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:32:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:32:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:32:48.318 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:32:48.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:32:48.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2021-01-15T16:32:48.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:32:48.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:32:48.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:32:48.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:32:48.336 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:48.335+0000 7f4f4ce76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:48.338 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:48.337+0000 7f4f4ce76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:48.339 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:48.339+0000 7f4f4ce76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:48.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:32:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:32:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:32:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:32:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:32:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:32:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:32:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:32:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:32:48.713 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:32:48.858 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:48.858+0000 7f4f4ce76080 -1 Falling back to public interface 2021-01-15T16:32:49.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:32:49.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:49.930+0000 7f4f4ce76080 -1 osd.0 29 log_to_monitors {default=true} 2021-01-15T16:32:50.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:32:50.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:32:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:32:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:32:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:32:50.113 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:32:50.542 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/78311,v1:127.0.0.1:6803/78311] [v2:127.0.0.1:6804/78311,v1:127.0.0.1:6805/78311] exists,up 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:32:50.543 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:32:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:32:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:32:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:32:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-01-15T16:32:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:32:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:32:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:32:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:32:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:32:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:32:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:32:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:32:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:32:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:32:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:32:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:32:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:32:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:32:50.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:32:50.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:32:50.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:32:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:32:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:32:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:32:50.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:32:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:32:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:32:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:32:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:32:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:32:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:32:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:32:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:32:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:32:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:32:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:32:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:32:50.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2021-01-15T16:32:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:32:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:32:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:32:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:32:50.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:50.571+0000 7fa0a58d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:50.572+0000 7fa0a58d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:50.573+0000 7fa0a58d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:32:50.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:32:50.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:32:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:32:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:32:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:32:50.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:32:50.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:32:50.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:32:50.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:32:50.948 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:32:51.093 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:51.093+0000 7fa0a58d5080 -1 Falling back to public interface 2021-01-15T16:32:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:32:52.244 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:32:52.244+0000 7fa0a58d5080 -1 osd.2 29 log_to_monitors {default=true} 2021-01-15T16:32:52.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:32:52.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:32:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:32:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:32:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:32:52.350 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:32:52.794 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/78800,v1:127.0.0.1:6811/78800] [v2:127.0.0.1:6812/78800,v1:127.0.0.1:6813/78800] exists,up baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:32:52.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:32:52.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:32:52.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:32:52.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:52.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2021-01-15T16:32:53.330 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T16:32:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:32:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2021-01-15T16:32:53.923 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already in. 2021-01-15T16:32:53.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2021-01-15T16:32:58.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2021-01-15T16:32:58.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2021-01-15T16:32:58.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2021-01-15T16:32:58.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:32:58.936 INFO:tasks.workunit.client.0.smithi145.stdout:writing non-divergent object existing_4 2021-01-15T16:32:59.318 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:32:59.318 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2021-01-15T16:32:54.264732+0000 29'100 40:121 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:32:59.318 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:32:59.318 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:32:59.320 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:32:59.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2021-01-15T16:32:59.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:32:59.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2021-01-15T16:32:59.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:32:59.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:32:59.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:32:59.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:32:59.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:32:59.453 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:32:59.454 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:32:59.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:32:59.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:32:59.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:32:59.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:32:59.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:32:59.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:32:59.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:32:59.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:32:59.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:32:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:32:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:32:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:32:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:32:59.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:33:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=154618822661 2021-01-15T16:33:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 154618822661 2021-01-15T16:33:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661' 2021-01-15T16:33:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:33:00.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:33:00.153 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:33:00.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:33:00.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:33:00.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:33:00.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:33:00.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:33:00.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789956 2021-01-15T16:33:00.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789956 2021-01-15T16:33:00.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-154618822661 2-158913789956' 2021-01-15T16:33:00.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:33:00.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-154618822661 2021-01-15T16:33:00.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:33:00.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:33:00.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-154618822661 2021-01-15T16:33:00.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:33:00.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=154618822661 2021-01-15T16:33:00.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2021-01-15T16:33:00.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:33:00.301 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 154618822661 2021-01-15T16:33:00.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 154618822661 -lt 154618822661 2021-01-15T16:33:00.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:33:00.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-158913789956 2021-01-15T16:33:00.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:33:00.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:33:00.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-158913789956 2021-01-15T16:33:00.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:33:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789956 2021-01-15T16:33:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 158913789956' 2021-01-15T16:33:00.701 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 158913789956 2021-01-15T16:33:00.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:33:01.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789956 -lt 158913789956 2021-01-15T16:33:01.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:33:01.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:01.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:01.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:33:01.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:01.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:01.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:01.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:01.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:01.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:01.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:02.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:02.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:02.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:02.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:02.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:02.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:33:02.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:33:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:33:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:33:02.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:02.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:02.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:02.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:02.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:02.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:02.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:02.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:03.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:03.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:03.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:03.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:03.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:03.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:03.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:03.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:03.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:03.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:04.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:04.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:04.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:33:04.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:04.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:33:04.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:04.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:04.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:04.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:04.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:04.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:04.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:04.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:04.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:04.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:04.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:04.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:05.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:05.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:05.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:05.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:05.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:05.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:05.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:05.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:05.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:33:05.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:05.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:33:06.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:06.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:06.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:06.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:06.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:06.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:06.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:06.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:06.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:06.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:06.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:06.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:06.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:06.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:06.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:06.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:06.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:06.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:06.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:06.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:06.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:07.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:07.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:07.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:33:07.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:07.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:33:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:08.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:08.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:09.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:09.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:09.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:09.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:09.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:09.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:09.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:09.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:33:09.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:09.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:33:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:11.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:11.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:11.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:11.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:11.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:11.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:11.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:11.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:11.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:11.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:12.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:12.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:12.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:12.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:12.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:12.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:12.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:12.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:12.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:12.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:33:12.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:12.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:33:15.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:15.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:15.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:15.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:15.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:15.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:15.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:15.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:16.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:16.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:16.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:16.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:16.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:16.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:16.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:16.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:33:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:33:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:23.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:23.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:23.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:23.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:23.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:23.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:24.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:24.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:24.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:24.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:24.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:24.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:24.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:24.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:24.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:25.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:25.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:25.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:33:25.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:25.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:33:32.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:32.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:32.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:32.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:32.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:32.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:32.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:32.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:32.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:32.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:32.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:33.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:33.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:33.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:33.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:33.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:33.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:33.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:33:33.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:33:34.505 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:33:34.506 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "c97721ee-650d-4eba-a0ec-f245b5795511", 2021-01-15T16:33:34.506 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:33:34.506 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:33:34.506 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:33:34.504817+0000", 2021-01-15T16:33:34.507 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:33:34.507 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:33:34.507 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:33:34.507 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:33:34.507 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:33:34.508 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:33:34.508 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:33:34.508 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:33:34.508 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:33:34.508 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.509 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:33:34.509 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:33:34.509 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.509 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:33:34.509 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:33:34.510 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:33:34.510 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:33:34.510 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:33:34.510 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.510 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:33:34.511 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.511 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:33:34.511 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.511 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.511 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:33:34.512 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.512 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:33:34.512 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:33:34.512 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:33:34.512 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-01-15T16:33:34.513 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:33:34.513 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.513 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:33:34.513 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.514 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2021-01-15T16:33:34.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.514 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.514 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:33:34.514 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.515 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:33:34.515 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:33:34.515 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:33:34.515 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:33:34.516 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:33:34.516 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.516 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:33:34.516 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.516 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:33:34.517 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.517 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.517 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:33:34.517 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.517 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.518 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:33:34.518 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.518 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.518 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:33:34.518 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.519 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:33:34.519 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:33:34.519 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:33:34.519 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:33:34.519 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:33:34.520 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:31:22.975023Z", 2021-01-15T16:33:34.520 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:31:22.975023Z", 2021-01-15T16:33:34.520 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:33:34.520 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:33:34.520 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:33:34.521 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:33:34.521 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:33:34.521 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:33:34.521 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:33:34.521 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:33:34.522 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:33:34.522 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:33:34.522 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:33:34.522 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:33:34.522 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:33:34.523 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:33:34.523 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:33:34.523 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.523 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:33:34.524 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.524 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:33:34.524 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.525 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:33:34.525 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:33:34.525 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:33:34.525 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.525 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.526 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.526 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:33:34.526 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:33:34.526 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.526 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.527 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.527 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:33:34.527 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:33:34.527 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:33:34.527 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:33:34.528 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:33:34.529 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.529 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:33:34.529 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 40, 2021-01-15T16:33:34.529 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:33:34.529 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:31:23.733503+0000", 2021-01-15T16:33:34.530 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:32:53.937151+0000", 2021-01-15T16:33:34.530 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:32:52.325565+0000", 2021-01-15T16:33:34.530 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:32:53.674032+0000", 2021-01-15T16:33:34.530 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:33:34.531 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:33:34.531 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:33:34.531 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:33:34.531 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:33:34.531 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:33:34.532 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:33:34.532 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:33:34.532 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:33:34.532 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:33:34.532 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.533 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 8, 2021-01-15T16:33:34.533 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:33:34.533 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:33:34.533 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:33:34.533 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:33:34.534 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:33:34.534 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:33:34.534 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:33:34.534 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:33:34.534 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:33:34.535 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:33:34.535 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.535 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:33:34.535 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:33:34.535 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:31:51.449694+0000", 2021-01-15T16:33:34.536 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 16389, 2021-01-15T16:33:34.536 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-01-15T16:33:34.536 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2021-01-15T16:33:34.536 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:33:34.536 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:33:34.537 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 1, 2021-01-15T16:33:34.537 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:33:34.537 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:33:34.537 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:33:34.537 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:33:34.538 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:33:34.538 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T16:33:34.538 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T16:33:34.538 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:33:34.538 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:33:34.539 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:33:34.540 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:33:34.540 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:33:34.540 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:33:34.540 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.540 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "30", 2021-01-15T16:33:34.541 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:33:34.541 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:33:34.541 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T16:33:34.541 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:33:34.542 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "pool", 2021-01-15T16:33:34.542 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 1, 2021-01-15T16:33:34.542 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 30, 2021-01-15T16:33:34.542 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [ 2021-01-15T16:33:34.543 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.543 INFO:tasks.workunit.client.0.smithi145.stdout: "snapid": 1, 2021-01-15T16:33:34.543 INFO:tasks.workunit.client.0.smithi145.stdout: "stamp": "2021-01-15T16:32:34.499003+0000", 2021-01-15T16:33:34.543 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "snap1" 2021-01-15T16:33:34.543 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.544 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.544 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:33:34.544 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:33:34.544 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:33:34.544 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:33:34.545 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:33:34.546 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:33:34.546 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:33:34.546 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:33:34.546 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "myprofile", 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.547 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:33:34.548 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:33:34.548 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:33:34.548 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:33:34.548 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:33:34.548 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:33:34.549 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:33:34.549 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:33:34.549 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 8192, 2021-01-15T16:33:34.549 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:33:34.549 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:33:34.550 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:33:34.550 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:33:34.550 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.550 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.550 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:33:34.551 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.551 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:33:34.551 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "6412922c-b560-401d-81b8-a8bbe5ef1e10", 2021-01-15T16:33:34.551 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:33:34.551 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:33:34.552 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:33:34.552 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:33:34.552 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 5, 2021-01-15T16:33:34.552 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:33:34.552 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 36, 2021-01-15T16:33:34.553 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 39, 2021-01-15T16:33:34.553 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 31, 2021-01-15T16:33:34.553 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:33:34.553 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:33:34.553 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.554 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.554 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.554 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:33:34.554 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.554 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.555 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.555 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.555 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:33:34.555 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.555 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.556 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.556 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.556 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:33:34.556 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.557 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.557 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:33:34.557 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.557 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.557 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.558 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.558 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:33:34.558 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.558 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.558 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.559 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:33:34.559 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.559 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.559 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.560 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:33:34.560 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.560 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.560 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.561 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:33:34.561 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.561 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.561 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.561 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.562 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:33:34.562 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.562 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.562 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.562 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:33:34.563 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.563 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.563 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.563 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:33:34.564 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78311 2021-01-15T16:33:34.564 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.564 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.564 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/78311", 2021-01-15T16:33:34.565 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/78311", 2021-01-15T16:33:34.565 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/78311", 2021-01-15T16:33:34.565 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/78311", 2021-01-15T16:33:34.565 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:33:34.565 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:33:34.566 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:33:34.566 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.566 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.566 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "833bc698-267b-4dab-a554-ed9c97578364", 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:33:34.567 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:33:34.568 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:33:34.568 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:33:34.568 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 25, 2021-01-15T16:33:34.568 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2021-01-15T16:33:34.568 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:33:34.569 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:33:34.569 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.569 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.569 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.569 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:33:34.570 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.570 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.570 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.570 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:33:34.571 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.571 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.571 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.571 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.571 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:33:34.572 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.572 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.572 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.572 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:33:34.572 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.573 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.573 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.573 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:33:34.573 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.574 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.574 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.574 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.574 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:33:34.574 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.575 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.575 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.575 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:33:34.575 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.576 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.576 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.576 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:33:34.576 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.576 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.577 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.577 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:33:34.577 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.577 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.578 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.578 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:33:34.578 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.578 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.579 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.579 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:33:34.579 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:33:34.579 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.579 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/72672", 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/72672", 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/72672", 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/72672", 2021-01-15T16:33:34.580 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:33:34.581 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:33:34.581 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.581 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.581 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.581 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:33:34.582 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "baffd605-38c3-450d-87ae-a21bfb354b1f", 2021-01-15T16:33:34.582 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:33:34.582 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:33:34.582 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:33:34.582 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:33:34.583 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 15, 2021-01-15T16:33:34.583 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 29, 2021-01-15T16:33:34.583 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 37, 2021-01-15T16:33:34.583 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:33:34.583 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 33, 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.584 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:33:34.585 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.585 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.585 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.585 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:33:34.586 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.586 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.586 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.586 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:33:34.587 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.587 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.587 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:33:34.587 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.588 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.588 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.588 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:33:34.588 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.589 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.589 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.589 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.589 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:33:34.589 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.590 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.590 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:33:34.590 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.590 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.591 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.591 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.591 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:33:34.591 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.591 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.592 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.592 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.592 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:33:34.592 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:33:34.592 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.593 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:33:34.594 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:33:34.594 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 78800 2021-01-15T16:33:34.594 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.594 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.595 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/78800", 2021-01-15T16:33:34.595 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/78800", 2021-01-15T16:33:34.595 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/78800", 2021-01-15T16:33:34.595 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/78800", 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.596 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.597 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:33:34.597 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.597 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:33:34.597 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:32:45.559638+0000", 2021-01-15T16:33:34.597 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:33:34.598 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:33:34.598 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:33:34.598 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:33:34.598 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:32.371078+0000", 2021-01-15T16:33:34.598 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:33:34.599 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.599 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:33:34.599 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:32:46.110699+0000", 2021-01-15T16:33:34.600 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:33:34.600 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:33:34.600 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:33:34.600 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:33:34.600 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:40.332739+0000", 2021-01-15T16:33:34.601 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:33:34.601 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.601 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:33:34.601 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:32:46.613531+0000", 2021-01-15T16:33:34.602 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:33:34.602 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:33:34.602 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:33:34.602 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:33:34.602 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:48.347963+0000", 2021-01-15T16:33:34.603 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:33:34.603 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.603 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.603 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:33:34.603 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:33:34.604 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:33:34.604 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:33:34.604 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:33:34.604 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:33:34.604 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.605 INFO:tasks.workunit.client.0.smithi145.stdout: "myprofile": { 2021-01-15T16:33:34.606 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-device-class": "", 2021-01-15T16:33:34.606 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-failure-domain": "osd", 2021-01-15T16:33:34.606 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-root": "default", 2021-01-15T16:33:34.606 INFO:tasks.workunit.client.0.smithi145.stdout: "jerasure-per-chunk-alignment": "false", 2021-01-15T16:33:34.607 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:33:34.607 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "1", 2021-01-15T16:33:34.607 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:33:34.607 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van", 2021-01-15T16:33:34.608 INFO:tasks.workunit.client.0.smithi145.stdout: "w": "8" 2021-01-15T16:33:34.608 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.608 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.608 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:33:34.609 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:33:34.609 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:33:34.609 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:33:34.609 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:33:34.609 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:33:34.610 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:33:34.610 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:33:34.610 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:33:34.610 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:33:34.610 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:33:34.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.611 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.611 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:33:34.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.611 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:33:34.612 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:33:34.612 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/78311,v1:127.0.0.1:6805/78311]", 2021-01-15T16:33:34.612 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:33:34.612 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:33:34.612 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:33:34.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:33:34.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:33:34.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:33:34.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:33:34.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:33:34.614 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:33:34.614 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:33:34.614 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:33:34.614 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:33:34.614 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:33:34.615 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:33:34.616 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:33:34.616 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:33:34.616 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:33:34.616 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:33:34.616 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:33:34.617 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:33:34.617 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/78311,v1:127.0.0.1:6803/78311]", 2021-01-15T16:33:34.617 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:33:34.617 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/78311,v1:127.0.0.1:6809/78311]", 2021-01-15T16:33:34.617 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/78311,v1:127.0.0.1:6807/78311]", 2021-01-15T16:33:34.618 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:33:34.618 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:33:34.618 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:33:34.618 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:33:34.618 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:33:34.619 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:33:34.619 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:33:34.619 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:33:34.619 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:33:34.619 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:33:34.620 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:33:34.620 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:33:34.620 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:33:34.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.621 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:33:34.621 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:33:34.621 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/72672,v1:127.0.0.1:6813/72672]", 2021-01-15T16:33:34.621 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:33:34.621 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:33:34.622 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:33:34.622 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:33:34.622 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:33:34.622 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:33:34.622 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:33:34.623 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:33:34.624 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:33:34.624 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:33:34.624 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:33:34.624 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:33:34.625 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:33:34.625 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:33:34.625 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:33:34.625 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:33:34.625 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:33:34.626 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:33:34.626 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:33:34.626 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:33:34.626 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/72672,v1:127.0.0.1:6811/72672]", 2021-01-15T16:33:34.626 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:33:34.627 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/72672,v1:127.0.0.1:6817/72672]", 2021-01-15T16:33:34.627 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/72672,v1:127.0.0.1:6815/72672]", 2021-01-15T16:33:34.627 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:33:34.627 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:33:34.628 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:33:34.628 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:33:34.628 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:33:34.628 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:33:34.628 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:33:34.629 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:33:34.629 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:33:34.629 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:33:34.629 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:33:34.629 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:33:34.630 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:33:34.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.630 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:33:34.630 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:33:34.631 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/78800,v1:127.0.0.1:6813/78800]", 2021-01-15T16:33:34.631 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:33:34.631 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:33:34.631 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:33:34.631 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:33:34.632 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:33:34.633 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:33:34.633 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:33:34.633 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:33:34.633 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:33:34.633 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:33:34.634 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:33:34.634 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:33:34.634 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:33:34.634 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:33:34.635 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:33:34.636 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/78800,v1:127.0.0.1:6811/78800]", 2021-01-15T16:33:34.636 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:33:34.636 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/78800,v1:127.0.0.1:6817/78800]", 2021-01-15T16:33:34.636 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/78800,v1:127.0.0.1:6815/78800]", 2021-01-15T16:33:34.636 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:33:34.637 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:33:34.637 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:33:34.637 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:33:34.637 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:33:34.637 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:33:34.638 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:33:34.638 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:33:34.638 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:33:34.638 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:33:34.638 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:33:34.639 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:33:34.639 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:33:34.639 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.639 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.639 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:33:34.640 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:33:34.640 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:33:34.640 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:33:34.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.640 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:33:34.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.641 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:33:34.641 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 36 2021-01-15T16:33:34.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.642 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:33:34.642 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:33:34.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.642 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 37 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:33:34.643 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 40, 2021-01-15T16:33:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:33:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:33:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:33:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:33:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:33:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:33:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:33:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:33:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:33:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:33:34.647 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:33:34.647 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.647 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.647 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:33:34.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:33:34.648 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:33:34.649 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.649 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.649 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:33:34.649 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:33:34.649 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.650 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:33:34.650 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:33:34.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.651 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:33:34.651 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:33:34.651 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.652 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:33:34.652 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:33:34.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.652 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:33:34.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.654 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:33:34.654 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:33:34.654 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.655 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:33:34.656 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:33:34.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.656 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:33:34.656 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:33:34.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.657 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.657 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:33:34.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.657 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:33:34.658 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:33:34.658 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:33:34.658 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:33:34.658 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.658 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:33:34.659 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:33:34.659 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:33:34.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.659 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:33:34.659 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.660 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:33:34.661 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:33:34.661 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:33:34.661 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:33:34.661 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.661 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:33:34.662 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:33:34.662 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:33:34.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.662 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:33:34.662 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.663 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:33:34.663 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.663 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.663 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.664 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.664 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:33:34.664 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:33:34.664 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:33:34.664 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:33:34.665 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.665 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:33:34.665 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:33:34.665 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:33:34.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:33:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:33:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.667 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:33:34.667 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.667 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:33:34.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.668 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:33:34.668 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.668 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:33:34.668 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.668 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:33:34.669 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:33:34.670 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:33:34.670 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:33:34.670 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:33:34.670 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:33:34.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.671 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:33:34.671 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.671 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:33:34.671 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.671 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.672 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:33:34.672 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.672 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:33:34.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.673 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:33:34.673 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:33:34.673 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:33:34.673 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.673 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.674 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.674 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.674 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:33:34.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.675 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:33:34.675 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:33:34.675 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:33:34.675 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:33:34.675 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:33:34.676 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:33:34.676 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:33:34.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.676 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:33:34.676 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:33:34.677 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:33:34.678 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.678 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:33:34.678 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.678 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.679 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 1, 2021-01-15T16:33:34.679 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "test", 2021-01-15T16:33:34.679 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 1, 2021-01-15T16:33:34.680 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2021-01-15T16:33:34.680 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 3, 2021-01-15T16:33:34.680 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 3, 2021-01-15T16:33:34.680 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:33:34.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.681 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_chooseleaf_tries", 2021-01-15T16:33:34.681 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 5 2021-01-15T16:33:34.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.681 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_choose_tries", 2021-01-15T16:33:34.682 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 100 2021-01-15T16:33:34.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.682 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.682 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:33:34.682 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:33:34.683 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:33:34.683 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.683 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_indep", 2021-01-15T16:33:34.683 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.684 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:33:34.685 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.685 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.685 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:33:34.685 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:33:34.685 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:33:34.686 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:33:34.686 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:33:34.686 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:33:34.686 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:33:34.686 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:33:34.687 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:33:34.687 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:33:34.687 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:33:34.687 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 1, 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:33:34.688 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:33:34.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:33:34.689 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:33:34.689 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:33:34.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.689 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:33:34.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.690 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:33:34.690 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:33:34.690 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:33:34.690 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:33:34.691 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:33:34.691 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:33:34.691 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:33:34.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:33:34.691 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:33:34.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:33:34.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:33:34.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:33:34.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:33:34.692 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:33:34.693 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:33:34.693 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:33:34.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.693 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:33:34.694 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:33:34.694 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:33:34.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.694 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:33:34.694 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:33:34.695 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:33:34.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:33:34.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:33:34.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2021-01-15T16:33:34.696 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:33:34.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:33:34.697 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:33:34.697 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:33:34.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2021-01-15T16:33:34.697 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2021-01-15T16:33:34.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2021-01-15T16:33:34.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2021-01-15T16:33:34.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:33:34.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:33:34.698 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2021-01-15T16:33:34.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:33:34.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:33:34.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2021-01-15T16:33:34.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:33:34.699 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:33:34.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:33:34.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:33:34.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:33:34.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:33:34.700 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:33:34.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:33:34.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:33:34.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:33:34.701 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:33:34.702 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:33:34.702 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:33:34.702 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:33:34.702 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:33:34.702 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:33:34.703 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:33:34.703 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:33:34.703 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:33:34.703 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:33:34.703 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:33:34.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:33:34.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:33:34.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:33:34.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:33:34.704 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:33:34.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:33:34.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:33:34.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:33:34.705 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:33:34.705 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.706 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:33:34.706 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:33:34.706 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:33:34.706 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:33:34.707 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:33:34.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:33:34.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:33:34.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:33:34.707 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:33:34.708 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:33:34.708 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:33:34.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.708 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2021-01-15T16:33:34.709 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2021-01-15T16:33:34.709 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2021-01-15T16:33:34.709 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2021-01-15T16:33:34.709 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:33:34.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.710 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:33:34.710 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:33:34.710 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:33:34.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:33:34.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:33:34.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:33:34.711 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:33:34.712 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:33:34.712 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18404, 2021-01-15T16:33:34.712 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2116, 2021-01-15T16:33:34.712 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:33:34.713 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16254, 2021-01-15T16:33:34.713 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554396, 2021-01-15T16:33:34.713 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:33:34.713 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:33:34.713 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103701504, 2021-01-15T16:33:34.714 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:33:34.714 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2166784, 2021-01-15T16:33:34.714 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1523781, 2021-01-15T16:33:34.714 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:33:34.715 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:33:34.715 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:33:34.715 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:33:34.715 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16644200 2021-01-15T16:33:34.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.716 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:33:34.716 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:33:34.716 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:33:34.716 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:33:34.716 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:33:34.717 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:33:34.718 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:33:34.718 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:33:34.718 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.718 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:33:34.719 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:33:34.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:33:34.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:33:34.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:33:34.720 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:33:34.720 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:33:34.721 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 18404, 2021-01-15T16:33:34.721 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2116, 2021-01-15T16:33:34.721 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 1, 2021-01-15T16:33:34.721 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16254, 2021-01-15T16:33:34.721 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314554396, 2021-01-15T16:33:34.722 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:33:34.722 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:33:34.722 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103701504, 2021-01-15T16:33:34.722 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:33:34.722 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2166784, 2021-01-15T16:33:34.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1523781, 2021-01-15T16:33:34.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:33:34.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:33:34.723 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:33:34.723 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 1944, 2021-01-15T16:33:34.724 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 16644200 2021-01-15T16:33:34.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.724 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:33:34.724 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:33:34.724 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:33:34.725 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:33:34.725 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:33:34.725 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:33:34.725 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:33:34.725 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.726 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:33:34.726 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:33:34.726 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:33:34.726 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:33:34.726 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:33:34.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.727 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:33:34.727 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:33:34.727 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.728 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:33:34.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.728 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:33:34.728 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:33:34.728 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:33:34.729 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:33:34.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2021-01-15T16:33:34.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:33:34.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:33:34.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2021-01-15T16:33:34.730 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:33:34.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:33:34.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:33:34.731 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:33:34.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:33:34.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:33:34.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:33:34.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:33:34.732 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:33:34.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:33:34.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:33:34.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:33:34.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:33:34.733 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:33:34.734 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:33:34.735 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:33:34.735 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:33:34.735 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:33:34.735 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:33:34.735 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:33:34.736 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:33:34.736 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:33:34.736 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:33:34.736 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:33:34.736 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:33:34.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.737 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:33:34.737 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:33:34.737 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:33:34.737 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:33:34.738 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1241088, 2021-01-15T16:33:34.738 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1241088, 2021-01-15T16:33:34.738 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:33:34.738 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:33:34.738 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:33:34.739 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:33:34.739 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:33:34.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.739 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2021-01-15T16:33:34.739 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2021-01-15T16:33:34.740 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2021-01-15T16:33:34.740 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2021-01-15T16:33:34.740 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:33:34.740 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.740 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 154618822667 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.742 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:33:34.742 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672981 2021-01-15T16:33:34.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.742 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:33:34.743 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 158913789963 2021-01-15T16:33:34.743 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.743 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.743 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.744 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:33:34.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.745 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:33:34.745 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:33:34.745 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:33:34.745 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:33:34.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.746 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2147483647, 2021-01-15T16:33:34.746 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:33:34.746 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:33:34.747 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:33:34.747 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.747 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:33:34.747 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:33:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:33:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:33:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:33:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T16:33:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:33:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:33:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:33:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:33:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:33:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:33:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:33:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:33:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 5, 2021-01-15T16:33:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:32:50.370955+0000", 2021-01-15T16:33:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "services": { 2021-01-15T16:33:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": { 2021-01-15T16:33:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "daemons": { 2021-01-15T16:33:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": "", 2021-01-15T16:33:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "0": { 2021-01-15T16:33:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:33:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:33:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:33:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:33:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:33:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:33:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 387, 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:33:34.755 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:33:34.756 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:33:34.757 INFO:tasks.workunit.client.0.smithi145.stderr:report 229136379 2021-01-15T16:33:34.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:33:34.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:33:34.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2021-01-15T16:33:34.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:33:34.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:33:34.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:33:34.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:33:34.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:33:34.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=0 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:33:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2021-01-15T16:33:34.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2021-01-15T16:33:35.347 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2021-01-15T16:33:35.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2021-01-15T16:33:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2021-01-15T16:33:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:33:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:33:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:33:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:33:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/0 2021-01-15T16:33:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:33:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:33:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:33:35.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:33:35.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2021-01-15T16:33:35.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2021-01-15T16:33:35.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:33:35.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:33:35.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:33:35.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:33:35.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:33:35.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:33:35.628 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:33:35.629 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:33:35.630 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:33:35.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:33:35.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:33:35.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:33:35.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:33:35.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:33:35.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:33:35.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:33:35.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:33:35.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:33:35.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:33:35.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/0 2021-01-15T16:33:35.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:33:35.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:33:35.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/0/whoami 2021-01-15T16:33:35.636 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:33:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:33:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:33:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:33:35.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:33:35.651 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:35.651+0000 7fec9bbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:35.653 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:35.653+0000 7fec9bbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:35.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:35.654+0000 7fec9bbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:33:36.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:33:36.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:33:36.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:33:36.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:33:36.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:33:36.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:33:36.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:33:36.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:33:36.032 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:33:36.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:36.171+0000 7fec9bbe4080 -1 Falling back to public interface 2021-01-15T16:33:36.444 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 36 up_thru 39 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/78311,v1:127.0.0.1:6803/78311] [v2:127.0.0.1:6804/78311,v1:127.0.0.1:6805/78311] exists,up 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:33:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:33:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:33:36.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:33:36.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:33:36.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2021-01-15T16:33:36.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:33:36.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:33:36.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:33:36.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:33:36.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:33:36.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:33:36.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2021-01-15T16:33:36.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:33:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:33:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2021-01-15T16:33:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:33:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2021-01-15T16:33:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2021-01-15T16:33:37.126 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:37.125+0000 7fec9bbe4080 -1 osd.0 40 log_to_monitors {default=true} 2021-01-15T16:33:37.645 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2021-01-15T16:33:37.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2021-01-15T16:33:37.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2021-01-15T16:33:37.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:33:37.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:33:37.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:33:37.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:33:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/2 2021-01-15T16:33:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:33:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:33:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:33:37.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:33:37.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2021-01-15T16:33:37.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2021-01-15T16:33:37.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:33:37.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:33:37.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:33:37.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:33:37.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:33:37.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:33:37.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:33:37.919 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:33:37.919 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:33:37.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:33:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:33:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:33:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:33:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:33:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:33:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:33:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:33:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:33:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:33:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/2 2021-01-15T16:33:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:33:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:33:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/2/whoami 2021-01-15T16:33:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:33:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:33:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:33:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:33:37.926 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:33:37.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:37.944+0000 7f3788b72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:37.946 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:37.946+0000 7f3788b72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:37.948 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:37.948+0000 7f3788b72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:33:38.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:33:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:33:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:33:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:33:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:33:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:33:38.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:33:38.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:33:38.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:33:38.323 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:33:38.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:38.467+0000 7f3788b72080 -1 Falling back to public interface 2021-01-15T16:33:38.734 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/78800,v1:127.0.0.1:6811/78800] [v2:127.0.0.1:6812/78800,v1:127.0.0.1:6813/78800] exists,up baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:33:38.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:33:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:33:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:33:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2021-01-15T16:33:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/divergent-priors 2021-01-15T16:33:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:33:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:33:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:33:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2021-01-15T16:33:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2021-01-15T16:33:39.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:33:39.609+0000 7f3788b72080 -1 osd.2 40 log_to_monitors {default=true} 2021-01-15T16:33:40.417 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2021-01-15T16:33:40.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2021-01-15T16:33:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2021-01-15T16:33:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2021-01-15T16:33:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:33:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:33:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:33:40.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:33:40.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 20 .1 2021-01-15T16:33:40.685 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:33:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:33:40.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:33:40.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:33:40.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:33:40.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:33:40.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:33:40.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:33:40.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:33:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:33:41.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:33:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:33:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:33:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:33:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:33:41.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:33:41.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=180388626435 2021-01-15T16:33:41.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 180388626435 2021-01-15T16:33:41.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-180388626435' 2021-01-15T16:33:41.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:33:41.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:33:41.394 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.1 2021-01-15T16:33:41.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T16:33:41.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T16:33:41.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T16:33:41.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:33:41.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:33:41.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528323 2021-01-15T16:33:41.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528323 2021-01-15T16:33:41.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-180388626435 2-193273528323' 2021-01-15T16:33:41.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:33:41.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-180388626435 2021-01-15T16:33:41.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:33:41.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:33:41.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-180388626435 2021-01-15T16:33:41.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:33:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=180388626435 2021-01-15T16:33:41.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 180388626435' 2021-01-15T16:33:41.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:33:41.542 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 180388626435 2021-01-15T16:33:41.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 180388626435 2021-01-15T16:33:41.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:33:42.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:33:42.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:33:43.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 180388626436 -lt 180388626435 2021-01-15T16:33:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:33:43.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-193273528323 2021-01-15T16:33:43.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:33:43.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:33:43.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-193273528323 2021-01-15T16:33:43.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:33:43.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528323 2021-01-15T16:33:43.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2021-01-15T16:33:43.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:33:43.345 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 193273528323 2021-01-15T16:33:43.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528323 -lt 193273528323 2021-01-15T16:33:43.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:33:43.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:43.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:44.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:33:44.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:44.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:44.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:44.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:44.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:44.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:44.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:44.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:44.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:44.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:44.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:45.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:45.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:33:45.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:33:45.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:33:45.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:45.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:33:45.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:45.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:45.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:45.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:45.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:45.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:45.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:45.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:45.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:45.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:45.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:45.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:46.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:46.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:46.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 8 )) 2021-01-15T16:33:46.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:46.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T16:33:46.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:46.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:46.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:47.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:47.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:47.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:47.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:47.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:47.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:47.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:47.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:47.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:47.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:47.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:47.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:47.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 8 )) 2021-01-15T16:33:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T16:33:48.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:48.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:48.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:48.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:48.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:48.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:48.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:48.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:49.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:49.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:49.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:49.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:49.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:49.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:49.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:49.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:49.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:50.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:50.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:50.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 8 )) 2021-01-15T16:33:50.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:50.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T16:33:50.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:50.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:50.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:50.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:50.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:50.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:50.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:51.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:51.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:51.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:51.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:51.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:51.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:51.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:51.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:51.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 8 )) 2021-01-15T16:33:52.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:52.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T16:33:53.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:53.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:53.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:53.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:53.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:53.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:53.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:53.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:54.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:54.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:54.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:54.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:54.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:54.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:54.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:54.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:54.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:54.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:54.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:54.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:33:55.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:33:55.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:33:55.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 8 )) 2021-01-15T16:33:55.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:33:55.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T16:33:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:33:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:33:58.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:33:58.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:33:58.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:33:58.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:33:58.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:33:58.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:33:59.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:33:59.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:33:59.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:33:59.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:33:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:33:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:33:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:33:59.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:33:59.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:33:59.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:33:59.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:33:59.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:33:59.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:34:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:34:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:34:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 8 )) 2021-01-15T16:34:00.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:34:00.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T16:34:06.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:34:06.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:34:06.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:34:06.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:34:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:34:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:34:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:34:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:34:06.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:34:06.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:34:06.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:34:06.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:34:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:34:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:34:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:34:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:34:07.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:34:07.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:34:07.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:34:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:34:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:34:07.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:34:07.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:34:07.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 8 )) 2021-01-15T16:34:07.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:34:07.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 7.3 2021-01-15T16:34:15.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:34:15.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:34:15.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:34:15.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:34:15.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:34:15.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:34:15.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:34:15.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:34:15.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:34:15.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:34:15.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:34:15.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:34:16.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:34:16.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:34:16.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:34:16.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:34:16.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:34:16.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:34:16.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:34:16.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:34:16.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:34:16.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T16:34:16.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T16:34:16.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 8 )) 2021-01-15T16:34:16.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T16:34:17.173 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:34:17.174 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "c97721ee-650d-4eba-a0ec-f245b5795511", 2021-01-15T16:34:17.174 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T16:34:17.174 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T16:34:17.174 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T16:34:17.172896+0000", 2021-01-15T16:34:17.175 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T16:34:17.175 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T16:34:17.175 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_WARN", 2021-01-15T16:34:17.175 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T16:34:17.175 INFO:tasks.workunit.client.0.smithi145.stdout: "OSDMAP_FLAGS": { 2021-01-15T16:34:17.176 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:34:17.176 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:34:17.176 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "nodown,noout,noin flag(s) set", 2021-01-15T16:34:17.176 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 17 2021-01-15T16:34:17.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.177 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T16:34:17.177 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:34:17.177 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.177 INFO:tasks.workunit.client.0.smithi145.stdout: "OSD_DOWN": { 2021-01-15T16:34:17.177 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:34:17.178 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:34:17.178 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 osds down", 2021-01-15T16:34:17.178 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:34:17.178 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.179 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:34:17.179 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.179 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "osd.1 (root=default,host=smithi145) is down" 2021-01-15T16:34:17.180 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.180 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.180 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:34:17.180 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.180 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T16:34:17.181 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:34:17.181 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:34:17.181 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2021-01-15T16:34:17.181 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:34:17.182 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.182 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:34:17.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.182 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2021-01-15T16:34:17.183 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.183 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.183 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:34:17.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.184 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T16:34:17.184 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T16:34:17.185 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T16:34:17.185 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T16:34:17.185 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T16:34:17.186 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.186 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T16:34:17.187 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.187 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'test'" 2021-01-15T16:34:17.187 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.187 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.188 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T16:34:17.188 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.188 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.188 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T16:34:17.189 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.189 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T16:34:17.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.190 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T16:34:17.190 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T16:34:17.190 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T16:34:17.190 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:34:17.191 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:34:17.191 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:31:22.975023Z", 2021-01-15T16:34:17.191 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:31:22.975023Z", 2021-01-15T16:34:17.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T16:34:17.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T16:34:17.192 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T16:34:17.193 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T16:34:17.193 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T16:34:17.193 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T16:34:17.193 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T16:34:17.193 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T16:34:17.194 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T16:34:17.194 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T16:34:17.194 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T16:34:17.194 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T16:34:17.195 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T16:34:17.195 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T16:34:17.195 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T16:34:17.195 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.195 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T16:34:17.196 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.196 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T16:34:17.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.197 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T16:34:17.197 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T16:34:17.198 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:34:17.198 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.199 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.199 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115", 2021-01-15T16:34:17.200 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T16:34:17.200 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.200 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.201 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.201 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7115/0", 2021-01-15T16:34:17.202 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7115/0", 2021-01-15T16:34:17.202 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T16:34:17.203 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T16:34:17.203 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T16:34:17.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.204 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.204 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.204 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T16:34:17.205 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T16:34:17.205 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.206 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T16:34:17.206 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 47, 2021-01-15T16:34:17.206 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "02a65673-94d5-4a66-a2ce-b561a2e1cd7f", 2021-01-15T16:34:17.206 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T16:31:23.733503+0000", 2021-01-15T16:34:17.207 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:33:40.495807+0000", 2021-01-15T16:34:17.207 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T16:33:39.984449+0000", 2021-01-15T16:34:17.207 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T16:32:53.674032+0000", 2021-01-15T16:34:17.207 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T16:34:17.207 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5800384, 2021-01-15T16:34:17.208 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T16:34:17.208 INFO:tasks.workunit.client.0.smithi145.stdout: "nodown", 2021-01-15T16:34:17.208 INFO:tasks.workunit.client.0.smithi145.stdout: "noin", 2021-01-15T16:34:17.208 INFO:tasks.workunit.client.0.smithi145.stdout: "noout", 2021-01-15T16:34:17.209 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T16:34:17.209 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T16:34:17.209 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T16:34:17.209 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T16:34:17.210 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.210 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 8, 2021-01-15T16:34:17.210 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T16:34:17.210 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T16:34:17.211 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T16:34:17.211 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T16:34:17.211 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 1, 2021-01-15T16:34:17.211 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 3, 2021-01-15T16:34:17.211 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T16:34:17.212 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T16:34:17.212 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T16:34:17.212 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T16:34:17.212 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.212 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:34:17.213 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "test", 2021-01-15T16:34:17.213 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T16:31:51.449694+0000", 2021-01-15T16:34:17.213 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 16389, 2021-01-15T16:34:17.213 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2021-01-15T16:34:17.214 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2021-01-15T16:34:17.214 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T16:34:17.214 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T16:34:17.214 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 1, 2021-01-15T16:34:17.215 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T16:34:17.215 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T16:34:17.215 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T16:34:17.215 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T16:34:17.216 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T16:34:17.216 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T16:34:17.216 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T16:34:17.216 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T16:34:17.217 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T16:34:17.217 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T16:34:17.217 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T16:34:17.217 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T16:34:17.218 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T16:34:17.218 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T16:34:17.224 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T16:34:17.224 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T16:34:17.224 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T16:34:17.225 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T16:34:17.225 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.225 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "30", 2021-01-15T16:34:17.225 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T16:34:17.226 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T16:34:17.226 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T16:34:17.226 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T16:34:17.227 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "pool", 2021-01-15T16:34:17.227 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 1, 2021-01-15T16:34:17.227 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 30, 2021-01-15T16:34:17.227 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [ 2021-01-15T16:34:17.228 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.228 INFO:tasks.workunit.client.0.smithi145.stdout: "snapid": 1, 2021-01-15T16:34:17.228 INFO:tasks.workunit.client.0.smithi145.stdout: "stamp": "2021-01-15T16:32:34.499003+0000", 2021-01-15T16:34:17.228 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "snap1" 2021-01-15T16:34:17.229 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.229 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.229 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T16:34:17.229 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T16:34:17.230 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T16:34:17.230 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T16:34:17.230 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T16:34:17.230 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T16:34:17.231 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T16:34:17.231 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T16:34:17.231 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T16:34:17.231 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T16:34:17.231 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T16:34:17.232 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T16:34:17.232 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T16:34:17.232 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T16:34:17.232 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T16:34:17.233 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "myprofile", 2021-01-15T16:34:17.233 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T16:34:17.233 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T16:34:17.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.234 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T16:34:17.234 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T16:34:17.234 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T16:34:17.234 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T16:34:17.235 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T16:34:17.235 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T16:34:17.235 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T16:34:17.235 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T16:34:17.236 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 8192, 2021-01-15T16:34:17.236 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T16:34:17.236 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T16:34:17.236 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T16:34:17.236 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T16:34:17.237 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.237 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.237 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T16:34:17.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.238 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:34:17.238 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "6412922c-b560-401d-81b8-a8bbe5ef1e10", 2021-01-15T16:34:17.238 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:34:17.238 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:34:17.238 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:34:17.239 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:34:17.239 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 36, 2021-01-15T16:34:17.239 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 40, 2021-01-15T16:34:17.239 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 42, 2021-01-15T16:34:17.240 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 45, 2021-01-15T16:34:17.240 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 41, 2021-01-15T16:34:17.240 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:34:17.240 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:34:17.240 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.241 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.241 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T16:34:17.241 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.242 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.242 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.242 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T16:34:17.243 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.243 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.243 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.244 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:34:17.245 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.245 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.246 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T16:34:17.246 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.247 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.248 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T16:34:17.248 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.249 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.249 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.250 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:34:17.250 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.251 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.251 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T16:34:17.252 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.252 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.252 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.252 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.253 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T16:34:17.253 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.253 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.254 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.254 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:34:17.254 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.254 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.254 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.255 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T16:34:17.255 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.255 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.256 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.256 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.256 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T16:34:17.256 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 80708 2021-01-15T16:34:17.256 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.257 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.257 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.257 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/80708", 2021-01-15T16:34:17.258 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/80708", 2021-01-15T16:34:17.258 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/80708", 2021-01-15T16:34:17.258 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/80708", 2021-01-15T16:34:17.258 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:34:17.259 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:34:17.259 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:34:17.259 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.259 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.260 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.260 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:34:17.260 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "833bc698-267b-4dab-a554-ed9c97578364", 2021-01-15T16:34:17.260 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 0, 2021-01-15T16:34:17.261 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:34:17.261 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:34:17.261 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:34:17.261 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T16:34:17.261 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T16:34:17.262 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 10, 2021-01-15T16:34:17.262 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 25, 2021-01-15T16:34:17.262 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 32, 2021-01-15T16:34:17.262 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:34:17.263 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:34:17.263 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.263 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.263 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.263 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:34:17.264 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.264 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.264 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.264 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.264 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:34:17.265 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.265 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.265 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.265 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.265 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:34:17.266 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.266 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.266 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.266 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:34:17.267 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.267 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.267 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.267 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.268 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:34:17.268 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.268 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.268 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.269 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.269 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:34:17.269 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.269 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.270 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.270 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:34:17.270 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.270 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.271 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.271 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.271 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:34:17.271 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.272 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.272 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.272 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.272 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:34:17.272 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.273 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.273 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.273 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:34:17.273 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.273 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.274 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.274 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.274 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:34:17.274 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 72672 2021-01-15T16:34:17.275 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.275 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.275 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.275 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/72672", 2021-01-15T16:34:17.275 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/72672", 2021-01-15T16:34:17.276 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/72672", 2021-01-15T16:34:17.276 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/72672", 2021-01-15T16:34:17.276 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:34:17.276 INFO:tasks.workunit.client.0.smithi145.stdout: "exists" 2021-01-15T16:34:17.276 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.277 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.277 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.277 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:34:17.278 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "baffd605-38c3-450d-87ae-a21bfb354b1f", 2021-01-15T16:34:17.278 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T16:34:17.278 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T16:34:17.278 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T16:34:17.279 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T16:34:17.279 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 37, 2021-01-15T16:34:17.279 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 40, 2021-01-15T16:34:17.279 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 45, 2021-01-15T16:34:17.280 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 0, 2021-01-15T16:34:17.280 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 44, 2021-01-15T16:34:17.280 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T16:34:17.280 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T16:34:17.280 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.281 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.281 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.281 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T16:34:17.281 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.282 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.282 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.282 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.282 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T16:34:17.283 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.283 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.283 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.283 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.284 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T16:34:17.284 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.284 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.284 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.284 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T16:34:17.285 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.285 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.285 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.285 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.286 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T16:34:17.286 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.286 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.286 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.286 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.287 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T16:34:17.287 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.287 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.287 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.287 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T16:34:17.288 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.288 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.288 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.289 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.289 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T16:34:17.289 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.289 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.289 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.290 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.290 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T16:34:17.290 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T16:34:17.290 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.291 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T16:34:17.291 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T16:34:17.291 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.291 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.291 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.292 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T16:34:17.292 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T16:34:17.292 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 81537 2021-01-15T16:34:17.292 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.293 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.293 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.293 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/81537", 2021-01-15T16:34:17.293 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/81537", 2021-01-15T16:34:17.293 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/81537", 2021-01-15T16:34:17.294 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/81537", 2021-01-15T16:34:17.294 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T16:34:17.294 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T16:34:17.294 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T16:34:17.295 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.295 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.295 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.295 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T16:34:17.295 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.296 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:34:17.296 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:33:37.218921+0000", 2021-01-15T16:34:17.296 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:34:17.296 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:34:17.297 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:34:17.297 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:34:17.297 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:32.371078+0000", 2021-01-15T16:34:17.297 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:34:17.297 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.298 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.298 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:34:17.298 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:32:46.110699+0000", 2021-01-15T16:34:17.299 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:34:17.299 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:34:17.299 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:34:17.299 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:34:17.299 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:40.332739+0000", 2021-01-15T16:34:17.300 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:34:17.300 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.300 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.300 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:34:17.301 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "2021-01-15T16:33:39.708882+0000", 2021-01-15T16:34:17.301 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T16:34:17.301 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T16:34:17.301 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T16:34:17.301 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T16:34:17.302 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T16:31:48.347963+0000", 2021-01-15T16:34:17.302 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T16:34:17.302 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.302 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T16:34:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T16:34:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T16:34:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T16:34:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T16:34:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T16:34:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T16:34:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:34:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T16:34:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:34:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T16:34:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "myprofile": { 2021-01-15T16:34:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-device-class": "", 2021-01-15T16:34:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-failure-domain": "osd", 2021-01-15T16:34:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: "crush-root": "default", 2021-01-15T16:34:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: "jerasure-per-chunk-alignment": "false", 2021-01-15T16:34:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T16:34:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "1", 2021-01-15T16:34:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T16:34:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van", 2021-01-15T16:34:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "w": "8" 2021-01-15T16:34:17.309 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.309 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.309 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T16:34:17.310 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T16:34:17.310 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T16:34:17.311 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T16:34:17.311 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T16:34:17.312 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T16:34:17.312 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T16:34:17.312 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T16:34:17.313 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T16:34:17.313 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T16:34:17.314 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T16:34:17.314 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.314 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.315 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T16:34:17.315 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.316 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:34:17.316 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:34:17.316 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/80708,v1:127.0.0.1:6805/80708]", 2021-01-15T16:34:17.317 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:34:17.317 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:34:17.318 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:34:17.318 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:34:17.319 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:34:17.319 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:34:17.319 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:34:17.320 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:34:17.320 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:34:17.321 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2021-01-15T16:34:17.321 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:34:17.322 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:34:17.322 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:34:17.322 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:34:17.323 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:34:17.323 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:34:17.324 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:34:17.324 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:34:17.325 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:34:17.325 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:34:17.325 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:34:17.326 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:34:17.326 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:34:17.327 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:34:17.327 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:34:17.328 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/80708,v1:127.0.0.1:6803/80708]", 2021-01-15T16:34:17.328 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:34:17.328 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/80708,v1:127.0.0.1:6809/80708]", 2021-01-15T16:34:17.329 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/80708,v1:127.0.0.1:6807/80708]", 2021-01-15T16:34:17.329 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:34:17.330 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:34:17.330 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:34:17.330 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:34:17.331 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:34:17.331 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:34:17.331 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:34:17.332 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:34:17.332 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:34:17.332 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/0", 2021-01-15T16:34:17.333 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:34:17.333 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:34:17.333 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:34:17.333 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.334 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.334 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:34:17.334 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:34:17.334 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/72672,v1:127.0.0.1:6813/72672]", 2021-01-15T16:34:17.335 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:34:17.335 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:34:17.335 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:34:17.335 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:34:17.336 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:34:17.336 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:34:17.336 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:34:17.336 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:34:17.337 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:34:17.337 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2021-01-15T16:34:17.337 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:34:17.337 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:34:17.338 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:34:17.338 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:34:17.338 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:34:17.338 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:34:17.338 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:34:17.339 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:34:17.339 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:34:17.339 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:34:17.339 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:34:17.340 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:34:17.340 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:34:17.340 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:34:17.340 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:34:17.340 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/72672,v1:127.0.0.1:6811/72672]", 2021-01-15T16:34:17.341 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:34:17.341 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/72672,v1:127.0.0.1:6817/72672]", 2021-01-15T16:34:17.341 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/72672,v1:127.0.0.1:6815/72672]", 2021-01-15T16:34:17.341 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:34:17.342 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:34:17.342 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:34:17.342 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:34:17.342 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:34:17.343 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:34:17.343 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:34:17.343 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:34:17.343 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:34:17.343 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/1", 2021-01-15T16:34:17.344 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:34:17.344 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:34:17.344 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:34:17.344 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.344 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.345 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:34:17.345 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T16:34:17.345 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/81537,v1:127.0.0.1:6813/81537]", 2021-01-15T16:34:17.345 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T16:34:17.346 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T16:34:17.346 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T16:34:17.346 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T16:34:17.346 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T16:34:17.346 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T16:34:17.347 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T16:34:17.347 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T16:34:17.347 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T16:34:17.347 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2021-01-15T16:34:17.347 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T16:34:17.348 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T16:34:17.348 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T16:34:17.348 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T16:34:17.348 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T16:34:17.349 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T16:34:17.349 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T16:34:17.349 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T16:34:17.349 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T16:34:17.349 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T16:34:17.350 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T16:34:17.350 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T16:34:17.350 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T16:34:17.350 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T16:34:17.351 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T16:34:17.351 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/81537,v1:127.0.0.1:6811/81537]", 2021-01-15T16:34:17.351 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T16:34:17.351 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/81537,v1:127.0.0.1:6817/81537]", 2021-01-15T16:34:17.351 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/81537,v1:127.0.0.1:6815/81537]", 2021-01-15T16:34:17.352 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T16:34:17.352 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T16:34:17.352 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T16:34:17.352 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T16:34:17.352 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T16:34:17.353 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T16:34:17.353 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T16:34:17.353 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T16:34:17.353 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T16:34:17.354 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/divergent-priors/2", 2021-01-15T16:34:17.354 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T16:34:17.354 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T16:34:17.354 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T16:34:17.354 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.355 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.355 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T16:34:17.355 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T16:34:17.355 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T16:34:17.356 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [] 2021-01-15T16:34:17.356 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.356 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T16:34:17.356 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.357 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:34:17.357 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 43 2021-01-15T16:34:17.357 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.357 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.358 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:34:17.358 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 10 2021-01-15T16:34:17.358 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.358 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.359 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:34:17.359 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 46 2021-01-15T16:34:17.359 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.359 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.359 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.360 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T16:34:17.360 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 47, 2021-01-15T16:34:17.360 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T16:34:17.360 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T16:34:17.361 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.361 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:34:17.361 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T16:34:17.361 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:34:17.361 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.362 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.362 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:34:17.362 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T16:34:17.362 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:34:17.363 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.363 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.363 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:34:17.363 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T16:34:17.363 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T16:34:17.364 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.364 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.364 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T16:34:17.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.364 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T16:34:17.365 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T16:34:17.365 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.365 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.365 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:34:17.365 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T16:34:17.366 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.366 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.366 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T16:34:17.367 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T16:34:17.367 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.367 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.367 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T16:34:17.367 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T16:34:17.368 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.368 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.368 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T16:34:17.368 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T16:34:17.368 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.369 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.369 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T16:34:17.369 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T16:34:17.369 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.370 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.370 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T16:34:17.370 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T16:34:17.370 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.371 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.371 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T16:34:17.371 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T16:34:17.371 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.372 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.372 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T16:34:17.372 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T16:34:17.372 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.372 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.373 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T16:34:17.373 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T16:34:17.373 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.373 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.374 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T16:34:17.374 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T16:34:17.374 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.374 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.374 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:34:17.375 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T16:34:17.375 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.375 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.375 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T16:34:17.375 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.376 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T16:34:17.376 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T16:34:17.376 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:34:17.376 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:34:17.377 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.377 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:34:17.377 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:34:17.377 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:34:17.377 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.378 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:34:17.378 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.378 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:34:17.378 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.378 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.379 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.379 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.379 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T16:34:17.379 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T16:34:17.380 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T16:34:17.380 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T16:34:17.380 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.380 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:34:17.380 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:34:17.381 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:34:17.381 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.381 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:34:17.381 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.381 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:34:17.382 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.382 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.382 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.382 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.382 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T16:34:17.383 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T16:34:17.383 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:34:17.383 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:34:17.383 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.384 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:34:17.384 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:34:17.384 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:34:17.384 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.384 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:34:17.385 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.385 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:34:17.385 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.385 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.386 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:34:17.386 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.386 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:34:17.387 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.387 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.387 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:34:17.387 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.387 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:34:17.388 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.388 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.388 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.388 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.389 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T16:34:17.389 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T16:34:17.389 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T16:34:17.389 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T16:34:17.390 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 19206, 2021-01-15T16:34:17.390 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T16:34:17.390 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T16:34:17.391 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:34:17.391 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.391 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T16:34:17.392 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.392 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T16:34:17.392 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.393 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.393 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T16:34:17.394 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.394 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T16:34:17.394 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.395 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.395 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T16:34:17.396 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T16:34:17.396 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T16:34:17.396 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.397 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.397 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.397 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.398 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T16:34:17.398 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.399 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T16:34:17.399 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T16:34:17.399 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T16:34:17.400 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T16:34:17.400 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T16:34:17.400 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T16:34:17.400 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:34:17.401 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.401 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:34:17.401 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:34:17.401 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:34:17.402 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.402 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.402 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T16:34:17.402 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:34:17.402 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:34:17.403 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.403 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.403 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:34:17.403 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.404 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.404 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.404 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.404 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 1, 2021-01-15T16:34:17.404 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "test", 2021-01-15T16:34:17.405 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 1, 2021-01-15T16:34:17.405 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 3, 2021-01-15T16:34:17.405 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 3, 2021-01-15T16:34:17.405 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 3, 2021-01-15T16:34:17.405 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T16:34:17.406 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.406 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_chooseleaf_tries", 2021-01-15T16:34:17.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 5 2021-01-15T16:34:17.406 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.407 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.407 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "set_choose_tries", 2021-01-15T16:34:17.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 100 2021-01-15T16:34:17.407 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.408 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.408 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T16:34:17.408 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T16:34:17.408 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T16:34:17.408 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.409 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.409 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_indep", 2021-01-15T16:34:17.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T16:34:17.409 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T16:34:17.409 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.410 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.410 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T16:34:17.410 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.410 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:34:17.411 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.411 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.411 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T16:34:17.411 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T16:34:17.411 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T16:34:17.412 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T16:34:17.412 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T16:34:17.412 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T16:34:17.412 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T16:34:17.412 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T16:34:17.413 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T16:34:17.413 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T16:34:17.413 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T16:34:17.413 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T16:34:17.413 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T16:34:17.414 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T16:34:17.414 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T16:34:17.414 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 1, 2021-01-15T16:34:17.414 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T16:34:17.415 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T16:34:17.415 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T16:34:17.415 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T16:34:17.415 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T16:34:17.415 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.416 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T16:34:17.416 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.416 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T16:34:17.416 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T16:34:17.417 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T16:34:17.417 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T16:34:17.417 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T16:34:17.417 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T16:34:17.417 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T16:34:17.418 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T16:34:17.418 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T16:34:17.418 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T16:34:17.418 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T16:34:17.418 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T16:34:17.419 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T16:34:17.419 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T16:34:17.419 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T16:34:17.419 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T16:34:17.420 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.420 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.420 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T16:34:17.420 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T16:34:17.420 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T16:34:17.421 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.421 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T16:34:17.421 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T16:34:17.421 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.421 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T16:34:17.422 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T16:34:17.422 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T16:34:17.422 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:34:17.422 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 5, 2021-01-15T16:34:17.423 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 3 2021-01-15T16:34:17.423 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.423 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:34:17.423 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 1, 2021-01-15T16:34:17.423 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T16:34:17.424 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 3, 2021-01-15T16:34:17.424 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T16:34:17.424 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:34:17.424 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2021-01-15T16:34:17.424 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2021-01-15T16:34:17.425 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2021-01-15T16:34:17.425 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2021-01-15T16:34:17.425 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:34:17.425 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:34:17.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2021-01-15T16:34:17.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:34:17.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:34:17.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2021-01-15T16:34:17.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:34:17.427 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:34:17.427 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:34:17.427 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:34:17.427 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:34:17.428 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:34:17.428 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:34:17.428 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:34:17.428 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:34:17.428 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:34:17.429 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:34:17.429 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:34:17.429 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:34:17.429 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:34:17.429 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:34:17.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:34:17.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:34:17.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:34:17.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:34:17.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:34:17.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:34:17.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:34:17.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:34:17.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:34:17.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:34:17.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:34:17.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:34:17.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:34:17.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:34:17.433 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:34:17.433 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.433 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:34:17.433 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:34:17.434 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:34:17.434 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:34:17.434 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T16:34:17.434 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T16:34:17.434 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:34:17.435 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:34:17.435 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:34:17.435 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:34:17.435 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:34:17.435 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.436 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2021-01-15T16:34:17.436 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2021-01-15T16:34:17.436 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2021-01-15T16:34:17.436 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2021-01-15T16:34:17.437 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T16:34:17.437 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.437 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T16:34:17.437 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:34:17.437 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:34:17.438 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:34:17.438 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:34:17.438 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:34:17.438 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:34:17.438 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:34:17.439 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19028, 2021-01-15T16:34:17.439 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2228, 2021-01-15T16:34:17.439 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 4, 2021-01-15T16:34:17.439 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16763, 2021-01-15T16:34:17.440 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553772, 2021-01-15T16:34:17.440 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:34:17.440 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:34:17.440 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103062528, 2021-01-15T16:34:17.440 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:34:17.441 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2281472, 2021-01-15T16:34:17.441 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1568297, 2021-01-15T16:34:17.441 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:34:17.441 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:34:17.441 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:34:17.442 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 4434, 2021-01-15T16:34:17.442 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17165998 2021-01-15T16:34:17.442 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.442 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:34:17.442 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:34:17.443 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:34:17.443 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:34:17.443 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:34:17.443 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:34:17.444 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:34:17.444 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.444 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:34:17.444 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:34:17.444 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:34:17.445 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:34:17.445 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:34:17.445 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.445 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:34:17.445 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:34:17.446 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.446 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T16:34:17.446 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T16:34:17.446 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T16:34:17.447 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T16:34:17.447 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 3, 2021-01-15T16:34:17.447 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 3, 2021-01-15T16:34:17.447 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 3, 2021-01-15T16:34:17.447 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 3, 2021-01-15T16:34:17.448 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 314572800, 2021-01-15T16:34:17.448 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 19028, 2021-01-15T16:34:17.448 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 2228, 2021-01-15T16:34:17.448 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 4, 2021-01-15T16:34:17.448 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 16763, 2021-01-15T16:34:17.449 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 314553772, 2021-01-15T16:34:17.449 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T16:34:17.449 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 322122547200, 2021-01-15T16:34:17.449 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 322103062528, 2021-01-15T16:34:17.450 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:34:17.450 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 2281472, 2021-01-15T16:34:17.450 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1568297, 2021-01-15T16:34:17.450 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:34:17.450 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:34:17.451 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:34:17.451 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 4434, 2021-01-15T16:34:17.451 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 17165998 2021-01-15T16:34:17.451 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.452 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T16:34:17.452 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T16:34:17.452 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T16:34:17.452 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 0, 2021-01-15T16:34:17.452 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T16:34:17.453 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [], 2021-01-15T16:34:17.453 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 1 2021-01-15T16:34:17.453 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.453 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T16:34:17.453 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T16:34:17.454 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T16:34:17.454 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T16:34:17.454 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T16:34:17.454 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.455 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T16:34:17.455 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T16:34:17.455 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.455 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.455 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T16:34:17.456 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.456 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T16:34:17.456 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T16:34:17.456 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T16:34:17.457 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 48430, 2021-01-15T16:34:17.457 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 101, 2021-01-15T16:34:17.457 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 1, 2021-01-15T16:34:17.457 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 303, 2021-01-15T16:34:17.457 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T16:34:17.458 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T16:34:17.458 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 101, 2021-01-15T16:34:17.458 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T16:34:17.458 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T16:34:17.459 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 101, 2021-01-15T16:34:17.459 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T16:34:17.459 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T16:34:17.459 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T16:34:17.459 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 101, 2021-01-15T16:34:17.460 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 101, 2021-01-15T16:34:17.460 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T16:34:17.460 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T16:34:17.460 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T16:34:17.460 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T16:34:17.461 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T16:34:17.461 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T16:34:17.461 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T16:34:17.461 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T16:34:17.461 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T16:34:17.462 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T16:34:17.462 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T16:34:17.462 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T16:34:17.462 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T16:34:17.463 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T16:34:17.463 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T16:34:17.463 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T16:34:17.463 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T16:34:17.463 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T16:34:17.464 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T16:34:17.464 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T16:34:17.464 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T16:34:17.464 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T16:34:17.464 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T16:34:17.465 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T16:34:17.465 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T16:34:17.465 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.465 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T16:34:17.466 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T16:34:17.466 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T16:34:17.466 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T16:34:17.466 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1241088, 2021-01-15T16:34:17.466 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 1241088, 2021-01-15T16:34:17.467 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T16:34:17.467 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T16:34:17.467 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T16:34:17.467 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T16:34:17.467 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T16:34:17.468 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.468 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 18, 2021-01-15T16:34:17.468 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 18, 2021-01-15T16:34:17.468 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2021-01-15T16:34:17.469 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2021-01-15T16:34:17.469 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T16:34:17.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.469 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.469 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T16:34:17.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.470 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:34:17.470 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 180388626442 2021-01-15T16:34:17.470 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.471 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:34:17.471 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 42949672981 2021-01-15T16:34:17.471 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.471 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.471 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:34:17.472 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 193273528330 2021-01-15T16:34:17.472 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.472 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.472 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T16:34:17.472 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.473 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+undersized+degraded", 2021-01-15T16:34:17.473 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T16:34:17.473 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.473 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.474 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T16:34:17.474 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.474 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T16:34:17.474 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:34:17.474 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:34:17.475 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:34:17.475 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.475 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.475 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2147483647, 2021-01-15T16:34:17.475 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:34:17.476 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:34:17.476 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:34:17.476 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.476 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.476 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T16:34:17.477 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 0, 2021-01-15T16:34:17.477 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 0, 2021-01-15T16:34:17.477 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:34:17.477 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.478 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.478 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T16:34:17.478 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T16:34:17.478 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 1, 2021-01-15T16:34:17.478 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T16:34:17.479 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.479 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.479 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T16:34:17.479 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:34:17.480 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T16:34:17.480 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T16:34:17.480 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.480 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:34:17.480 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T16:34:17.481 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 5, 2021-01-15T16:34:17.481 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T16:32:50.370955+0000", 2021-01-15T16:34:17.481 INFO:tasks.workunit.client.0.smithi145.stdout: "services": { 2021-01-15T16:34:17.481 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": { 2021-01-15T16:34:17.481 INFO:tasks.workunit.client.0.smithi145.stdout: "daemons": { 2021-01-15T16:34:17.482 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": "", 2021-01-15T16:34:17.482 INFO:tasks.workunit.client.0.smithi145.stdout: "0": { 2021-01-15T16:34:17.482 INFO:tasks.workunit.client.0.smithi145.stdout: "start_epoch": 0, 2021-01-15T16:34:17.482 INFO:tasks.workunit.client.0.smithi145.stdout: "start_stamp": "0.000000", 2021-01-15T16:34:17.482 INFO:tasks.workunit.client.0.smithi145.stdout: "gid": 0, 2021-01-15T16:34:17.483 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "(unrecognized address family 0)/0", 2021-01-15T16:34:17.483 INFO:tasks.workunit.client.0.smithi145.stdout: "metadata": {}, 2021-01-15T16:34:17.483 INFO:tasks.workunit.client.0.smithi145.stdout: "task_status": {} 2021-01-15T16:34:17.483 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.484 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:34:17.484 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T16:34:17.485 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T16:34:17.485 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 506, 2021-01-15T16:34:17.485 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T16:34:17.485 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T16:34:17.485 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:34:17.486 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:34:17.487 INFO:tasks.workunit.client.0.smithi145.stderr:report 4286554038 2021-01-15T16:34:17.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T16:34:17.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2021-01-15T16:34:17.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:17.488 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery 2021-01-15T16:34:17.553 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:17.553 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-01-15T16:33:40.730375+0000 40'102 47:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:17.554 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:17.554 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:17.555 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:17.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:34:17.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2021-01-15T16:34:17.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:34:17.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:34:17.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:34:17.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:34:17.566 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:34:17.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.0.asok 2021-01-15T16:34:17.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-01-15T16:34:17.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.0.asok set_recovery_delay 100000 2021-01-15T16:34:17.668 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:34:17.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2021-01-15T16:34:17.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2021-01-15T16:34:17.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T16:34:17.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T16:34:17.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:34:17.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:34:17.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:34:17.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.2.asok 2021-01-15T16:34:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2021-01-15T16:34:17.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.2.asok set_recovery_delay 100000 2021-01-15T16:34:17.781 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:34:17.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2021-01-15T16:34:17.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:17.790 INFO:tasks.workunit.client.0.smithi145.stdout:revive divergent 1 2021-01-15T16:34:18.160 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:18.160 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-01-15T16:33:40.730375+0000 40'102 47:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:18.160 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:18.160 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:18.162 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:18.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2021-01-15T16:34:18.932 INFO:tasks.workunit.client.0.smithi145.stderr:noup is set 2021-01-15T16:34:18.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-01-15T16:34:18.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:34:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:34:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:34:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:34:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:34:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:34:18.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:34:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:34:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:34:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:34:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:34:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:34:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:34:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:34:18.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:34:18.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:34:18.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:34:18.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:34:18.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:34:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:34:18.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:34:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:34:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:34:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:34:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:34:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:34:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:34:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:34:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:34:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:34:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:34:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:34:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:34:18.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:34:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:34:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:34:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:34:18.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:34:18.954 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:34:18.971 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:18.971+0000 7f8d195d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:18.973 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:18.973+0000 7f8d195d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:18.974 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:18.974+0000 7f8d195d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:19.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: activate_osd: return 0 2021-01-15T16:34:19.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2021-01-15T16:34:19.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:19.490+0000 7f8d195d1080 -1 Falling back to public interface 2021-01-15T16:34:20.343 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:20.342+0000 7f8d195d1080 -1 osd.1 30 log_to_monitors {default=true} 2021-01-15T16:34:24.350 INFO:tasks.workunit.client.0.smithi145.stdout:delay recovery divergent 2021-01-15T16:34:24.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2021-01-15T16:34:24.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:24.748 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:24.748 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-01-15T16:33:40.730375+0000 40'102 47:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:24.748 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:24.748 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:24.750 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:24.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2021-01-15T16:34:24.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:34:24.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:34:24.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:34:24.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:34:24.761 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:34:24.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.33425/ceph-osd.1.asok 2021-01-15T16:34:24.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2021-01-15T16:34:24.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.33425/ceph-osd.1.asok set_recovery_delay 100000 2021-01-15T16:34:24.863 INFO:tasks.workunit.client.0.smithi145.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2021-01-15T16:34:24.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2021-01-15T16:34:25.630 INFO:tasks.workunit.client.0.smithi145.stderr:noup is unset 2021-01-15T16:34:25.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2021-01-15T16:34:25.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:34:25.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:34:25.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:34:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:34:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:34:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:34:25.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:34:25.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:34:25.643 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:34:26.043 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 42 up_thru 45 down_at 41 last_clean_interval [36,40) [v2:127.0.0.1:6802/80708,v1:127.0.0.1:6803/80708] [v2:127.0.0.1:6804/80708,v1:127.0.0.1:6805/80708] exists,up 6412922c-b560-401d-81b8-a8bbe5ef1e10 2021-01-15T16:34:26.043 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:34:26.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:34:26.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:34:26.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:34:26.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2021-01-15T16:34:26.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:34:26.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:34:26.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:34:26.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:34:26.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:34:26.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:34:26.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:34:26.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:34:26.438 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/83413,v1:127.0.0.1:6819/83413] [v2:127.0.0.1:6820/83413,v1:127.0.0.1:6821/83413] exists,up 833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:34:26.438 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:34:26.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:34:26.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:34:26.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:34:26.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2021-01-15T16:34:26.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:34:26.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:34:26.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:34:26.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:34:26.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:34:26.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:34:26.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:34:26.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:34:26.866 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 44 last_clean_interval [37,40) [v2:127.0.0.1:6810/81537,v1:127.0.0.1:6811/81537] [v2:127.0.0.1:6812/81537,v1:127.0.0.1:6813/81537] exists,up baffd605-38c3-450d-87ae-a21bfb354b1f 2021-01-15T16:34:26.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:34:26.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:34:26.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:34:26.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:27.239 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:27.240 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-01-15T16:33:40.730375+0000 40'102 47:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:27.240 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:27.240 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:27.241 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:27.251 INFO:tasks.workunit.client.0.smithi145.stdout:wait for peering 2021-01-15T16:34:27.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2021-01-15T16:34:27.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:27.620 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:27.620 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2021-01-15T16:33:40.730375+0000 40'102 47:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:27.620 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:27.621 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:27.622 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:27.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2021-01-15T16:34:36.826 INFO:tasks.workunit.client.0.smithi145.stdout:killing divergent 1 2021-01-15T16:34:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2021-01-15T16:34:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:37.209 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:37.210 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-01-15T16:34:25.846412+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:37.210 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:37.210 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:37.211 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2021-01-15T16:34:37.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:34:37.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:34:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:34:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:34:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:34:37.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:34:37.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2021-01-15T16:34:37.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:37.328 INFO:tasks.workunit.client.0.smithi145.stdout:reviving divergent 1 2021-01-15T16:34:37.700 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:37.700 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2021-01-15T16:34:25.846412+0000 40'102 52:129 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:37.701 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:37.701 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:37.702 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:37.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2021-01-15T16:34:37.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/divergent-priors 2021-01-15T16:34:37.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:34:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:34:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:34:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/divergent-priors/1 2021-01-15T16:34:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2021-01-15T16:34:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:34:37.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:34:37.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:34:37.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2021-01-15T16:34:37.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2021-01-15T16:34:37.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:34:37.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:34:37.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2021-01-15T16:34:37.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:34:37.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:34:37.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:34:37.718 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:34:37.718 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:34:37.718 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:34:37.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:34:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' 2021-01-15T16:34:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:34:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2021-01-15T16:34:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2021-01-15T16:34:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:34:37.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:34:37.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:34:37.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:34:37.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:34:37.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/divergent-priors/1 2021-01-15T16:34:37.721 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:34:37.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:34:37.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=02a65673-94d5-4a66-a2ce-b561a2e1cd7f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.33425/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:34:37.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/divergent-priors/1/whoami 2021-01-15T16:34:37.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:34:37.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:34:37.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:34:37.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:34:37.742 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:37.742+0000 7fe6c6bc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:37.744 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:37.744+0000 7fe6c6bc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:37.746 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:37.745+0000 7fe6c6bc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:34:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:34:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:34:38.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:34:38.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:34:38.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:34:38.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:34:38.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:34:38.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:34:38.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:34:38.120 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:34:38.266 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:38.265+0000 7fe6c6bc4080 -1 Falling back to public interface 2021-01-15T16:34:38.537 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 51 up_thru 51 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/83413,v1:127.0.0.1:6819/83413] [v2:127.0.0.1:6820/83413,v1:127.0.0.1:6821/83413] exists,up 833bc698-267b-4dab-a554-ed9c97578364 2021-01-15T16:34:38.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:34:38.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:34:38.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:34:38.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2021-01-15T16:34:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:34:39.246+0000 7fe6c6bc4080 -1 osd.1 52 log_to_monitors {default=true} 2021-01-15T16:34:58.538 INFO:tasks.workunit.client.0.smithi145.stdout:allowing recovery 2021-01-15T16:34:58.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2021-01-15T16:34:58.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:58.932 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:58.933 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-01-15T16:34:40.808875+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:58.933 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:58.933 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:58.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-01-15T16:34:58.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T16:34:59.076 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:34:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-01-15T16:34:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T16:34:59.217 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:34:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2021-01-15T16:34:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2021-01-15T16:34:59.359 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:34:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2021-01-15T16:34:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2021-01-15T16:34:59.368 INFO:tasks.workunit.client.0.smithi145.stdout:reading divergent objects 2021-01-15T16:34:59.737 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:34:59.738 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2021-01-15T16:34:40.808875+0000 52'103 55:153 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T16:31:51.528302+0000 0'0 2021-01-15T16:31:51.528302+0000 0 2021-01-15T16:34:59.738 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:34:59.738 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:34:59.739 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:34:59.749 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2021-01-15T16:34:59.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2021-01-15T16:34:59.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:34:59.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2021-01-15T16:34:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:34:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2021-01-15T16:34:59.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:34:59.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2021-01-15T16:34:59.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2021-01-15T16:34:59.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2021-01-15T16:34:59.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2021-01-15T16:34:59.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-01-15T16:34:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:34:59.910 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-01-15T16:32:27.167463+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.911 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2021-01-15T16:32:27.167463+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.911 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:34:59.911 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2021-01-15T16:34:59.911 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2021-01-15T16:34:59.912 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2021-01-15T16:34:59.912 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2021-01-15T16:32:34.753222+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.912 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2021-01-15T16:32:34.753222+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.912 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:34:59.912 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2021-01-15T16:34:59.913 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2021-01-15T16:34:59.913 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:34:59.913 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2021-01-15T16:34:59.913 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2021-01-15T16:32:34.753222+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.913 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2021-01-15T16:34:59.914 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2021-01-15T16:32:33.492800+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2021-01-15T16:32:34.500165+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.914 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2021-01-15T16:32:33.492800+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.915 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2021-01-15T16:32:34.500165+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.915 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2021-01-15T16:34:59.915 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2021-01-15T16:34:59.915 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2021-01-15T16:34:59.915 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2021-01-15T16:34:59.916 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2021-01-15T16:34:59.916 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2021-01-15T16:32:34.500165+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2021-01-15T16:34:59.916 INFO:tasks.workunit.client.0.smithi145.stdout:td/divergent-priors/osd.1.log:2021-01-15T16:34:25.656+0000 7f8cf6aa5700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2021-01-15T16:34:59.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2021-01-15T16:34:59.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log td/divergent-priors/osd.2.log 2021-01-15T16:34:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2021-01-15T16:34:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2021-01-15T16:34:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T16:34:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T16:34:59.919 INFO:tasks.workunit.client.0.smithi145.stdout:success 2021-01-15T16:35:00.426 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T16:35:00.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2021-01-15T16:35:00.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:00.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:00.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:00.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:00.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:00.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2021-01-15T16:35:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:35:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:35:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:35:00.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:00.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:00.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:00.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:00.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:00.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:00.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:00.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:00.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:00.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:00.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:00.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:00.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:00.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:35:00.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:35:00.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:00.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:00.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:35:00.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:35:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:00.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T16:35:00.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/divergent-priors 0 2021-01-15T16:35:00.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2021-01-15T16:35:00.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T16:35:00.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2021-01-15T16:35:00.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:00.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:00.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:00.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:00.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:00.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:00.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:00.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:00.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:00.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:00.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:00.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:00.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:00.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:00.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:00.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T16:35:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2021-01-15T16:35:00.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:00.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:00.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.33425 2021-01-15T16:35:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.33425 2021-01-15T16:35:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T16:35:00.581 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T16:35:00.582 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T16:35:00.653 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2021-01-15T16:35:00.654 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2021-01-15T16:35:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:35:00.724 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:35:00.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:35:00.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:35:00.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:35:00.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:35:00.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:35:00.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:35:00.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:35:00.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:35:00.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/ec-error-rollforward 2021-01-15T16:35:00.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2021-01-15T16:35:00.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2021-01-15T16:35:00.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2021-01-15T16:35:00.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2021-01-15T16:35:00.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2021-01-15T16:35:00.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2021-01-15T16:35:00.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none ' 2021-01-15T16:35:00.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:00.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2021-01-15T16:35:00.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2021-01-15T16:35:00.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2021-01-15T16:35:00.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2021-01-15T16:35:00.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2021-01-15T16:35:00.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2021-01-15T16:35:00.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2021-01-15T16:35:00.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:35:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2021-01-15T16:35:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2021-01-15T16:35:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2021-01-15T16:35:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2021-01-15T16:35:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2021-01-15T16:35:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-01-15T16:35:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:35:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-01-15T16:35:00.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:00.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:00.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:00.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:00.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:00.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:00.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:00.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:00.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:00.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:00.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:00.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:00.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:00.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:35:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-01-15T16:35:00.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:00.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:00.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:00.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84897 2021-01-15T16:35:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2021-01-15T16:35:00.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:35:00.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:00.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.84897 2021-01-15T16:35:00.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:35:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:35:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:35:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:35:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2021-01-15T16:35:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2021-01-15T16:35:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2021-01-15T16:35:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2021-01-15T16:35:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2021-01-15T16:35:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:35:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:35:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:35:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2021-01-15T16:35:00.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2021-01-15T16:35:01.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:35:01.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:01.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:01.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:01.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:35:01.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:35:01.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:35:01.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:35:01.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:35:01.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:35:01.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:35:01.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:35:01.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:35:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:35:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:35:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84897/ceph-mon.a.asok 2021-01-15T16:35:01.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:35:01.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.84897/ceph-mon.a.asok config get fsid 2021-01-15T16:35:01.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:35:01.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:35:01.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:35:01.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:35:01.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:35:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:35:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:35:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:35:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:35:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:01.302 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:01.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.84897/ceph-mon.a.asok 2021-01-15T16:35:01.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:35:01.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.84897/ceph-mon.a.asok config get mon_host 2021-01-15T16:35:01.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2021-01-15T16:35:01.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/ec-error-rollforward 2021-01-15T16:35:01.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:35:01.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:35:01.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:35:01.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/ec-error-rollforward/x 2021-01-15T16:35:01.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:35:01.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:35:01.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:01.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:01.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:01.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:01.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:01.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:01.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:35:01.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:35:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2021-01-15T16:35:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:01.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:35:01.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:01.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/0 2021-01-15T16:35:01.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:01.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:01.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:01.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-01-15T16:35:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-01-15T16:35:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:01.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:01.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:01.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:01.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:01.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:01.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:01.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:01.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:01.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:01.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:01.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/0 2021-01-15T16:35:01.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:01.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9ac67ebe-ca64-4bd4-856c-257a1482c98d 2021-01-15T16:35:01.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 9ac67ebe-ca64-4bd4-856c-257a1482c98d' 2021-01-15T16:35:01.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:01.942 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 9ac67ebe-ca64-4bd4-856c-257a1482c98d 2021-01-15T16:35:01.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA1xAFgsIEpOBAA0byXEeo9iDHwrlO/XRD+Mw== 2021-01-15T16:35:01.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA1xAFgsIEpOBAA0byXEeo9iDHwrlO/XRD+Mw=="}' 2021-01-15T16:35:01.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9ac67ebe-ca64-4bd4-856c-257a1482c98d -i td/ec-error-rollforward/0/new.json 2021-01-15T16:35:02.279 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:02.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/0/new.json 2021-01-15T16:35:02.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA1xAFgsIEpOBAA0byXEeo9iDHwrlO/XRD+Mw== --osd-uuid 9ac67ebe-ca64-4bd4-856c-257a1482c98d 2021-01-15T16:35:02.312 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.312+0000 7f975b91f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:02.314 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.314+0000 7f975b91f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.316+0000 7f975b91f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.316+0000 7f975b91f080 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-01-15T16:35:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.316+0000 7f975b91f080 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-01-15T16:35:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.316+0000 7f975b91f080 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2021-01-15T16:35:02.402 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:02.402+0000 7f975b91f080 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2021-01-15T16:35:06.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2021-01-15T16:35:06.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:35:06.861 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:35:06.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:35:06.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:35:07.383 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:35:07.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:35:07.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:35:07.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:07.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:35:07.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:35:07.403 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:07.402+0000 7f9b36629080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:07.404 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:07.404+0000 7f9b36629080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:07.406 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:07.406+0000 7f9b36629080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:07.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:35:07.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:07.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:35:07.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:07.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:07.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:07.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:07.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:07.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:35:07.806 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:07.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:07.923+0000 7f9b36629080 -1 Falling back to public interface 2021-01-15T16:35:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:08.710 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:08.710+0000 7f9b36629080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:35:09.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:09.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:09.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:09.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:09.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:35:09.215 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:09.601 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85936,v1:127.0.0.1:6803/85936] [v2:127.0.0.1:6804/85936,v1:127.0.0.1:6805/85936] exists,up 9ac67ebe-ca64-4bd4-856c-257a1482c98d 2021-01-15T16:35:09.602 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 e4ce34be-017e-4434-970b-b4a9bab4d3e2 2021-01-15T16:35:09.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:09.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:09.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:09.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2021-01-15T16:35:09.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:09.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:09.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:35:09.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:09.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/1 2021-01-15T16:35:09.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:09.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:09.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:09.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:09.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-01-15T16:35:09.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-01-15T16:35:09.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:09.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:09.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:09.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:09.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:09.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:09.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:09.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:09.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:09.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:09.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/1 2021-01-15T16:35:09.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e4ce34be-017e-4434-970b-b4a9bab4d3e2 2021-01-15T16:35:09.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 e4ce34be-017e-4434-970b-b4a9bab4d3e2' 2021-01-15T16:35:09.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:09.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA9xAFg+X2UJBAABfwXG2HZBuJuTrnxyhvQlA== 2021-01-15T16:35:09.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA9xAFg+X2UJBAABfwXG2HZBuJuTrnxyhvQlA=="}' 2021-01-15T16:35:09.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e4ce34be-017e-4434-970b-b4a9bab4d3e2 -i td/ec-error-rollforward/1/new.json 2021-01-15T16:35:10.126 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:10.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/1/new.json 2021-01-15T16:35:10.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9xAFg+X2UJBAABfwXG2HZBuJuTrnxyhvQlA== --osd-uuid e4ce34be-017e-4434-970b-b4a9bab4d3e2 2021-01-15T16:35:10.162 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.161+0000 7f478ce89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:10.163 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.163+0000 7f478ce89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:10.165 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.165+0000 7f478ce89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:10.167 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.166+0000 7f478ce89080 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-01-15T16:35:10.167 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.166+0000 7f478ce89080 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-01-15T16:35:10.167 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.166+0000 7f478ce89080 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2021-01-15T16:35:10.225 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:10.225+0000 7f478ce89080 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2021-01-15T16:35:14.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2021-01-15T16:35:14.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:35:14.970 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:35:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:35:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:35:15.491 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:35:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:35:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:35:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:35:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:35:15.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:15.510+0000 7f3d4b3fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:15.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:15.514+0000 7f3d4b3fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:15.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:15.515+0000 7f3d4b3fc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:35:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:15.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:35:15.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:15.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:15.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:15.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:15.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:15.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:35:15.888 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:16.033 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:16.033+0000 7f3d4b3fc080 -1 Falling back to public interface 2021-01-15T16:35:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:16.817 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:16.816+0000 7f3d4b3fc080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:35:17.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:17.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:17.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:17.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:17.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:35:17.294 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:17.769 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/87200,v1:127.0.0.1:6811/87200] [v2:127.0.0.1:6812/87200,v1:127.0.0.1:6813/87200] exists,up e4ce34be-017e-4434-970b-b4a9bab4d3e2 2021-01-15T16:35:17.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:17.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:17.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2021-01-15T16:35:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:35:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/2 2021-01-15T16:35:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:17.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-01-15T16:35:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-01-15T16:35:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:17.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:17.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:17.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:17.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:17.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:17.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:17.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/2 2021-01-15T16:35:17.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:17.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=153190b9-fe76-4e5e-9a3b-8b6df45fefae 2021-01-15T16:35:17.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 153190b9-fe76-4e5e-9a3b-8b6df45fefae' 2021-01-15T16:35:17.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:17.779 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 153190b9-fe76-4e5e-9a3b-8b6df45fefae 2021-01-15T16:35:17.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBFxAFgr+jFLhAA+UIblHwjtIEXVX24gdNaRQ== 2021-01-15T16:35:17.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBFxAFgr+jFLhAA+UIblHwjtIEXVX24gdNaRQ=="}' 2021-01-15T16:35:17.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 153190b9-fe76-4e5e-9a3b-8b6df45fefae -i td/ec-error-rollforward/2/new.json 2021-01-15T16:35:18.245 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:35:18.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/2/new.json 2021-01-15T16:35:18.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBFxAFgr+jFLhAA+UIblHwjtIEXVX24gdNaRQ== --osd-uuid 153190b9-fe76-4e5e-9a3b-8b6df45fefae 2021-01-15T16:35:18.278 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.278+0000 7fbdd49c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.279+0000 7fbdd49c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.281+0000 7fbdd49c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:18.282 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.282+0000 7fbdd49c9080 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-01-15T16:35:18.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.282+0000 7fbdd49c9080 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-01-15T16:35:18.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.282+0000 7fbdd49c9080 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2021-01-15T16:35:18.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:18.329+0000 7fbdd49c9080 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2021-01-15T16:35:22.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2021-01-15T16:35:22.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:35:22.897 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:35:22.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:35:22.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:35:23.421 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:35:23.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:35:23.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:35:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:35:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:35:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:23.440+0000 7fbeab359080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:23.442 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:23.441+0000 7fbeab359080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:23.442 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:23.442+0000 7fbeab359080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:23.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:35:23.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:35:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:35:23.810 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:23.959 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:23.959+0000 7fbeab359080 -1 Falling back to public interface 2021-01-15T16:35:24.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:24.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:24.693+0000 7fbeab359080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:35:25.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:25.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:25.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:25.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:25.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:35:25.201 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:25.621 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/88464,v1:127.0.0.1:6819/88464] [v2:127.0.0.1:6820/88464,v1:127.0.0.1:6821/88464] exists,up 153190b9-fe76-4e5e-9a3b-8b6df45fefae 2021-01-15T16:35:25.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:25.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:25.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:25.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2021-01-15T16:35:25.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:35:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/ec-error-rollforward/3 2021-01-15T16:35:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:25.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:25.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:25.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:25.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-01-15T16:35:25.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-01-15T16:35:25.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:25.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:25.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:25.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:25.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:25.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:25.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:25.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:25.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:25.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:25.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:25.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:25.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:25.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:25.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:25.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:25.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/ec-error-rollforward/3 2021-01-15T16:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=19b9d9e5-e346-4871-b6a6-de443c09c0af 2021-01-15T16:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 19b9d9e5-e346-4871-b6a6-de443c09c0af' 2021-01-15T16:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:25.632 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 19b9d9e5-e346-4871-b6a6-de443c09c0af 2021-01-15T16:35:25.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBNxAFgNvDxJRAAWUED/gyc/xUphgBgwfHBoA== 2021-01-15T16:35:25.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBNxAFgNvDxJRAAWUED/gyc/xUphgBgwfHBoA=="}' 2021-01-15T16:35:25.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 19b9d9e5-e346-4871-b6a6-de443c09c0af -i td/ec-error-rollforward/3/new.json 2021-01-15T16:35:26.219 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:35:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/ec-error-rollforward/3/new.json 2021-01-15T16:35:26.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBNxAFgNvDxJRAAWUED/gyc/xUphgBgwfHBoA== --osd-uuid 19b9d9e5-e346-4871-b6a6-de443c09c0af 2021-01-15T16:35:26.252 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.251+0000 7ff74d260080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:26.253 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.253+0000 7ff74d260080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:26.254 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.254+0000 7ff74d260080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:26.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.254+0000 7ff74d260080 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-01-15T16:35:26.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.254+0000 7ff74d260080 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-01-15T16:35:26.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.254+0000 7ff74d260080 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2021-01-15T16:35:26.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:26.315+0000 7ff74d260080 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2021-01-15T16:35:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2021-01-15T16:35:30.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:35:30.568 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:35:30.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:35:30.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:35:31.088 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:35:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:35:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:35:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:35:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:35:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:31.106+0000 7f60c8ba5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:31.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:31.108+0000 7f60c8ba5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:31.109+0000 7f60c8ba5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:31.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:35:31.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:31.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:35:31.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:31.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:31.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:31.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:31.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:31.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:35:31.497 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:31.625 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:31.625+0000 7f60c8ba5080 -1 Falling back to public interface 2021-01-15T16:35:31.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:32.393 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:32.392+0000 7f60c8ba5080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:35:32.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:32.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:32.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:32.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:32.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:35:32.914 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:33.401 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/89728,v1:127.0.0.1:6827/89728] [v2:127.0.0.1:6828/89728,v1:127.0.0.1:6829/89728] exists,up 19b9d9e5-e346-4871-b6a6-de443c09c0af 2021-01-15T16:35:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2021-01-15T16:35:34.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2021-01-15T16:35:34.783 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'ec' already exists 2021-01-15T16:35:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2021-01-15T16:35:36.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2021-01-15T16:35:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 88464 2021-01-15T16:35:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=90280 2021-01-15T16:35:36.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2021-01-15T16:35:36.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2021-01-15T16:35:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 90301' 2021-01-15T16:35:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2021-01-15T16:35:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 90302' 2021-01-15T16:35:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2021-01-15T16:35:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 90303' 2021-01-15T16:35:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2021-01-15T16:35:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2021-01-15T16:35:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2021-01-15T16:35:38.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:38.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:38.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:38.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:38.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:38.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 90280 90301 90302 90303 2021-01-15T16:35:38.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2021-01-15T16:35:38.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2021-01-15T16:35:38.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:38.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:35:38.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:35:38.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:35:38.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/0 2021-01-15T16:35:38.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:38.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:38.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:38.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2021-01-15T16:35:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2021-01-15T16:35:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:35:38.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:38.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:35:38.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:38.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:38.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:38.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:38.448 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:38.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:38.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:38.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:35:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:35:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/0 2021-01-15T16:35:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:35:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:38.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/0/whoami 2021-01-15T16:35:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:35:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:35:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:35:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:35:38.453 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:35:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:38.461+0000 7ff979833080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:38.464 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:38.463+0000 7ff979833080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:38.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:38.465+0000 7ff979833080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:38.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:35:38.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:38.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:35:38.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:35:38.841 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:38.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:38.982+0000 7ff979833080 -1 Falling back to public interface 2021-01-15T16:35:39.230 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85936,v1:127.0.0.1:6803/85936] [v2:127.0.0.1:6804/85936,v1:127.0.0.1:6805/85936] exists,up 9ac67ebe-ca64-4bd4-856c-257a1482c98d 2021-01-15T16:35:39.231 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:35:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2021-01-15T16:35:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:35:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:35:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:35:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/1 2021-01-15T16:35:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2021-01-15T16:35:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2021-01-15T16:35:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:35:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:39.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:35:39.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:39.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:39.235 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:39.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:35:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:35:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/1 2021-01-15T16:35:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:35:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:39.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/1/whoami 2021-01-15T16:35:39.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:35:39.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:35:39.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:35:39.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:35:39.248 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:39.248+0000 7f90be396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:39.250 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:39.250+0000 7f90be396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:39.252 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:39.251+0000 7f90be396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:39.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:35:39.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:39.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:35:39.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:39.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:39.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:39.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:39.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:39.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:35:39.631 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:39.772+0000 7f90be396080 -1 Falling back to public interface 2021-01-15T16:35:40.024 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/87200,v1:127.0.0.1:6811/87200] [v2:127.0.0.1:6812/87200,v1:127.0.0.1:6813/87200] exists,up e4ce34be-017e-4434-970b-b4a9bab4d3e2 2021-01-15T16:35:40.024 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:35:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2021-01-15T16:35:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:35:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:35:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:35:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/2 2021-01-15T16:35:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2021-01-15T16:35:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2021-01-15T16:35:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:35:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:40.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:35:40.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:40.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:40.029 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:40.029 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:40.029 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:40.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:35:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:35:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/2 2021-01-15T16:35:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:35:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:40.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/2/whoami 2021-01-15T16:35:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:35:40.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:35:40.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:35:40.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:35:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.042+0000 7fa556768080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:40.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.058+0000 7fa556768080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:40.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.060+0000 7fa556768080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:40.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:35:40.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:35:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:40.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:40.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:35:40.421 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:40.478 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.478+0000 7ff979833080 -1 osd.0 26 log_to_monitors {default=true} 2021-01-15T16:35:40.580 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.579+0000 7fa556768080 -1 Falling back to public interface 2021-01-15T16:35:40.919 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/88464,v1:127.0.0.1:6819/88464] [v2:127.0.0.1:6820/88464,v1:127.0.0.1:6821/88464] exists,up 153190b9-fe76-4e5e-9a3b-8b6df45fefae 2021-01-15T16:35:40.919 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:35:40.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2021-01-15T16:35:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/ec-error-rollforward 2021-01-15T16:35:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:35:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2021-01-15T16:35:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:35:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/ec-error-rollforward/3 2021-01-15T16:35:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2021-01-15T16:35:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2021-01-15T16:35:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2021-01-15T16:35:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:35:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2021-01-15T16:35:40.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:35:40.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:40.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:40.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' 2021-01-15T16:35:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2021-01-15T16:35:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2021-01-15T16:35:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:35:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:35:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/ec-error-rollforward/3 2021-01-15T16:35:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2021-01-15T16:35:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=db5513f0-d8c5-49eb-8143-4f8a7367ce8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.84897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:40.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/ec-error-rollforward/3/whoami 2021-01-15T16:35:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2021-01-15T16:35:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:35:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:35:40.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:35:40.939 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.939+0000 7f9c73dd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:40.941 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.941+0000 7f9c73dd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:40.942 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:40.941+0000 7f9c73dd3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:41.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:41.458+0000 7f9c73dd3080 -1 Falling back to public interface 2021-01-15T16:35:41.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2021-01-15T16:35:41.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:41.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:35:41.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:41.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:41.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:41.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:41.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:41.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:35:41.497 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:41.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:41.678+0000 7f90be396080 -1 osd.1 26 log_to_monitors {default=true} 2021-01-15T16:35:41.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:42.732 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:42.732+0000 7fa556768080 -1 osd.2 25 log_to_monitors {default=true} 2021-01-15T16:35:42.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:42.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:42.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:42.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:42.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:35:42.901 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:43.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:43.604 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:43.603+0000 7f9c73dd3080 -1 osd.3 26 log_to_monitors {default=true} 2021-01-15T16:35:44.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:44.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:44.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T16:35:44.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:44.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:35:44.495 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:35:44.890 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 37 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/91390,v1:127.0.0.1:6827/91390] [v2:127.0.0.1:6828/91390,v1:127.0.0.1:6829/91390] exists,up 19b9d9e5-e346-4871-b6a6-de443c09c0af 2021-01-15T16:35:44.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:44.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:44.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:44.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2021-01-15T16:35:44.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:35:44.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:35:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:35:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:35:44.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:35:44.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:35:44.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:35:44.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:35:44.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:35:44.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:35:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:35:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:35:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:35:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:35:44.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:35:45.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:35:45.367 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:35:45.367 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:35:45.367 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T16:35:45.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:35:45.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:35:45.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:35:45.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084292 2021-01-15T16:35:45.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084292 2021-01-15T16:35:45.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084292' 2021-01-15T16:35:45.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:35:45.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:35:45.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986179 2021-01-15T16:35:45.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986179 2021-01-15T16:35:45.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084292 1-133143986179' 2021-01-15T16:35:45.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:35:45.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:35:45.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2021-01-15T16:35:45.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2021-01-15T16:35:45.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-150323855363' 2021-01-15T16:35:45.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:35:45.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:35:45.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=158913789955 2021-01-15T16:35:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 158913789955 2021-01-15T16:35:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084292 1-133143986179 2-150323855363 3-158913789955' 2021-01-15T16:35:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:35:45.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-120259084292 2021-01-15T16:35:45.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:35:45.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:35:45.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-120259084292 2021-01-15T16:35:45.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:35:45.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084292 2021-01-15T16:35:45.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2021-01-15T16:35:45.908 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 120259084292 2021-01-15T16:35:45.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:35:46.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084292 -lt 120259084292 2021-01-15T16:35:46.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:35:46.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-133143986179 2021-01-15T16:35:46.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:35:46.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:35:46.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-133143986179 2021-01-15T16:35:46.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:35:46.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986179 2021-01-15T16:35:46.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2021-01-15T16:35:46.310 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 133143986179 2021-01-15T16:35:46.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:35:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986179 -lt 133143986179 2021-01-15T16:35:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:35:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-150323855363 2021-01-15T16:35:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:35:46.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:35:46.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-150323855363 2021-01-15T16:35:46.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:35:46.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2021-01-15T16:35:46.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2021-01-15T16:35:46.712 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 150323855363 2021-01-15T16:35:46.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:35:47.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 150323855363 2021-01-15T16:35:47.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:35:48.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:35:48.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:35:48.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855363 -lt 150323855363 2021-01-15T16:35:48.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:35:48.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-158913789955 2021-01-15T16:35:48.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:35:48.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:35:48.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-158913789955 2021-01-15T16:35:48.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:35:48.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=158913789955 2021-01-15T16:35:48.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 158913789955' 2021-01-15T16:35:48.496 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 158913789955 2021-01-15T16:35:48.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:35:48.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 158913789955 -lt 158913789955 2021-01-15T16:35:48.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:35:48.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:35:48.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:35:49.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:35:49.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:35:49.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:35:49.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:35:49.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:35:49.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:35:49.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:35:49.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:35:49.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:35:49.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:35:49.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:35:49.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:35:50.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:35:50.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:35:50.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:35:50.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2021-01-15T16:35:50.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-01-15T16:35:50.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:35:50.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-01-15T16:35:50.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:50.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:50.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:50.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:50.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:50.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:50.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:50.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:50.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:50.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:50.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:50.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:50.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:50.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-01-15T16:35:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:50.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84897 2021-01-15T16:35:50.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:50.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:50.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T16:35:50.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/ec-error-rollforward 0 2021-01-15T16:35:50.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2021-01-15T16:35:50.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T16:35:50.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2021-01-15T16:35:50.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:50.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:50.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:50.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:50.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:50.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:50.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:50.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:50.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:50.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:50.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:50.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:50.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T16:35:50.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2021-01-15T16:35:50.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:50.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.84897 2021-01-15T16:35:50.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.84897 2021-01-15T16:35:50.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:50.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:50.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T16:35:50.415 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T16:35:50.415 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T16:35:50.479 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2021-01-15T16:35:50.479 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2021-01-15T16:35:50.539 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:35:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:35:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:35:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:35:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:35:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:35:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:35:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:35:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:35:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:35:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-prio 2021-01-15T16:35:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2021-01-15T16:35:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2021-01-15T16:35:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-01-15T16:35:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-01-15T16:35:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2021-01-15T16:35:50.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2021-01-15T16:35:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none ' 2021-01-15T16:35:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:35:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:35:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2021-01-15T16:35:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2021-01-15T16:35:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2021-01-15T16:35:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2021-01-15T16:35:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2021-01-15T16:35:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2021-01-15T16:35:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2021-01-15T16:35:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2021-01-15T16:35:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2021-01-15T16:35:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2021-01-15T16:35:50.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2021-01-15T16:35:50.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:35:50.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2021-01-15T16:35:50.557 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_priority' 2021-01-15T16:35:50.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2021-01-15T16:35:50.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2021-01-15T16:35:50.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2021-01-15T16:35:50.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2021-01-15T16:35:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-01-15T16:35:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:35:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-01-15T16:35:50.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:35:50.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:35:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:35:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:35:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:35:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:35:50.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:35:50.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:35:50.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:35:50.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:35:50.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:35:50.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:35:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:35:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:35:50.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:35:50.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:35:50.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:35:50.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-01-15T16:35:50.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:35:50.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:50.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92827 2021-01-15T16:35:50.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:35:50.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:35:50.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2021-01-15T16:35:50.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:35:50.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:50.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92827 2021-01-15T16:35:50.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:35:50.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:35:50.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:35:50.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:35:50.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2021-01-15T16:35:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2021-01-15T16:35:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2021-01-15T16:35:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2021-01-15T16:35:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2021-01-15T16:35:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2021-01-15T16:35:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2021-01-15T16:35:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:35:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:35:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:35:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2021-01-15T16:35:50.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2021-01-15T16:35:50.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:35:50.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:50.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:50.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:50.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:50.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:50.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:35:50.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:35:50.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:35:50.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:35:50.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:35:50.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:35:50.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:35:50.952 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:35:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:35:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:35:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:35:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:50.954 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-mon.a.asok 2021-01-15T16:35:50.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:35:50.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.92827/ceph-mon.a.asok config get fsid 2021-01-15T16:35:51.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:35:51.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:35:51.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:35:51.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:35:51.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:35:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:35:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:35:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:35:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:35:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-mon.a.asok 2021-01-15T16:35:51.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:35:51.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.92827/ceph-mon.a.asok config get mon_host 2021-01-15T16:35:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2021-01-15T16:35:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-prio 2021-01-15T16:35:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:35:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:35:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:35:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-prio/x 2021-01-15T16:35:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:35:51.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:35:51.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:51.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:51.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:51.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:51.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:51.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:51.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:35:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:35:51.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2021-01-15T16:35:51.677 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2021-01-15T16:35:51.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2021-01-15T16:35:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:35:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2021-01-15T16:35:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:35:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:35:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/0 2021-01-15T16:35:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:35:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:51.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:51.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:51.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2021-01-15T16:35:51.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2021-01-15T16:35:51.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:51.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:51.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:35:51.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:51.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:51.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:51.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:51.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:51.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:51.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:35:51.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:35:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:51.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:51.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/0 2021-01-15T16:35:51.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:51.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=15be9022-166a-41c5-a332-8b09cab38a51 2021-01-15T16:35:51.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 15be9022-166a-41c5-a332-8b09cab38a51' 2021-01-15T16:35:51.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:51.690 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 15be9022-166a-41c5-a332-8b09cab38a51 2021-01-15T16:35:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBnxAFgGHClKRAAXa7xq7jIBOv+mEV7yUI3fQ== 2021-01-15T16:35:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBnxAFgGHClKRAAXa7xq7jIBOv+mEV7yUI3fQ=="}' 2021-01-15T16:35:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 15be9022-166a-41c5-a332-8b09cab38a51 -i td/osd-backfill-prio/0/new.json 2021-01-15T16:35:52.043 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:52.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/0/new.json 2021-01-15T16:35:52.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBnxAFgGHClKRAAXa7xq7jIBOv+mEV7yUI3fQ== --osd-uuid 15be9022-166a-41c5-a332-8b09cab38a51 2021-01-15T16:35:52.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.081+0000 7f981b5c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:52.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.083+0000 7f981b5c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:52.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.084+0000 7f981b5c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:52.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.085+0000 7f981b5c2080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:35:52.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.085+0000 7f981b5c2080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:35:52.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.085+0000 7f981b5c2080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:35:52.141 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:52.140+0000 7f981b5c2080 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2021-01-15T16:35:56.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2021-01-15T16:35:56.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:35:56.614 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:35:56.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:35:56.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:35:57.145 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:35:57.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:35:57.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:35:57.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:57.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:35:57.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:35:57.170 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:57.170+0000 7ffa86640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:57.190 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:57.189+0000 7ffa86640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:57.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:57.190+0000 7ffa86640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:57.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:35:57.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:35:57.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:35:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:35:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:35:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:35:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:57.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:35:57.551 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:35:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:57.707+0000 7ffa86640080 -1 Falling back to public interface 2021-01-15T16:35:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:35:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:58.490+0000 7ffa86640080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:35:58.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:35:58.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:35:58.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:35:58.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:35:58.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:35:58.947 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:59.362 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/93869,v1:127.0.0.1:6803/93869] [v2:127.0.0.1:6804/93869,v1:127.0.0.1:6805/93869] exists,up 15be9022-166a-41c5-a332-8b09cab38a51 2021-01-15T16:35:59.362 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 01977618-54b1-4a5d-8901-44e062f34c8f 2021-01-15T16:35:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:35:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:35:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:35:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:35:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2021-01-15T16:35:59.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:35:59.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:35:59.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:35:59.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:35:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/1 2021-01-15T16:35:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:35:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:35:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:35:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:35:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2021-01-15T16:35:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2021-01-15T16:35:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:35:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:35:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:35:59.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:35:59.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:35:59.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:35:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:35:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:35:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:35:59.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:35:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:35:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:35:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:35:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:35:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:35:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:35:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:35:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:35:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:35:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:35:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/1 2021-01-15T16:35:59.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:35:59.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=01977618-54b1-4a5d-8901-44e062f34c8f 2021-01-15T16:35:59.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 01977618-54b1-4a5d-8901-44e062f34c8f' 2021-01-15T16:35:59.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:35:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBvxAFgYA+PFhAAZGAvXUIG2sbyyMBBVKGqCg== 2021-01-15T16:35:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBvxAFgYA+PFhAAZGAvXUIG2sbyyMBBVKGqCg=="}' 2021-01-15T16:35:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 01977618-54b1-4a5d-8901-44e062f34c8f -i td/osd-backfill-prio/1/new.json 2021-01-15T16:35:59.875 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:35:59.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/1/new.json 2021-01-15T16:35:59.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvxAFgYA+PFhAAZGAvXUIG2sbyyMBBVKGqCg== --osd-uuid 01977618-54b1-4a5d-8901-44e062f34c8f 2021-01-15T16:35:59.914 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.914+0000 7f21dfc02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:59.916 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.916+0000 7f21dfc02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:59.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.918+0000 7f21dfc02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:35:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.918+0000 7f21dfc02080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:35:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.918+0000 7f21dfc02080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:35:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:35:59.918+0000 7f21dfc02080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:36:00.013 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:00.013+0000 7f21dfc02080 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2021-01-15T16:36:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2021-01-15T16:36:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:36:04.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:36:04.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:36:04.614 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:36:05.142 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:36:05.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:36:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:36:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:36:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:36:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:36:05.167 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:05.166+0000 7f2871783080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:36:05.168 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:05.168+0000 7f2871783080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:36:05.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:05.169+0000 7f2871783080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:36:05.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:36:05.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:36:05.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:36:05.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:36:05.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:36:05.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:36:05.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:36:05.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:36:05.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:36:05.560 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:36:05.690 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:05.690+0000 7f2871783080 -1 Falling back to public interface 2021-01-15T16:36:05.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:36:06.463 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:36:06.462+0000 7f2871783080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:36:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:36:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:36:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:36:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:36:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:36:06.968 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:36:07.364 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/95133,v1:127.0.0.1:6811/95133] [v2:127.0.0.1:6812/95133,v1:127.0.0.1:6813/95133] exists,up 01977618-54b1-4a5d-8901-44e062f34c8f 2021-01-15T16:36:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:36:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:36:07.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:36:07.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2021-01-15T16:36:07.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:07.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2021-01-15T16:36:07.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T16:36:07.975 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T16:36:07.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:36:08.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2021-01-15T16:36:09.695 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T16:36:09.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:09.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2021-01-15T16:36:09.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T16:36:10.320 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T16:36:10.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:36:11.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2021-01-15T16:36:12.154 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T16:36:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2021-01-15T16:36:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T16:36:12.806 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T16:36:12.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:36:13.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2021-01-15T16:36:14.630 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T16:36:14.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2021-01-15T16:36:19.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2021-01-15T16:36:19.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:36:19.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:36:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:36:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:36:19.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:36:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:36:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:36:19.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:36:19.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:36:19.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:36:19.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:36:19.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:36:19.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:36:19.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:36:19.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:36:20.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:36:20.141 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T16:36:20.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:36:20.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:36:20.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:36:20.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:36:20.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:36:20.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:36:20.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:36:20.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:36:20.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672966 2021-01-15T16:36:20.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672966 2021-01-15T16:36:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-01-15T16:36:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:36:20.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:36:20.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:36:20.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:36:20.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:36:20.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:36:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:36:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:36:20.440 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:36:20.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:36:20.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836487 2021-01-15T16:36:20.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:36:21.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:36:21.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:36:22.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T16:36:22.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:36:22.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2021-01-15T16:36:22.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:36:22.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:36:22.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-01-15T16:36:22.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:36:22.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672966 2021-01-15T16:36:22.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-01-15T16:36:22.244 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672966 2021-01-15T16:36:22.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:36:22.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672966 2021-01-15T16:36:22.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:36:22.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:36:22.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:36:23.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T16:36:23.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:36:23.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:36:23.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:36:23.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:36:23.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:36:23.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:36:23.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:36:23.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T16:36:23.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:36:23.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:36:23.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:36:24.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T16:36:24.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:36:24.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:36:24.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2021-01-15T16:36:24.461 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:36:24.461 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:36:15.612913+0000 0'0 28:16 [1,0] 1 [1,0] 1 0'0 2021-01-15T16:36:12.519504+0000 0'0 2021-01-15T16:36:12.519504+0000 0 2021-01-15T16:36:24.462 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:36:10.010674+0000 0'0 27:27 [1,0] 1 [1,0] 1 0'0 2021-01-15T16:36:07.777334+0000 0'0 2021-01-15T16:36:07.777334+0000 0 2021-01-15T16:36:24.462 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:36:12.695544+0000 0'0 27:23 [0,1] 0 [0,1] 0 0'0 2021-01-15T16:36:10.068161+0000 0'0 2021-01-15T16:36:10.068161+0000 0 2021-01-15T16:36:24.462 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:36:24.462 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:36:24.463 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:36:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2021-01-15T16:36:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2021-01-15T16:36:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2021-01-15T16:36:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2021-01-15T16:36:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2021-01-15T16:36:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2021-01-15T16:36:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2021-01-15T16:36:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2021-01-15T16:36:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2021-01-15T16:36:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2021-01-15T16:36:24.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2021-01-15T16:36:24.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:24.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2021-01-15T16:36:24.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2021-01-15T16:36:24.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:36:24.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2021-01-15T16:36:24.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:36:24.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2021-01-15T16:36:24.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2021-01-15T16:36:24.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2021-01-15T16:36:24.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2021-01-15T16:36:24.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2021-01-15T16:36:24.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2021-01-15T16:36:24.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2021-01-15T16:36:24.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:24.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2021-01-15T16:36:24.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2021-01-15T16:36:25.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:36:25.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2021-01-15T16:36:25.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:36:25.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2021-01-15T16:36:25.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2021-01-15T16:36:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2021-01-15T16:36:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2021-01-15T16:36:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2021-01-15T16:36:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2021-01-15T16:36:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2021-01-15T16:36:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2021-01-15T16:36:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2021-01-15T16:36:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2021-01-15T16:36:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2021-01-15T16:36:25.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2021-01-15T16:36:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-01-15T16:36:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-01-15T16:36:25.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-01-15T16:36:25.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-01-15T16:36:25.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2021-01-15T16:36:25.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T16:36:25.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T16:36:25.798 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T16:36:25.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2021-01-15T16:36:25.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2021-01-15T16:36:25.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2021-01-15T16:36:25.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2021-01-15T16:36:25.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2021-01-15T16:36:25.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2021-01-15T16:36:25.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T16:36:26.592 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T16:36:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-01-15T16:36:27.261 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 recovery_priority to 1 2021-01-15T16:36:27.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-01-15T16:36:28.005 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 1 2021-01-15T16:36:28.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-01-15T16:36:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 recovery_priority to 2 2021-01-15T16:36:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2021-01-15T16:36:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:36:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:36:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:36:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:36:28.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:36:28.703 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:36:28.703 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:36:28.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:36:28.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:36:28.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:36:28.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:36:28.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:36:28.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:36:28.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:36:28.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:36:29.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:36:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T16:36:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:36:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:36:29.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:36:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T16:36:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T16:36:29.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T16:36:29.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:36:29.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:36:29.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T16:36:29.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T16:36:29.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672970' 2021-01-15T16:36:29.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:36:29.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T16:36:29.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:36:29.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:36:29.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T16:36:29.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:36:29.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T16:36:29.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T16:36:29.476 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T16:36:29.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:36:29.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2021-01-15T16:36:29.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:36:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T16:36:29.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:36:29.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:36:29.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T16:36:29.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:36:29.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T16:36:29.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T16:36:29.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:36:29.949 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T16:36:30.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672970 2021-01-15T16:36:30.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:36:31.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:36:31.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:36:31.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T16:36:31.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:36:31.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:36:31.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:36:32.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T16:36:32.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:36:32.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:36:32.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:36:32.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:36:32.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:36:32.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:36:32.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:36:32.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:36:32.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:36:32.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:36:32.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:36:33.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T16:36:33.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:36:33.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:36:33.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2021-01-15T16:36:33.152 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2021-01-15T16:36:33.152 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2021-01-15T16:36:33.152 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0456171 s, 230 MB/s 2021-01-15T16:36:33.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2021-01-15T16:36:33.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2021-01-15T16:36:33.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2021-01-15T16:36:33.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:33.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2021-01-15T16:36:33.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:33.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2021-01-15T16:36:33.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:33.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2021-01-15T16:36:34.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:34.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2021-01-15T16:36:34.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:34.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2021-01-15T16:36:34.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:34.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2021-01-15T16:36:35.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:35.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2021-01-15T16:36:35.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:35.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2021-01-15T16:36:35.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:35.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2021-01-15T16:36:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2021-01-15T16:36:37.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:37.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2021-01-15T16:36:37.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:37.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2021-01-15T16:36:37.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:37.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2021-01-15T16:36:37.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:37.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2021-01-15T16:36:38.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:38.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2021-01-15T16:36:38.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:38.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2021-01-15T16:36:38.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:38.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2021-01-15T16:36:39.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:39.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2021-01-15T16:36:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:40.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2021-01-15T16:36:40.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:40.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2021-01-15T16:36:40.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:40.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2021-01-15T16:36:41.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:41.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2021-01-15T16:36:41.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:41.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2021-01-15T16:36:41.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:41.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2021-01-15T16:36:42.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:42.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2021-01-15T16:36:42.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:42.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2021-01-15T16:36:43.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:43.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2021-01-15T16:36:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:44.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2021-01-15T16:36:44.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:44.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2021-01-15T16:36:44.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:44.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2021-01-15T16:36:45.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:45.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2021-01-15T16:36:45.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:45.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2021-01-15T16:36:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:45.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2021-01-15T16:36:46.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:46.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2021-01-15T16:36:46.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:46.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2021-01-15T16:36:47.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:47.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2021-01-15T16:36:47.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:47.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2021-01-15T16:36:47.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:47.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2021-01-15T16:36:48.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:48.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2021-01-15T16:36:48.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:48.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2021-01-15T16:36:48.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:48.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2021-01-15T16:36:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2021-01-15T16:36:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2021-01-15T16:36:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2021-01-15T16:36:50.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:50.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2021-01-15T16:36:51.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:51.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2021-01-15T16:36:51.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2021-01-15T16:36:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2021-01-15T16:36:52.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:52.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2021-01-15T16:36:52.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:52.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2021-01-15T16:36:53.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2021-01-15T16:36:53.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2021-01-15T16:36:53.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2021-01-15T16:36:53.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2021-01-15T16:36:53.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:53.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2021-01-15T16:36:53.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:53.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2021-01-15T16:36:54.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:54.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2021-01-15T16:36:54.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:54.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2021-01-15T16:36:55.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:55.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2021-01-15T16:36:55.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:55.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2021-01-15T16:36:56.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:56.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2021-01-15T16:36:56.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:56.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2021-01-15T16:36:56.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:56.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2021-01-15T16:36:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2021-01-15T16:36:57.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:57.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2021-01-15T16:36:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2021-01-15T16:36:58.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:58.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2021-01-15T16:36:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2021-01-15T16:36:59.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:59.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2021-01-15T16:36:59.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:36:59.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2021-01-15T16:37:00.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:00.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2021-01-15T16:37:00.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:00.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2021-01-15T16:37:00.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:00.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2021-01-15T16:37:01.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:01.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2021-01-15T16:37:01.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:01.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2021-01-15T16:37:01.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:01.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2021-01-15T16:37:02.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:02.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2021-01-15T16:37:02.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:02.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2021-01-15T16:37:03.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:03.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2021-01-15T16:37:03.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:03.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2021-01-15T16:37:04.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2021-01-15T16:37:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:04.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2021-01-15T16:37:04.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:04.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2021-01-15T16:37:05.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:05.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2021-01-15T16:37:06.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:06.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2021-01-15T16:37:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2021-01-15T16:37:06.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:06.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2021-01-15T16:37:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2021-01-15T16:37:07.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:07.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2021-01-15T16:37:07.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:07.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2021-01-15T16:37:08.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:08.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2021-01-15T16:37:08.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:08.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2021-01-15T16:37:08.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:08.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2021-01-15T16:37:09.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:09.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2021-01-15T16:37:09.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:09.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2021-01-15T16:37:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2021-01-15T16:37:10.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:10.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2021-01-15T16:37:10.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:10.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2021-01-15T16:37:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2021-01-15T16:37:11.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:11.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2021-01-15T16:37:11.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:11.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2021-01-15T16:37:13.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:13.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2021-01-15T16:37:14.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:14.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2021-01-15T16:37:14.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-01-15T16:37:14.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2021-01-15T16:37:14.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2021-01-15T16:37:14.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2021-01-15T16:37:14.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2021-01-15T16:37:14.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2021-01-15T16:37:14.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2021-01-15T16:37:14.656 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2021-01-15T16:37:14.656 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2021-01-15T16:37:14.656 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2021-01-15T16:37:14.657 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T16:37:14.657 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T16:37:15.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T16:37:15.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T16:37:15.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T16:37:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2021-01-15T16:37:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2021-01-15T16:37:15.835 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:37:15.835 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:36:26.748032+0000 37'150 37:191 [1] 1 [1] 1 0'0 2021-01-15T16:36:07.777334+0000 0'0 2021-01-15T16:36:07.777334+0000 0 2021-01-15T16:37:15.835 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:36:28.170023+0000 37'150 37:187 [0] 0 [0] 0 0'0 2021-01-15T16:36:10.068161+0000 0'0 2021-01-15T16:36:10.068161+0000 0 2021-01-15T16:37:15.836 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:37:15.836 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:37:15.837 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:37:15.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2021-01-15T16:37:15.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2021-01-15T16:37:16.514 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T16:37:16.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2021-01-15T16:37:17.269 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T16:37:17.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2021-01-15T16:37:22.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2021-01-15T16:37:22.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:37:22.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:37:22.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:37:22.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:37:22.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:37:22.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:37:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2021-01-15T16:37:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:37:22.405 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 2021-01-15T16:37:22.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2021-01-15T16:37:22.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-01-15T16:37:22.408 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:37:22.408 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:37:22.408 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:37:22.409 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:37:22.409 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:37:22.409 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:37:22.410 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:37:22.410 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:37:22.410 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 152, 2021-01-15T16:37:22.411 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:37:22.411 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.411 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:37:22.411 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:37:22.412 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:37:22.412 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:37:22.412 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:37:22.412 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:37:22.412 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:37:22.413 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:37:22.413 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2021-01-15T16:37:22.413 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 153, 2021-01-15T16:37:22.413 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:37:22.414 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.414 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:37:22.414 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.414 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:37:22.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2021-01-15T16:37:22.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T16:37:22.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T16:37:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:37:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:37:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:37:22.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.0.asok 2021-01-15T16:37:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2021-01-15T16:37:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.0.asok dump_recovery_reservations 2021-01-15T16:37:22.516 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 2021-01-15T16:37:22.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2021-01-15T16:37:22.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-01-15T16:37:22.517 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:37:22.517 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:37:22.518 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:37:22.518 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:37:22.518 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:37:22.518 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:37:22.518 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:37:22.519 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2021-01-15T16:37:22.519 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 153, 2021-01-15T16:37:22.519 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:37:22.519 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.519 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:37:22.520 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:37:22.520 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:37:22.520 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:37:22.520 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:37:22.520 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:37:22.521 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:37:22.521 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:37:22.521 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:37:22.521 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 152, 2021-01-15T16:37:22.521 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:37:22.522 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.522 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:37:22.522 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:37:22.522 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:37:22.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-01-15T16:37:22.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T16:37:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2021-01-15T16:37:22.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2021-01-15T16:37:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-01-15T16:37:22.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-01-15T16:37:22.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T16:37:22.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2021-01-15T16:37:22.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2021-01-15T16:37:22.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-01-15T16:37:22.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-01-15T16:37:22.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2021-01-15T16:37:22.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2021-01-15T16:37:22.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-01-15T16:37:22.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-01-15T16:37:22.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-01-15T16:37:22.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2021-01-15T16:37:22.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2021-01-15T16:37:22.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-01-15T16:37:22.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T16:37:22.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2021-01-15T16:37:22.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2021-01-15T16:37:22.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-01-15T16:37:22.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-01-15T16:37:22.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T16:37:22.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2021-01-15T16:37:22.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2021-01-15T16:37:22.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-01-15T16:37:22.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-01-15T16:37:22.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2021-01-15T16:37:22.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2021-01-15T16:37:22.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-01-15T16:37:22.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-01-15T16:37:22.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-01-15T16:37:22.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2021-01-15T16:37:22.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2021-01-15T16:37:22.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2021-01-15T16:37:22.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:37:22.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:37:22.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:37:22.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:37:22.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:37:22.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:37:22.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:37:22.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:37:22.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:37:22.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:37:22.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:37:22.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:37:22.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:37:22.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:37:22.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:37:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:37:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T16:37:23.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:37:23.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:37:23.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:37:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836503 2021-01-15T16:37:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836503 2021-01-15T16:37:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503' 2021-01-15T16:37:23.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:37:23.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:37:23.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672983 2021-01-15T16:37:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672983 2021-01-15T16:37:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672983' 2021-01-15T16:37:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:37:23.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836503 2021-01-15T16:37:23.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:37:23.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:37:23.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836503 2021-01-15T16:37:23.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:37:23.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836503 2021-01-15T16:37:23.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2021-01-15T16:37:23.403 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836503 2021-01-15T16:37:23.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:37:23.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836503 2021-01-15T16:37:23.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:37:24.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672983 2021-01-15T16:37:24.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:37:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:37:24.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672983 2021-01-15T16:37:24.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:37:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672983 2021-01-15T16:37:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2021-01-15T16:37:24.003 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672983 2021-01-15T16:37:24.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:37:24.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672983 -lt 42949672983 2021-01-15T16:37:24.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:37:24.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:24.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:25.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T16:37:25.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:25.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:25.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:25.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:25.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:25.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:25.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:25.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:25.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:25.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:25.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:37:26.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:37:26.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:37:26.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:26.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:26.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:26.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:26.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:26.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:26.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:26.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:26.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:26.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:26.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:26.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:27.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:27.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:27.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:27.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:27.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:27.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:28.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15825700 2021-01-15T16:37:28.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15825700 '!=' null 2021-01-15T16:37:28.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:28.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:28.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:28.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:28.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:28.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:28.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:28.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:28.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:28.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:28.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:29.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:29.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:29.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:29.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:29.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:29.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:29.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:29.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:30.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17587016 2021-01-15T16:37:30.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17587016 '!=' null 2021-01-15T16:37:30.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:30.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:30.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:30.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:30.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:30.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:30.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:30.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:30.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:30.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:30.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:30.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:31.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:31.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:31.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:31.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:31.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:31.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:31.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:31.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:31.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967527 2021-01-15T16:37:32.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967527 '!=' null 2021-01-15T16:37:32.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:32.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:32.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:32.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:32.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:32.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:32.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:32.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:32.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:32.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:32.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:32.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:33.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:33.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:33.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:33.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:33.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:33.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:33.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:33.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:33.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:34.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336025 2021-01-15T16:37:34.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336025 '!=' null 2021-01-15T16:37:34.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:34.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:34.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:34.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:34.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:34.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:34.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:34.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:34.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:34.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:34.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:34.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:34.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:34.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:35.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:35.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:35.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:35.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:35.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:35.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:35.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:35.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:35.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:36.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094297 2021-01-15T16:37:36.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094297 '!=' null 2021-01-15T16:37:36.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:36.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:36.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:36.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:36.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:36.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:36.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:36.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:36.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:36.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:36.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:36.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:36.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:36.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:37.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:37.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:38.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094297 2021-01-15T16:37:38.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094297 '!=' null 2021-01-15T16:37:38.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:38.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:38.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:38.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:38.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:38.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:38.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:38.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:38.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:38.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:38.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:39.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:39.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:39.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:39.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:39.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:39.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:39.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:39.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:39.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:39.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:39.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:39.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:39.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:40.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967532 2021-01-15T16:37:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967532 '!=' null 2021-01-15T16:37:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:40.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:40.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:40.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:40.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:40.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:40.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:40.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:40.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:40.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:40.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:40.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:40.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:41.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:41.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:41.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:41.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:42.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093930 2021-01-15T16:37:42.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093930 '!=' null 2021-01-15T16:37:42.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:42.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:42.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:42.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:42.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:42.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:42.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:42.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:42.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:42.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:42.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:42.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:42.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:42.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:42.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:43.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:43.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:43.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:43.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:43.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:43.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:43.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:43.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:43.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:43.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093930 2021-01-15T16:37:43.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093930 '!=' null 2021-01-15T16:37:43.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:43.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:43.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:43.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:43.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:43.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:43.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:43.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:43.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:43.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:43.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:44.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:44.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:44.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:44.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:44.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:44.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:44.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:44.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:44.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:44.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:44.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:44.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:44.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:45.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093857 2021-01-15T16:37:45.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093857 '!=' null 2021-01-15T16:37:45.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:45.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:45.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:45.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:45.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:45.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:45.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:45.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:45.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:45.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:45.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:45.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:45.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:45.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:45.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:46.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:46.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:46.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:46.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:46.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:46.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:46.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:46.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:46.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:47.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093895 2021-01-15T16:37:47.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093895 '!=' null 2021-01-15T16:37:47.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:47.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:47.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:47.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:47.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:47.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:47.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:47.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:47.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:48.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:48.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:48.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:48.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:48.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:48.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:48.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:48.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:48.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:49.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852133 2021-01-15T16:37:49.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852133 '!=' null 2021-01-15T16:37:49.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:49.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:49.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:49.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:49.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:49.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:49.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:49.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:49.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:49.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:49.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:50.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:50.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:50.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:50.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:50.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:50.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:51.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094152 2021-01-15T16:37:51.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094152 '!=' null 2021-01-15T16:37:51.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:51.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:51.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:51.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:51.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:51.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:51.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:51.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:52.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:52.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:52.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:52.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:52.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:52.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:52.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:52.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:52.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841660 2021-01-15T16:37:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841660 '!=' null 2021-01-15T16:37:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:53.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:53.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:53.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:53.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:53.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:53.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:53.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:53.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:53.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:53.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:54.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:54.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:54.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:54.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:54.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:54.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:54.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:54.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:54.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:54.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588855 2021-01-15T16:37:54.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588855 '!=' null 2021-01-15T16:37:54.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:54.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:54.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:54.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:54.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:54.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:54.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:54.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:54.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:54.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:54.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:55.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:55.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:55.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:55.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:55.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:55.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:55.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:55.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:56.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462599 2021-01-15T16:37:56.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462599 '!=' null 2021-01-15T16:37:56.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:56.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:56.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:56.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:56.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:56.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:56.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:57.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:57.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:57.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:57.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:57.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:57.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:57.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:57.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:57.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:57.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:57.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:57.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:57.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:37:58.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220576 2021-01-15T16:37:58.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220576 '!=' null 2021-01-15T16:37:58.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:37:58.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:37:58.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:37:58.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:37:58.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:37:58.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:37:58.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:37:58.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:37:58.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:37:58.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:37:58.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:37:59.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:37:59.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:37:59.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:37:59.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:37:59.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:37:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:37:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:37:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:37:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:37:59.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:37:59.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:37:59.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:37:59.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:00.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462486 2021-01-15T16:38:00.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462486 '!=' null 2021-01-15T16:38:00.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:00.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:00.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:00.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:00.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:00.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:00.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:00.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:00.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:00.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:00.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:01.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:01.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:01.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:01.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:01.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:01.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:01.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:02.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588851 2021-01-15T16:38:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588851 '!=' null 2021-01-15T16:38:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:02.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:02.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:02.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:02.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:03.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:03.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:03.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:03.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:03.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:03.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:03.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:03.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:03.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:03.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:03.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:03.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:03.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:04.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715416 2021-01-15T16:38:04.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715416 '!=' null 2021-01-15T16:38:04.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:04.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:04.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:04.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:04.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:04.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:04.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:04.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:04.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:04.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:04.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:05.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:05.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:05.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:05.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:05.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:05.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:05.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:05.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:05.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:06.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20968133 2021-01-15T16:38:06.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20968133 '!=' null 2021-01-15T16:38:06.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:06.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:06.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:06.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:06.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:06.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:06.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:06.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:06.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:06.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:06.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:06.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:06.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:06.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:06.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:07.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:07.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:07.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:07.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:07.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:07.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:07.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:07.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:07.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:08.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18871251 2021-01-15T16:38:08.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18871251 '!=' null 2021-01-15T16:38:08.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:08.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:08.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:08.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:08.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:08.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:08.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:08.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:08.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:08.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:08.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:08.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:09.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7862637 2021-01-15T16:38:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7862637 '!=' null 2021-01-15T16:38:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:09.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:09.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:09.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:09.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:09.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:09.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:09.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:09.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:09.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:09.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:10.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:38:10.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:10.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:10.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T16:38:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:38:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:10.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:10.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:10.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:10.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:10.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7862637 2021-01-15T16:38:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7862637 '!=' null 2021-01-15T16:38:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:11.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:11.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:11.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:11.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:11.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:11.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:11.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:11.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:11.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:38:11.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:11.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:11.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2021-01-15T16:38:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2021-01-15T16:38:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:38:12.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2021-01-15T16:38:12.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:12.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:12.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:12.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:12.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:12.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:12.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:12.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:12.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:12.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:12.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:38:12.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:12.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:12.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:13.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2021-01-15T16:38:13.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:38:13.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:38:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:38:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:38:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:38:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:38:13.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:38:13.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:38:13.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12230875 2021-01-15T16:38:13.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12230875 '!=' null 2021-01-15T16:38:13.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:38:13.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:38:13.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:38:13.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:38:13.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:38:13.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:38:13.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:38:13.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:38:13.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:38:13.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:38:13.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:38:14.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:38:14.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:38:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:38:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:38:14.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T16:38:14.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:38:14.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:38:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2021-01-15T16:38:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2021-01-15T16:38:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2021-01-15T16:38:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T16:38:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T16:38:14.714 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2021-01-15T16:38:15.228 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T16:38:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2021-01-15T16:38:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T16:38:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T16:38:15.862 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T16:38:15.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2021-01-15T16:38:15.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:38:15.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:38:15.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:38:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:38:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:38:16.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:38:16.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2021-01-15T16:38:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2021-01-15T16:38:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:38:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-01-15T16:38:16.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:38:16.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:38:16.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:38:16.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:38:16.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:38:16.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:38:16.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:38:16.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:38:16.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:38:16.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:38:16.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:38:16.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:38:16.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:38:16.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:38:16.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:38:16.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:38:16.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:38:16.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:38:16.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-01-15T16:38:16.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:38:16.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92827 2021-01-15T16:38:16.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:38:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:38:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2021-01-15T16:38:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2021-01-15T16:38:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2021-01-15T16:38:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:38:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-01-15T16:38:16.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:38:16.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:38:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:38:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:38:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:38:16.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:38:16.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:38:16.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:38:16.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:38:16.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:38:16.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:38:16.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:38:16.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:38:16.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:38:16.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:38:16.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:38:16.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:38:16.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:38:16.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-01-15T16:38:16.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:38:16.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92827 2021-01-15T16:38:16.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:38:16.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:38:16.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2021-01-15T16:38:16.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:38:16.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.92827 2021-01-15T16:38:16.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:38:16.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:38:16.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:38:16.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2021-01-15T16:38:16.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2021-01-15T16:38:16.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2021-01-15T16:38:16.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2021-01-15T16:38:16.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2021-01-15T16:38:16.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2021-01-15T16:38:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2021-01-15T16:38:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2021-01-15T16:38:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:38:16.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:38:16.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:38:16.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2021-01-15T16:38:16.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2021-01-15T16:38:16.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:38:16.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:16.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:16.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:16.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:16.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:38:16.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:38:16.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:38:16.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:38:16.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:38:16.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:38:16.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:38:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:38:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:38:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:38:16.710 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:38:16.711 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.711 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-mon.a.asok 2021-01-15T16:38:16.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:38:16.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.92827/ceph-mon.a.asok config get fsid 2021-01-15T16:38:16.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:38:16.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:38:16.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:38:16.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:38:16.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:38:16.811 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:38:16.811 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:38:16.812 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:38:16.812 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:38:16.812 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:16.812 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:16.812 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-mon.a.asok 2021-01-15T16:38:16.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:38:16.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.92827/ceph-mon.a.asok config get mon_host 2021-01-15T16:38:16.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2021-01-15T16:38:16.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-prio 2021-01-15T16:38:16.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:38:16.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:38:16.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:38:16.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-prio/x 2021-01-15T16:38:16.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:38:17.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:38:17.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:17.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:17.402 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:17.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:17.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:38:17.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:38:17.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2021-01-15T16:38:17.427 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2021-01-15T16:38:17.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2021-01-15T16:38:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:38:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2021-01-15T16:38:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:38:17.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:38:17.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:38:17.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:38:17.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/0 2021-01-15T16:38:17.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:38:17.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:38:17.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:38:17.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:38:17.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2021-01-15T16:38:17.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2021-01-15T16:38:17.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:38:17.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:38:17.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:38:17.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:38:17.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:17.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:17.435 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:17.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:38:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:38:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:38:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:38:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:38:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:38:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:38:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:38:17.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:38:17.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/0 2021-01-15T16:38:17.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:38:17.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b6748777-594e-426c-8043-f4ab5385e412 2021-01-15T16:38:17.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 b6748777-594e-426c-8043-f4ab5385e412' 2021-01-15T16:38:17.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:38:17.441 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 b6748777-594e-426c-8043-f4ab5385e412 2021-01-15T16:38:17.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD5xAFg45W+GhAAxS6FVMiOSm65lIYPXtcJfA== 2021-01-15T16:38:17.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD5xAFg45W+GhAAxS6FVMiOSm65lIYPXtcJfA=="}' 2021-01-15T16:38:17.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b6748777-594e-426c-8043-f4ab5385e412 -i td/osd-backfill-prio/0/new.json 2021-01-15T16:38:17.791 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:17.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/0/new.json 2021-01-15T16:38:17.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD5xAFg45W+GhAAxS6FVMiOSm65lIYPXtcJfA== --osd-uuid b6748777-594e-426c-8043-f4ab5385e412 2021-01-15T16:38:17.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.830+0000 7f39b995a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:17.832 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.832+0000 7f39b995a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:17.834 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.834+0000 7f39b995a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:17.835 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.835+0000 7f39b995a080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:38:17.835 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.835+0000 7f39b995a080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:38:17.836 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.835+0000 7f39b995a080 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-01-15T16:38:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:17.873+0000 7f39b995a080 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2021-01-15T16:38:22.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2021-01-15T16:38:22.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:38:22.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:38:22.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:38:22.530 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:38:23.054 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:38:23.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:38:23.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:23.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:38:23.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:38:23.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:38:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:23.080+0000 7fd49798a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:23.106+0000 7fd49798a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:23.107+0000 7fd49798a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:23.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:38:23.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:38:23.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:38:23.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:38:23.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:38:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:38:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:38:23.466 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:23.623 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:23.622+0000 7fd49798a080 -1 Falling back to public interface 2021-01-15T16:38:23.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:38:24.417 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:24.416+0000 7fd49798a080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:38:24.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:38:24.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:38:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:38:24.862 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:25.283 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/102364,v1:127.0.0.1:6803/102364] [v2:127.0.0.1:6804/102364,v1:127.0.0.1:6805/102364] exists,up b6748777-594e-426c-8043-f4ab5385e412 2021-01-15T16:38:25.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:38:25.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:38:25.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:38:25.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:38:25.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2021-01-15T16:38:25.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:38:25.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:38:25.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:38:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:38:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/1 2021-01-15T16:38:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:38:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:38:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:38:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:38:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2021-01-15T16:38:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2021-01-15T16:38:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:38:25.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:38:25.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:38:25.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:38:25.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:25.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:25.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:25.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:25.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:25.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:38:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:38:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:38:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:38:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:38:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:38:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:38:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:25.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:38:25.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:38:25.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/1 2021-01-15T16:38:25.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:38:25.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=46acbce2-7b6e-4675-b287-b80cbbf02feb 2021-01-15T16:38:25.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 46acbce2-7b6e-4675-b287-b80cbbf02feb' 2021-01-15T16:38:25.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:38:25.294 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 46acbce2-7b6e-4675-b287-b80cbbf02feb 2021-01-15T16:38:25.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQABxQFg/+LyERAAb+FRzHj0r/DUD9GySp2aXQ== 2021-01-15T16:38:25.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQABxQFg/+LyERAAb+FRzHj0r/DUD9GySp2aXQ=="}' 2021-01-15T16:38:25.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 46acbce2-7b6e-4675-b287-b80cbbf02feb -i td/osd-backfill-prio/1/new.json 2021-01-15T16:38:25.808 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/1/new.json 2021-01-15T16:38:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQABxQFg/+LyERAAb+FRzHj0r/DUD9GySp2aXQ== --osd-uuid 46acbce2-7b6e-4675-b287-b80cbbf02feb 2021-01-15T16:38:25.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.846+0000 7fb1255d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.848+0000 7fb1255d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.849+0000 7fb1255d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.850+0000 7fb1255d0080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:38:25.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.850+0000 7fb1255d0080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:38:25.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.850+0000 7fb1255d0080 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-01-15T16:38:25.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:25.954+0000 7fb1255d0080 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2021-01-15T16:38:30.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2021-01-15T16:38:30.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:38:30.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:38:30.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:38:30.211 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:38:30.735 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:38:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:38:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:38:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:38:30.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:38:30.760 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:30.760+0000 7f34114ff080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:30.787 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:30.786+0000 7f34114ff080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:30.788 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:30.788+0000 7f34114ff080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:31.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:38:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:38:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:38:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:38:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:38:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:38:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:38:31.131 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:31.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:31.307+0000 7f34114ff080 -1 Falling back to public interface 2021-01-15T16:38:31.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:38:32.104 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:32.104+0000 7f34114ff080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:38:32.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:38:32.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:32.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:38:32.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:32.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:38:32.537 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:32.964 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/103628,v1:127.0.0.1:6811/103628] [v2:127.0.0.1:6812/103628,v1:127.0.0.1:6813/103628] exists,up 46acbce2-7b6e-4675-b287-b80cbbf02feb 2021-01-15T16:38:32.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:38:32.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:38:32.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:38:32.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:38:32.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2021-01-15T16:38:32.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:38:32.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:38:32.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:38:32.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:38:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/2 2021-01-15T16:38:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:38:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:38:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:38:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:38:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2021-01-15T16:38:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2021-01-15T16:38:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:38:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:38:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:38:32.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:38:32.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:32.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:32.981 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:32.981 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:32.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:32.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:32.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:38:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:38:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:38:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:38:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:38:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:38:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:38:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:38:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:38:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/2 2021-01-15T16:38:32.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:38:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2b081d42-e62f-4e83-8553-b69975a32419 2021-01-15T16:38:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 2b081d42-e62f-4e83-8553-b69975a32419' 2021-01-15T16:38:32.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:38:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAIxQFgS0t7OhAA7FqVHjnnQO1ZC5j8Xiwkkg== 2021-01-15T16:38:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAIxQFgS0t7OhAA7FqVHjnnQO1ZC5j8Xiwkkg=="}' 2021-01-15T16:38:32.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2b081d42-e62f-4e83-8553-b69975a32419 -i td/osd-backfill-prio/2/new.json 2021-01-15T16:38:32.986 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 2b081d42-e62f-4e83-8553-b69975a32419 2021-01-15T16:38:33.556 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:38:33.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/2/new.json 2021-01-15T16:38:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAIxQFgS0t7OhAA7FqVHjnnQO1ZC5j8Xiwkkg== --osd-uuid 2b081d42-e62f-4e83-8553-b69975a32419 2021-01-15T16:38:33.595 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.595+0000 7f12f250b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:33.597 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.596+0000 7f12f250b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:33.598 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.598+0000 7f12f250b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:33.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.598+0000 7f12f250b080 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-01-15T16:38:33.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.598+0000 7f12f250b080 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-01-15T16:38:33.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.598+0000 7f12f250b080 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-01-15T16:38:33.635 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:33.635+0000 7f12f250b080 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2021-01-15T16:38:38.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2021-01-15T16:38:38.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:38:38.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:38:38.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:38:38.281 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:38:38.805 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:38:38.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:38:38.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:38.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:38:38.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:38:38.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:38:38.829 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:38.829+0000 7f17eb73d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:38.854 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:38.853+0000 7f17eb73d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:38.856 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:38.855+0000 7f17eb73d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:39.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:38:39.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:38:39.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:38:39.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:38:39.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:38:39.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:39.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:38:39.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:39.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:38:39.203 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:39.381 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:39.380+0000 7f17eb73d080 -1 Falling back to public interface 2021-01-15T16:38:39.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:38:40.196 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:40.196+0000 7f17eb73d080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:38:40.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:38:40.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:40.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:38:40.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:40.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:38:40.660 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:41.082 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/104900,v1:127.0.0.1:6819/104900] [v2:127.0.0.1:6820/104900,v1:127.0.0.1:6821/104900] exists,up 2b081d42-e62f-4e83-8553-b69975a32419 2021-01-15T16:38:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:38:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:38:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:38:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:38:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2021-01-15T16:38:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:38:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:38:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:38:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:38:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/3 2021-01-15T16:38:41.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:38:41.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:38:41.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:38:41.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:38:41.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2021-01-15T16:38:41.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2021-01-15T16:38:41.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:38:41.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:38:41.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:38:41.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:38:41.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:41.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:41.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:41.088 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:41.088 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:41.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:41.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:41.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:38:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:38:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:38:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:38:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:38:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:38:41.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:38:41.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:41.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:38:41.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:38:41.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/3 2021-01-15T16:38:41.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:38:41.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=66793acd-d9ed-4371-8f02-4e6b0fa8c5d2 2021-01-15T16:38:41.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 66793acd-d9ed-4371-8f02-4e6b0fa8c5d2' 2021-01-15T16:38:41.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:38:41.092 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 66793acd-d9ed-4371-8f02-4e6b0fa8c5d2 2021-01-15T16:38:41.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQARxQFgrd/vBRAA6J08PD1qkswz6QYqQs0k8g== 2021-01-15T16:38:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQARxQFgrd/vBRAA6J08PD1qkswz6QYqQs0k8g=="}' 2021-01-15T16:38:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 66793acd-d9ed-4371-8f02-4e6b0fa8c5d2 -i td/osd-backfill-prio/3/new.json 2021-01-15T16:38:41.640 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:38:41.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/3/new.json 2021-01-15T16:38:41.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQARxQFgrd/vBRAA6J08PD1qkswz6QYqQs0k8g== --osd-uuid 66793acd-d9ed-4371-8f02-4e6b0fa8c5d2 2021-01-15T16:38:41.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.679+0000 7f83ec8c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:41.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.681+0000 7f83ec8c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:41.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.681+0000 7f83ec8c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:41.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.682+0000 7f83ec8c5080 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-01-15T16:38:41.683 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.682+0000 7f83ec8c5080 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-01-15T16:38:41.683 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.682+0000 7f83ec8c5080 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-01-15T16:38:41.729 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:41.729+0000 7f83ec8c5080 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2021-01-15T16:38:46.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2021-01-15T16:38:46.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:38:46.338 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:38:46.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:38:46.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:38:46.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:38:46.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:38:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:38:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:38:46.874 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:38:46.898 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:46.897+0000 7f0fef9fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:46.928 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:46.927+0000 7f0fef9fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:46.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:46.929+0000 7f0fef9fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:47.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:38:47.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:38:47.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:38:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:38:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:38:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:38:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:47.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:38:47.312 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:47.446 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:47.446+0000 7f0fef9fa080 -1 Falling back to public interface 2021-01-15T16:38:47.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:38:48.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:48.229+0000 7f0fef9fa080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:38:48.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:38:48.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:48.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:38:48.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:48.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:38:48.722 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:49.144 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/106164,v1:127.0.0.1:6827/106164] [v2:127.0.0.1:6828/106164,v1:127.0.0.1:6829/106164] exists,up 66793acd-d9ed-4371-8f02-4e6b0fa8c5d2 2021-01-15T16:38:49.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:38:49.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:38:49.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:38:49.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:38:49.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2021-01-15T16:38:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-prio 2021-01-15T16:38:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:38:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T16:38:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:38:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-prio/4 2021-01-15T16:38:49.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:38:49.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:38:49.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:38:49.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:38:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2021-01-15T16:38:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2021-01-15T16:38:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:38:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:38:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-01-15T16:38:49.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:38:49.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:38:49.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:38:49.149 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:38:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:38:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:38:49.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' 2021-01-15T16:38:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:38:49.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:38:49.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:38:49.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-01-15T16:38:49.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-01-15T16:38:49.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:38:49.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:38:49.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:49.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:38:49.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:38:49.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-prio/4 2021-01-15T16:38:49.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:38:49.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d5a9b9bd-959c-402d-9a13-2f1a6a143ab4 2021-01-15T16:38:49.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 d5a9b9bd-959c-402d-9a13-2f1a6a143ab4' 2021-01-15T16:38:49.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:38:49.155 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 d5a9b9bd-959c-402d-9a13-2f1a6a143ab4 2021-01-15T16:38:49.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAZxQFgYS+cCRAA/sI4DSGSD3yclrfB21/yxg== 2021-01-15T16:38:49.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAZxQFgYS+cCRAA/sI4DSGSD3yclrfB21/yxg=="}' 2021-01-15T16:38:49.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d5a9b9bd-959c-402d-9a13-2f1a6a143ab4 -i td/osd-backfill-prio/4/new.json 2021-01-15T16:38:49.808 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T16:38:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-prio/4/new.json 2021-01-15T16:38:49.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZxQFgYS+cCRAA/sI4DSGSD3yclrfB21/yxg== --osd-uuid d5a9b9bd-959c-402d-9a13-2f1a6a143ab4 2021-01-15T16:38:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.846+0000 7f0a31f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.848+0000 7f0a31f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:49.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.849+0000 7f0a31f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:49.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.850+0000 7f0a31f0c080 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-01-15T16:38:49.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.850+0000 7f0a31f0c080 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-01-15T16:38:49.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.850+0000 7f0a31f0c080 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-01-15T16:38:49.928 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:49.928+0000 7f0a31f0c080 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2021-01-15T16:38:54.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2021-01-15T16:38:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:38:54.413 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T16:38:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T16:38:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:38:54.945 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T16:38:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T16:38:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=2536e8df-7ac7-475b-9091-63fb9eb29ad8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.92827/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:38:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:38:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:38:54.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:38:54.971 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:54.971+0000 7f841238b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:54.973 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:54.973+0000 7f841238b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:54.974 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:54.973+0000 7f841238b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:38:55.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T16:38:55.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:38:55.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T16:38:55.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:38:55.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:38:55.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:55.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:38:55.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:55.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:38:55.346 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:38:55.493 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:55.493+0000 7f841238b080 -1 Falling back to public interface 2021-01-15T16:38:55.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:38:56.312 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:38:56.312+0000 7f841238b080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T16:38:56.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:38:56.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:38:56.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:38:56.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:38:56.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:38:56.745 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:38:57.210 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/107428,v1:127.0.0.1:6835/107428] [v2:127.0.0.1:6836/107428,v1:127.0.0.1:6837/107428] exists,up d5a9b9bd-959c-402d-9a13-2f1a6a143ab4 2021-01-15T16:38:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:38:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:38:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:38:57.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2021-01-15T16:38:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:38:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2021-01-15T16:38:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T16:38:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T16:38:57.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:38:58.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2021-01-15T16:38:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T16:38:59.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:38:59.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2021-01-15T16:38:59.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T16:39:00.253 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T16:39:00.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2021-01-15T16:39:02.102 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T16:39:02.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:02.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2021-01-15T16:39:02.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T16:39:02.920 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T16:39:02.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:03.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2021-01-15T16:39:04.994 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T16:39:05.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:05.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2021-01-15T16:39:05.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2021-01-15T16:39:05.726 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' already exists 2021-01-15T16:39:05.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:06.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2021-01-15T16:39:07.623 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2021-01-15T16:39:07.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:07.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2021-01-15T16:39:07.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2021-01-15T16:39:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' already exists 2021-01-15T16:39:08.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:09.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2021-01-15T16:39:10.002 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 2 2021-01-15T16:39:10.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:10.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2021-01-15T16:39:10.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2021-01-15T16:39:10.629 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' already exists 2021-01-15T16:39:10.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:11.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2021-01-15T16:39:12.412 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 2 2021-01-15T16:39:12.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:12.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2021-01-15T16:39:12.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2021-01-15T16:39:13.026 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' already exists 2021-01-15T16:39:13.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:14.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2021-01-15T16:39:14.873 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T16:39:14.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:14.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2021-01-15T16:39:14.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2021-01-15T16:39:15.568 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' already exists 2021-01-15T16:39:15.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:16.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2021-01-15T16:39:17.536 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 2 2021-01-15T16:39:17.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:17.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2021-01-15T16:39:17.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2021-01-15T16:39:18.319 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' already exists 2021-01-15T16:39:18.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:19.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2021-01-15T16:39:20.367 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 9 size to 2 2021-01-15T16:39:20.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:20.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2021-01-15T16:39:20.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2021-01-15T16:39:20.979 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' already exists 2021-01-15T16:39:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:39:21.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2021-01-15T16:39:23.012 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 10 size to 2 2021-01-15T16:39:23.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2021-01-15T16:39:28.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2021-01-15T16:39:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:39:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:39:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:39:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:39:28.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:39:28.025 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:39:28.026 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:39:28.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:39:28.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:39:28.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:39:28.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:39:28.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:39:28.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:39:28.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:39:28.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:39:28.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:39:28.521 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:39:28.521 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:39:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:39:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:39:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:39:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:28.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:39:28.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T16:39:28.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T16:39:28.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T16:39:28.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:28.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:39:28.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:39:28.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:39:28.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T16:39:28.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:28.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:39:28.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T16:39:28.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T16:39:28.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T16:39:28.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:28.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:39:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345930 2021-01-15T16:39:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345930 2021-01-15T16:39:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-85899345930' 2021-01-15T16:39:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:29.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:39:29.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182409 2021-01-15T16:39:29.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182409 2021-01-15T16:39:29.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-85899345930 4-107374182409' 2021-01-15T16:39:29.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:29.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T16:39:29.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:29.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:39:29.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T16:39:29.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:29.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T16:39:29.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T16:39:29.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:39:29.286 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T16:39:29.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2021-01-15T16:39:29.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:29.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:39:29.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:29.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:39:29.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:39:29.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:29.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:39:29.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:39:29.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:39:29.646 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:39:30.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T16:39:30.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:30.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T16:39:30.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:30.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:39:30.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T16:39:30.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:30.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T16:39:30.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T16:39:30.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:39:30.036 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T16:39:30.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T16:39:30.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:30.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345930 2021-01-15T16:39:30.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:30.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:39:30.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345930 2021-01-15T16:39:30.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:30.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345930 2021-01-15T16:39:30.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2021-01-15T16:39:30.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:39:30.451 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345930 2021-01-15T16:39:30.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345930 -lt 85899345930 2021-01-15T16:39:30.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:30.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182409 2021-01-15T16:39:30.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:30.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:39:30.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182409 2021-01-15T16:39:30.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:30.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182409 2021-01-15T16:39:30.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2021-01-15T16:39:30.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:39:30.870 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182409 2021-01-15T16:39:31.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182407 -lt 107374182409 2021-01-15T16:39:31.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:39:32.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:39:32.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:39:32.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182409 -lt 107374182409 2021-01-15T16:39:32.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:39:32.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:39:32.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:39:33.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2021-01-15T16:39:33.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:39:33.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:39:33.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:39:33.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:39:33.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:39:33.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:39:33.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:39:33.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2021-01-15T16:39:33.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:39:33.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:39:33.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:39:34.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2021-01-15T16:39:34.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:39:34.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:39:34.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2021-01-15T16:39:34.489 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:39:34.490 INFO:tasks.workunit.client.0.smithi145.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:23.620421+0000 0'0 79:20 [2,1] 2 [2,1] 2 0'0 2021-01-15T16:39:20.755980+0000 0'0 2021-01-15T16:39:20.755980+0000 0 2021-01-15T16:39:34.490 INFO:tasks.workunit.client.0.smithi145.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:21.116463+0000 0'0 78:24 [1,2] 1 [1,2] 1 0'0 2021-01-15T16:39:18.060586+0000 0'0 2021-01-15T16:39:18.060586+0000 0 2021-01-15T16:39:34.490 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:17.935550+0000 0'0 78:26 [3,4] 3 [3,4] 3 0'0 2021-01-15T16:39:15.331619+0000 0'0 2021-01-15T16:39:15.331619+0000 0 2021-01-15T16:39:34.491 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:15.700154+0000 0'0 78:32 [1,4] 1 [1,4] 1 0'0 2021-01-15T16:39:12.788297+0000 0'0 2021-01-15T16:39:12.788297+0000 0 2021-01-15T16:39:34.491 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:03.106109+0000 0'0 78:51 [3,1] 3 [3,1] 3 0'0 2021-01-15T16:39:00.007363+0000 0'0 2021-01-15T16:39:00.007363+0000 0 2021-01-15T16:39:34.491 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:00.442588+0000 0'0 78:56 [1,0] 1 [1,0] 1 0'0 2021-01-15T16:38:57.619195+0000 0'0 2021-01-15T16:38:57.619195+0000 0 2021-01-15T16:39:34.491 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:05.159102+0000 0'0 78:46 [1,2] 1 [1,2] 1 0'0 2021-01-15T16:39:02.666268+0000 0'0 2021-01-15T16:39:02.666268+0000 0 2021-01-15T16:39:34.491 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:08.407595+0000 0'0 75:40 [4,0] 4 [4,0] 4 0'0 2021-01-15T16:39:05.403830+0000 0'0 2021-01-15T16:39:05.403830+0000 0 2021-01-15T16:39:34.492 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:10.534888+0000 0'0 78:35 [3,4] 3 [3,4] 3 0'0 2021-01-15T16:39:07.996948+0000 0'0 2021-01-15T16:39:07.996948+0000 0 2021-01-15T16:39:34.492 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:39:13.448444+0000 0'0 75:25 [4,2] 4 [4,2] 4 0'0 2021-01-15T16:39:10.358581+0000 0'0 2021-01-15T16:39:10.358581+0000 0 2021-01-15T16:39:34.492 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:39:34.492 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:39:34.493 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:39:34.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2021-01-15T16:39:34.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2021-01-15T16:39:34.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2021-01-15T16:39:34.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2021-01-15T16:39:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2021-01-15T16:39:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2021-01-15T16:39:34.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2021-01-15T16:39:34.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2021-01-15T16:39:34.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2021-01-15T16:39:34.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2021-01-15T16:39:34.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2021-01-15T16:39:34.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2021-01-15T16:39:34.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2021-01-15T16:39:34.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:34.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2021-01-15T16:39:34.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:34.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:34.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2021-01-15T16:39:34.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:34.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2021-01-15T16:39:34.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2021-01-15T16:39:34.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2021-01-15T16:39:34.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2021-01-15T16:39:34.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2021-01-15T16:39:34.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2021-01-15T16:39:34.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2021-01-15T16:39:34.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:34.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2021-01-15T16:39:34.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:35.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:35.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2021-01-15T16:39:35.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:35.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2021-01-15T16:39:35.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:35.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-01-15T16:39:35.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-01-15T16:39:35.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:35.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2021-01-15T16:39:35.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:35.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:35.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2021-01-15T16:39:35.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:35.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2021-01-15T16:39:35.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:35.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-01-15T16:39:35.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2021-01-15T16:39:35.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2021-01-15T16:39:35.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2021-01-15T16:39:35.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2021-01-15T16:39:35.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:35.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2021-01-15T16:39:35.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:36.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2021-01-15T16:39:36.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2021-01-15T16:39:36.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-01-15T16:39:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-01-15T16:39:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2021-01-15T16:39:36.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:36.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2021-01-15T16:39:36.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2021-01-15T16:39:36.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:36.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-01-15T16:39:36.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-01-15T16:39:36.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:36.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2021-01-15T16:39:36.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:36.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2021-01-15T16:39:36.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2021-01-15T16:39:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-01-15T16:39:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-01-15T16:39:36.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:36.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2021-01-15T16:39:36.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2021-01-15T16:39:37.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-01-15T16:39:37.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2021-01-15T16:39:37.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-01-15T16:39:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2021-01-15T16:39:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2021-01-15T16:39:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-01-15T16:39:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-01-15T16:39:37.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2021-01-15T16:39:37.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2021-01-15T16:39:37.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2021-01-15T16:39:37.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2021-01-15T16:39:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2021-01-15T16:39:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2021-01-15T16:39:37.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2021-01-15T16:39:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-01-15T16:39:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-01-15T16:39:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2021-01-15T16:39:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T16:39:37.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T16:39:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T16:39:37.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:37.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-01-15T16:39:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-01-15T16:39:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2021-01-15T16:39:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2021-01-15T16:39:37.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-01-15T16:39:38.514 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' does not exist 2021-01-15T16:39:38.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:38.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-01-15T16:39:38.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2021-01-15T16:39:38.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2021-01-15T16:39:38.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-01-15T16:39:39.107 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' does not exist 2021-01-15T16:39:39.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:39.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-01-15T16:39:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2021-01-15T16:39:39.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2021-01-15T16:39:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-01-15T16:39:39.740 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' does not exist 2021-01-15T16:39:39.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:39.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-01-15T16:39:39.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:39.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-01-15T16:39:39.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2021-01-15T16:39:39.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2021-01-15T16:39:39.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-01-15T16:39:40.300 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' does not exist 2021-01-15T16:39:40.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:40.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-01-15T16:39:40.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2021-01-15T16:39:40.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2021-01-15T16:39:40.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-01-15T16:39:40.936 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' does not exist 2021-01-15T16:39:40.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-01-15T16:39:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-01-15T16:39:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2021-01-15T16:39:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2021-01-15T16:39:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-01-15T16:39:41.581 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' does not exist 2021-01-15T16:39:41.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-01-15T16:39:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 1 2021-01-15T16:39:42.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-01-15T16:39:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 1 2021-01-15T16:39:43.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2021-01-15T16:39:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:39:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:39:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:39:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:39:43.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:39:43.406 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:39:43.407 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:39:43.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:39:43.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:39:43.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:39:43.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:39:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:39:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:39:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:39:43.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:39:44.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:39:44.009 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:39:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:39:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:39:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:39:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:39:44.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:44.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:39:44.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2021-01-15T16:39:44.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2021-01-15T16:39:44.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2021-01-15T16:39:44.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:44.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:39:44.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T16:39:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T16:39:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2021-01-15T16:39:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:44.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:39:44.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T16:39:44.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T16:39:44.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457' 2021-01-15T16:39:44.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:44.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:39:44.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345935 2021-01-15T16:39:44.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345935 2021-01-15T16:39:44.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935' 2021-01-15T16:39:44.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:39:44.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:39:44.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T16:39:44.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T16:39:44.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935 4-107374182414' 2021-01-15T16:39:44.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:44.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2021-01-15T16:39:44.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:44.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:39:44.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-01-15T16:39:44.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:44.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2021-01-15T16:39:44.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-01-15T16:39:44.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:39:44.713 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836500 2021-01-15T16:39:45.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836500 2021-01-15T16:39:45.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:39:46.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:39:46.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:39:46.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2021-01-15T16:39:46.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:46.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T16:39:46.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:46.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:39:46.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T16:39:46.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:46.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T16:39:46.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T16:39:46.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:39:46.517 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T16:39:46.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672979 2021-01-15T16:39:46.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:46.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T16:39:46.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:46.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:39:46.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T16:39:46.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:46.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T16:39:46.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T16:39:46.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:39:46.912 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T16:39:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2021-01-15T16:39:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345935 2021-01-15T16:39:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:39:47.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345935 2021-01-15T16:39:47.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345935 2021-01-15T16:39:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2021-01-15T16:39:47.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:39:47.323 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345935 2021-01-15T16:39:47.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345935 2021-01-15T16:39:47.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:39:47.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T16:39:47.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:39:47.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:39:47.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T16:39:47.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:39:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T16:39:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T16:39:47.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:39:47.723 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T16:39:48.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2021-01-15T16:39:48.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:39:48.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:39:48.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:39:48.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T16:39:48.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:39:48.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:39:48.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:39:48.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:39:48.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:39:48.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:39:48.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:39:49.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T16:39:49.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:39:49.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:39:49.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:39:49.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T16:39:49.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:39:49.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:39:49.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2021-01-15T16:39:49.593 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2021-01-15T16:39:49.593 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2021-01-15T16:39:49.593 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0454694 s, 231 MB/s 2021-01-15T16:39:49.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2021-01-15T16:39:49.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T16:39:49.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2021-01-15T16:39:49.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:49.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2021-01-15T16:39:50.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:50.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2021-01-15T16:39:50.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2021-01-15T16:39:51.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:51.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2021-01-15T16:39:52.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:52.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2021-01-15T16:39:52.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:52.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2021-01-15T16:39:53.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:53.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2021-01-15T16:39:54.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:54.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2021-01-15T16:39:55.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:55.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2021-01-15T16:39:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:55.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2021-01-15T16:39:56.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:56.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2021-01-15T16:39:57.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:57.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2021-01-15T16:39:58.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:58.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2021-01-15T16:39:58.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:58.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2021-01-15T16:39:59.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:39:59.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2021-01-15T16:40:00.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:00.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2021-01-15T16:40:00.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:00.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2021-01-15T16:40:01.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:01.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2021-01-15T16:40:02.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:02.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2021-01-15T16:40:03.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:03.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2021-01-15T16:40:03.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:03.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2021-01-15T16:40:04.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2021-01-15T16:40:05.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:05.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2021-01-15T16:40:06.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:06.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2021-01-15T16:40:07.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:07.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2021-01-15T16:40:08.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:08.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2021-01-15T16:40:09.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:09.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2021-01-15T16:40:09.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:09.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2021-01-15T16:40:10.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:10.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2021-01-15T16:40:11.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2021-01-15T16:40:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2021-01-15T16:40:12.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:12.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2021-01-15T16:40:13.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:13.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2021-01-15T16:40:14.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:14.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2021-01-15T16:40:15.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:15.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2021-01-15T16:40:15.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:15.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2021-01-15T16:40:16.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:16.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2021-01-15T16:40:17.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:17.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2021-01-15T16:40:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2021-01-15T16:40:18.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:18.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2021-01-15T16:40:19.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:19.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2021-01-15T16:40:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2021-01-15T16:40:20.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:20.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2021-01-15T16:40:21.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:21.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2021-01-15T16:40:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2021-01-15T16:40:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2021-01-15T16:40:23.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:23.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2021-01-15T16:40:25.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:25.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2021-01-15T16:40:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:26.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2021-01-15T16:40:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:27.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2021-01-15T16:40:27.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2021-01-15T16:40:27.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2021-01-15T16:40:27.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T16:40:27.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2021-01-15T16:40:27.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:27.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2021-01-15T16:40:27.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2021-01-15T16:40:28.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:28.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2021-01-15T16:40:28.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:28.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2021-01-15T16:40:29.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:29.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2021-01-15T16:40:29.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:29.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2021-01-15T16:40:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:30.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2021-01-15T16:40:30.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:30.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2021-01-15T16:40:30.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:30.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2021-01-15T16:40:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2021-01-15T16:40:31.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2021-01-15T16:40:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2021-01-15T16:40:32.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:32.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2021-01-15T16:40:32.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:32.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2021-01-15T16:40:33.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:33.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2021-01-15T16:40:33.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:33.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2021-01-15T16:40:34.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:34.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2021-01-15T16:40:34.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:34.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2021-01-15T16:40:34.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:34.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2021-01-15T16:40:35.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:35.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2021-01-15T16:40:35.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:35.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2021-01-15T16:40:37.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:37.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2021-01-15T16:40:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2021-01-15T16:40:37.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:37.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2021-01-15T16:40:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:38.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2021-01-15T16:40:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2021-01-15T16:40:38.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:38.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2021-01-15T16:40:39.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:39.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2021-01-15T16:40:40.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:40.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2021-01-15T16:40:40.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:40.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2021-01-15T16:40:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2021-01-15T16:40:40.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:40.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2021-01-15T16:40:41.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:41.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2021-01-15T16:40:41.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:41.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2021-01-15T16:40:41.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:41.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2021-01-15T16:40:42.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:42.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2021-01-15T16:40:43.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:43.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2021-01-15T16:40:43.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:43.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2021-01-15T16:40:43.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:43.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2021-01-15T16:40:44.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:44.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2021-01-15T16:40:44.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:44.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2021-01-15T16:40:44.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:44.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2021-01-15T16:40:45.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:45.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2021-01-15T16:40:46.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:46.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2021-01-15T16:40:46.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:46.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2021-01-15T16:40:47.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:47.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2021-01-15T16:40:48.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:48.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2021-01-15T16:40:50.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:50.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2021-01-15T16:40:50.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:50.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2021-01-15T16:40:52.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2021-01-15T16:40:53.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2021-01-15T16:40:53.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2021-01-15T16:40:53.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T16:40:53.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2021-01-15T16:40:53.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:53.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2021-01-15T16:40:53.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:53.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2021-01-15T16:40:53.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:53.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2021-01-15T16:40:54.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:54.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2021-01-15T16:40:54.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:54.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2021-01-15T16:40:54.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2021-01-15T16:40:55.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:55.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2021-01-15T16:40:56.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:56.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2021-01-15T16:40:56.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:56.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2021-01-15T16:40:56.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:56.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2021-01-15T16:40:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2021-01-15T16:40:57.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:57.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2021-01-15T16:40:57.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:57.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2021-01-15T16:40:58.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:58.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2021-01-15T16:40:59.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:59.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2021-01-15T16:40:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2021-01-15T16:40:59.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:40:59.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2021-01-15T16:41:00.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:00.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2021-01-15T16:41:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:00.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2021-01-15T16:41:01.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:01.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2021-01-15T16:41:01.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:01.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2021-01-15T16:41:02.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:02.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2021-01-15T16:41:02.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:02.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2021-01-15T16:41:03.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:03.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2021-01-15T16:41:03.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:03.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2021-01-15T16:41:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:03.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2021-01-15T16:41:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2021-01-15T16:41:04.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:04.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2021-01-15T16:41:04.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:04.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2021-01-15T16:41:05.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2021-01-15T16:41:05.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:05.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2021-01-15T16:41:06.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:06.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2021-01-15T16:41:06.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:06.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2021-01-15T16:41:06.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:06.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2021-01-15T16:41:07.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:07.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2021-01-15T16:41:07.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:07.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2021-01-15T16:41:08.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:08.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2021-01-15T16:41:08.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:08.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2021-01-15T16:41:09.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:09.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2021-01-15T16:41:09.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:09.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2021-01-15T16:41:09.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:09.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2021-01-15T16:41:10.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:10.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2021-01-15T16:41:10.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:10.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2021-01-15T16:41:11.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:11.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2021-01-15T16:41:12.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:12.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2021-01-15T16:41:12.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:12.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2021-01-15T16:41:12.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:12.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2021-01-15T16:41:13.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:13.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2021-01-15T16:41:13.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:13.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2021-01-15T16:41:13.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-01-15T16:41:13.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2021-01-15T16:41:14.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2021-01-15T16:41:14.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2021-01-15T16:41:14.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2021-01-15T16:41:14.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2021-01-15T16:41:14.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2021-01-15T16:41:14.007 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2021-01-15T16:41:14.007 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2021-01-15T16:41:14.007 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2021-01-15T16:41:14.007 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T16:41:14.008 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T16:41:14.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T16:41:14.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T16:41:14.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T16:41:15.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2021-01-15T16:41:15.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2021-01-15T16:41:15.633 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:41:15.634 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 48 0 0 0 0 503316480 0 0 44 44 active+clean 2021-01-15T16:39:43.777534+0000 92'144 92:192 [1] 1 [1] 1 0'0 2021-01-15T16:39:12.788297+0000 0'0 2021-01-15T16:39:12.788297+0000 0 2021-01-15T16:41:15.634 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:39:00.442588+0000 92'150 92:215 [1,0] 1 [1,0] 1 0'0 2021-01-15T16:38:57.619195+0000 0'0 2021-01-15T16:38:57.619195+0000 0 2021-01-15T16:41:15.634 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:39:42.908397+0000 92'150 92:212 [1] 1 [1] 1 0'0 2021-01-15T16:39:02.666268+0000 0'0 2021-01-15T16:39:02.666268+0000 0 2021-01-15T16:41:15.635 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:41:15.635 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:41:15.635 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:41:15.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2021-01-15T16:41:15.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2021-01-15T16:41:16.308 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T16:41:16.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2021-01-15T16:41:17.202 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:41:17.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2021-01-15T16:41:17.856 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T16:41:17.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2021-01-15T16:41:19.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:19.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:19.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:19.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:19.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:19.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:19.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:19.978 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:19.979 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:19.979 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:19.979 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:19.979 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:19.980 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:19.980 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:19.980 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:19.980 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:41:19.980 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:19.981 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:19.981 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:19.981 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:19.981 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:19.981 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:19.982 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:19.982 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:19.982 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:19.982 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:19.982 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:19.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2021-01-15T16:41:19.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-01-15T16:41:19.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-01-15T16:41:19.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-01-15T16:41:20.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2021-01-15T16:41:20.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2021-01-15T16:41:22.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-01-15T16:41:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-01-15T16:41:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-01-15T16:41:22.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 2 = 10 ']' 2021-01-15T16:41:22.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2021-01-15T16:41:24.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-01-15T16:41:24.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-01-15T16:41:24.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-01-15T16:41:25.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 3 = 10 ']' 2021-01-15T16:41:25.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2021-01-15T16:41:27.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-01-15T16:41:27.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-01-15T16:41:27.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-01-15T16:41:27.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2021-01-15T16:41:27.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2021-01-15T16:41:27.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:41:27.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:41:27.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:41:27.960 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:41:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:41:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:41:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:41:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:41:27.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:27.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:41:28.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836523 2021-01-15T16:41:28.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836523 2021-01-15T16:41:28.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523' 2021-01-15T16:41:28.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:28.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:41:28.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673001 2021-01-15T16:41:28.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673001 2021-01-15T16:41:28.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-42949673001' 2021-01-15T16:41:28.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:28.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:41:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509480 2021-01-15T16:41:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509480 2021-01-15T16:41:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509480' 2021-01-15T16:41:28.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:28.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:41:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345958 2021-01-15T16:41:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345958 2021-01-15T16:41:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509480 3-85899345958' 2021-01-15T16:41:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:28.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:41:28.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182437 2021-01-15T16:41:28.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182437 2021-01-15T16:41:28.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509480 3-85899345958 4-107374182437' 2021-01-15T16:41:28.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:28.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836523 2021-01-15T16:41:28.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:28.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:41:28.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836523 2021-01-15T16:41:28.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:28.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836523 2021-01-15T16:41:28.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2021-01-15T16:41:28.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:28.663 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836523 2021-01-15T16:41:29.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836521 -lt 21474836523 2021-01-15T16:41:29.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:41:30.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:41:30.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:30.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836523 -lt 21474836523 2021-01-15T16:41:30.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:30.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673001 2021-01-15T16:41:30.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:30.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:41:30.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673001 2021-01-15T16:41:30.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:30.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673001 2021-01-15T16:41:30.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2021-01-15T16:41:30.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:41:30.460 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673001 2021-01-15T16:41:30.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673002 -lt 42949673001 2021-01-15T16:41:30.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:30.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509480 2021-01-15T16:41:30.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:30.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:41:30.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509480 2021-01-15T16:41:30.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:30.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509480 2021-01-15T16:41:30.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2021-01-15T16:41:30.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:41:30.859 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509480 2021-01-15T16:41:31.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509480 -lt 64424509480 2021-01-15T16:41:31.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:31.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345958 2021-01-15T16:41:31.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:31.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:41:31.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345958 2021-01-15T16:41:31.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:31.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345958 2021-01-15T16:41:31.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345958' 2021-01-15T16:41:31.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:41:31.256 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345958 2021-01-15T16:41:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345959 -lt 85899345958 2021-01-15T16:41:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182437 2021-01-15T16:41:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:31.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:41:31.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182437 2021-01-15T16:41:31.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:31.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182437 2021-01-15T16:41:31.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2021-01-15T16:41:31.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:41:31.671 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182437 2021-01-15T16:41:32.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182437 -lt 107374182437 2021-01-15T16:41:32.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:32.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:32.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:32.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:32.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:32.167 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:32.167 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:32.168 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:32.169 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:32.169 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:32.169 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:32.169 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:32.169 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:32.170 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:32.170 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:32.170 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:32.170 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:32.170 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:32.171 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:32.171 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:32.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2021-01-15T16:41:32.712 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T16:41:32.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2021-01-15T16:41:34.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2021-01-15T16:41:34.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:41:34.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:41:35.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:41:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:41:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:41:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:41:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:41:35.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:41:35.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:35.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:41:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836526 2021-01-15T16:41:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836526 2021-01-15T16:41:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526' 2021-01-15T16:41:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:35.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:41:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673005 2021-01-15T16:41:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673005 2021-01-15T16:41:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-42949673005' 2021-01-15T16:41:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:35.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:41:35.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509483 2021-01-15T16:41:35.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509483 2021-01-15T16:41:35.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-42949673005 2-64424509483' 2021-01-15T16:41:35.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:35.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:41:35.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345962 2021-01-15T16:41:35.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345962 2021-01-15T16:41:35.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-42949673005 2-64424509483 3-85899345962' 2021-01-15T16:41:35.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:35.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:41:35.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182440 2021-01-15T16:41:35.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182440 2021-01-15T16:41:35.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836526 1-42949673005 2-64424509483 3-85899345962 4-107374182440' 2021-01-15T16:41:35.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:35.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836526 2021-01-15T16:41:35.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:35.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:41:35.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836526 2021-01-15T16:41:35.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836526 2021-01-15T16:41:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2021-01-15T16:41:35.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:35.838 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836526 2021-01-15T16:41:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 0 -lt 21474836526 2021-01-15T16:41:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:41:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:41:37.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:37.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836527 -lt 21474836526 2021-01-15T16:41:37.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:37.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673005 2021-01-15T16:41:37.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:37.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:41:37.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673005 2021-01-15T16:41:37.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:37.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673005 2021-01-15T16:41:37.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2021-01-15T16:41:37.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:41:37.663 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673005 2021-01-15T16:41:38.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673005 -lt 42949673005 2021-01-15T16:41:38.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:38.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509483 2021-01-15T16:41:38.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:38.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:41:38.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509483 2021-01-15T16:41:38.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:38.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509483 2021-01-15T16:41:38.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509483' 2021-01-15T16:41:38.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:41:38.056 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509483 2021-01-15T16:41:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509484 -lt 64424509483 2021-01-15T16:41:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:38.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345962 2021-01-15T16:41:38.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:38.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:41:38.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345962 2021-01-15T16:41:38.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345962 2021-01-15T16:41:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345962' 2021-01-15T16:41:38.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:41:38.453 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345962 2021-01-15T16:41:38.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345962 -lt 85899345962 2021-01-15T16:41:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:38.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182440 2021-01-15T16:41:38.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:38.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:41:38.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182440 2021-01-15T16:41:38.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:38.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182440 2021-01-15T16:41:38.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2021-01-15T16:41:38.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:41:38.861 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182440 2021-01-15T16:41:39.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182440 -lt 107374182440 2021-01-15T16:41:39.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:39.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:39.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:39.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:39.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:39.269 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:39.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:39.369 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:39.370 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:39.370 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:39.371 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:39.371 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:41:39.371 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:39.371 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:41:39.371 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:39.372 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:39.372 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:41:39.372 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:41:39.372 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:39.373 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:39.373 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:39.373 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:39.373 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:41:39.374 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:39.374 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:39.374 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:39.374 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:39.374 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:39.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2021-01-15T16:41:39.751 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:41:39.752 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2021-01-15T16:41:27.449867+0000 92'150 103:217 [1,4] 1 [1] 1 0'0 2021-01-15T16:39:12.788297+0000 0'0 2021-01-15T16:39:12.788297+0000 0 2021-01-15T16:41:39.752 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2021-01-15T16:41:34.158450+0000 92'150 104:232 [1,2] 1 [1,0] 1 0'0 2021-01-15T16:38:57.619195+0000 0'0 2021-01-15T16:38:57.619195+0000 0 2021-01-15T16:41:39.753 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:39:42.908397+0000 92'150 103:221 [1] 1 [1] 1 0'0 2021-01-15T16:39:02.666268+0000 0'0 2021-01-15T16:39:02.666268+0000 0 2021-01-15T16:41:39.753 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:41:39.754 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:41:39.754 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:41:39.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2021-01-15T16:41:40.577 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T16:41:40.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2021-01-15T16:41:42.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2021-01-15T16:41:42.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:41:42.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:41:42.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:41:42.993 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:41:42.993 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:41:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:41:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:41:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:41:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:42.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:41:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836530 2021-01-15T16:41:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836530 2021-01-15T16:41:43.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530' 2021-01-15T16:41:43.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:43.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:41:43.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673008 2021-01-15T16:41:43.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673008 2021-01-15T16:41:43.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2021-01-15T16:41:43.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:43.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:41:43.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509487 2021-01-15T16:41:43.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509487 2021-01-15T16:41:43.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509487' 2021-01-15T16:41:43.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:43.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:41:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345965 2021-01-15T16:41:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345965 2021-01-15T16:41:43.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509487 3-85899345965' 2021-01-15T16:41:43.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:43.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:41:43.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182444 2021-01-15T16:41:43.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182444 2021-01-15T16:41:43.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509487 3-85899345965 4-107374182444' 2021-01-15T16:41:43.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:43.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836530 2021-01-15T16:41:43.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:43.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:41:43.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836530 2021-01-15T16:41:43.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:43.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836530 2021-01-15T16:41:43.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2021-01-15T16:41:43.716 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836530 2021-01-15T16:41:43.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:44.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836530 -lt 21474836530 2021-01-15T16:41:44.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:44.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673008 2021-01-15T16:41:44.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:44.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:41:44.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673008 2021-01-15T16:41:44.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:44.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673008 2021-01-15T16:41:44.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2021-01-15T16:41:44.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:41:44.109 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673008 2021-01-15T16:41:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673006 -lt 42949673008 2021-01-15T16:41:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:41:45.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:41:45.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:41:45.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673009 -lt 42949673008 2021-01-15T16:41:45.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:45.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509487 2021-01-15T16:41:45.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:45.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:41:45.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509487 2021-01-15T16:41:45.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:45.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509487 2021-01-15T16:41:45.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509487' 2021-01-15T16:41:45.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:41:45.912 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509487 2021-01-15T16:41:46.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509487 -lt 64424509487 2021-01-15T16:41:46.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:46.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345965 2021-01-15T16:41:46.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:46.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:41:46.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345965 2021-01-15T16:41:46.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:46.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345965 2021-01-15T16:41:46.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2021-01-15T16:41:46.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:41:46.307 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345965 2021-01-15T16:41:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345966 -lt 85899345965 2021-01-15T16:41:46.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182444 2021-01-15T16:41:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:46.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:41:46.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182444 2021-01-15T16:41:46.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:46.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182444 2021-01-15T16:41:46.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2021-01-15T16:41:46.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:41:46.712 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182444 2021-01-15T16:41:47.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182444 -lt 107374182444 2021-01-15T16:41:47.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:47.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:47.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:47.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:47.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:47.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:47.213 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:47.213 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:47.214 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:47.214 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:47.214 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:41:47.214 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:47.215 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:41:47.215 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:47.215 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:47.215 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:41:47.216 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:41:47.216 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:47.217 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:47.217 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:47.217 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:47.218 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:47.218 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:41:47.219 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:47.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:47.219 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:41:47.220 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:41:47.221 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:47.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:47.221 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:47.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:47.222 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:41:47.223 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:47.223 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:47.223 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:47.223 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:47.223 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:47.224 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:47.224 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:47.224 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:47.225 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:47.225 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:47.225 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:47.225 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:47.226 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:47.226 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:47.226 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:47.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2021-01-15T16:41:47.583 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:41:47.583 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2021-01-15T16:41:27.449867+0000 92'150 107:220 [1,4] 1 [1] 1 0'0 2021-01-15T16:39:12.788297+0000 0'0 2021-01-15T16:39:12.788297+0000 0 2021-01-15T16:41:47.584 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2021-01-15T16:41:34.158450+0000 92'150 107:235 [1,2] 1 [1,0] 1 0'0 2021-01-15T16:38:57.619195+0000 0'0 2021-01-15T16:38:57.619195+0000 0 2021-01-15T16:41:47.584 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2021-01-15T16:41:42.647169+0000 92'150 108:231 [1,2] 1 [1] 1 0'0 2021-01-15T16:39:02.666268+0000 0'0 2021-01-15T16:39:02.666268+0000 0 2021-01-15T16:41:47.584 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:41:47.584 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:41:47.585 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:41:47.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:47.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-01-15T16:41:47.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2021-01-15T16:41:47.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2021-01-15T16:41:47.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:47.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T16:41:47.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2021-01-15T16:41:47.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2021-01-15T16:41:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2021-01-15T16:41:47.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:47.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T16:41:47.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2021-01-15T16:41:47.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-01-15T16:41:47.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2021-01-15T16:41:47.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-01-15T16:41:47.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2021-01-15T16:41:47.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-01-15T16:41:47.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2021-01-15T16:41:47.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2021-01-15T16:41:49.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:49.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:49.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:49.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:49.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:49.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:50.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:50.110 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:50.110 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:50.110 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:50.111 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:50.111 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:41:50.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:50.111 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:41:50.111 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:50.112 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:50.112 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:41:50.112 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:41:50.112 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:50.112 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:50.113 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:50.113 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:50.113 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:50.113 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 254, 2021-01-15T16:41:50.113 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:50.114 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:50.114 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:41:50.114 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:50.114 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:50.115 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:50.115 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:50.115 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:50.116 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:41:50.116 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:50.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:50.116 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:50.116 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:50.117 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:50.117 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:50.117 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:50.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:50.118 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:50.119 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:50.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:50.119 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:50.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:50.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-01-15T16:41:50.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2021-01-15T16:41:50.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-01-15T16:41:50.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2021-01-15T16:41:50.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:41:50.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:41:50.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:41:50.506 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:41:50.506 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:41:50.507 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:41:50.507 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:41:50.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:41:50.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:50.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:41:50.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836533 2021-01-15T16:41:50.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836533 2021-01-15T16:41:50.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533' 2021-01-15T16:41:50.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:50.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:41:50.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673012 2021-01-15T16:41:50.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673012 2021-01-15T16:41:50.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012' 2021-01-15T16:41:50.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:50.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:41:50.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509490 2021-01-15T16:41:50.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509490 2021-01-15T16:41:50.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490' 2021-01-15T16:41:50.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:50.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:41:51.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345969 2021-01-15T16:41:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345969 2021-01-15T16:41:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490 3-85899345969' 2021-01-15T16:41:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:41:51.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:41:51.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182447 2021-01-15T16:41:51.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182447 2021-01-15T16:41:51.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836533 1-42949673012 2-64424509490 3-85899345969 4-107374182447' 2021-01-15T16:41:51.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:51.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836533 2021-01-15T16:41:51.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:51.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:41:51.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836533 2021-01-15T16:41:51.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:51.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836533 2021-01-15T16:41:51.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2021-01-15T16:41:51.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:41:51.206 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836533 2021-01-15T16:41:51.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836534 -lt 21474836533 2021-01-15T16:41:51.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:51.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673012 2021-01-15T16:41:51.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:41:51.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673012 2021-01-15T16:41:51.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:51.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673012 2021-01-15T16:41:51.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2021-01-15T16:41:51.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:41:51.603 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673012 2021-01-15T16:41:52.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673012 -lt 42949673012 2021-01-15T16:41:52.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:52.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509490 2021-01-15T16:41:52.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:52.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:41:52.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509490 2021-01-15T16:41:52.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:52.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509490 2021-01-15T16:41:52.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509490' 2021-01-15T16:41:52.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:41:52.005 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509490 2021-01-15T16:41:52.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509490 -lt 64424509490 2021-01-15T16:41:52.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:52.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345969 2021-01-15T16:41:52.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:52.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:41:52.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345969 2021-01-15T16:41:52.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:52.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345969 2021-01-15T16:41:52.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345969' 2021-01-15T16:41:52.406 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345969 2021-01-15T16:41:52.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:41:52.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345969 -lt 85899345969 2021-01-15T16:41:52.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:41:52.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182447 2021-01-15T16:41:52.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:41:52.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:41:52.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182447 2021-01-15T16:41:52.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:41:52.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182447 2021-01-15T16:41:52.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182447' 2021-01-15T16:41:52.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:41:52.823 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182447 2021-01-15T16:41:53.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182447 -lt 107374182447 2021-01-15T16:41:53.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2021-01-15T16:41:53.615 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2021-01-15T16:41:53.615 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:41:53.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2021-01-15T16:41:55.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:41:55.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:41:55.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:41:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:41:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:41:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:41:55.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:41:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:41:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:41:55.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:55.746 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:41:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:41:55.746 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:41:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:41:55.747 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:41:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:41:55.748 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:41:55.749 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:41:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:41:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:41:55.750 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:55.751 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:41:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:41:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:41:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:41:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:41:55.752 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:41:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:41:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:41:55.753 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:41:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:41:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:41:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:41:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:41:55.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:41:55.755 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:41:55.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:55.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-01-15T16:41:55.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2021-01-15T16:41:55.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2021-01-15T16:41:55.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:55.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T16:41:55.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2021-01-15T16:41:55.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2021-01-15T16:41:55.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2021-01-15T16:41:55.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:41:55.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T16:41:55.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2021-01-15T16:41:55.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-01-15T16:41:55.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2021-01-15T16:41:56.132 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2021-01-15T16:41:56.133 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:41:56.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2021-01-15T16:42:01.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:42:01.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:42:01.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:42:01.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:42:01.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:42:01.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:42:01.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:42:01.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:42:01.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:01.255 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:01.255 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:01.255 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:01.256 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:01.256 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:01.256 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:01.256 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:01.257 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:01.257 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:01.257 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:01.257 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:01.258 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:01.258 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:01.258 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:01.258 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:01.258 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:01.259 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:01.259 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:01.259 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:01.259 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:01.259 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:01.260 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:01.260 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:01.260 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:01.260 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:01.260 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:01.261 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:01.261 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:01.261 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:01.261 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:01.261 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:01.262 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:01.262 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:01.262 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:01.262 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:01.263 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2021-01-15T16:42:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:42:01.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:42:01.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:42:01.684 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:42:01.684 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:42:01.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836538 2021-01-15T16:42:01.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836538 2021-01-15T16:42:01.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538' 2021-01-15T16:42:01.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:01.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:42:01.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673016 2021-01-15T16:42:01.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673016 2021-01-15T16:42:01.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016' 2021-01-15T16:42:01.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:01.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:42:02.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509495 2021-01-15T16:42:02.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509495 2021-01-15T16:42:02.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016 2-64424509495' 2021-01-15T16:42:02.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:02.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:42:02.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345973 2021-01-15T16:42:02.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345973 2021-01-15T16:42:02.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016 2-64424509495 3-85899345973' 2021-01-15T16:42:02.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:02.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:42:02.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182451 2021-01-15T16:42:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182451 2021-01-15T16:42:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836538 1-42949673016 2-64424509495 3-85899345973 4-107374182451' 2021-01-15T16:42:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836538 2021-01-15T16:42:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:02.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:42:02.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836538 2021-01-15T16:42:02.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:02.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836538 2021-01-15T16:42:02.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2021-01-15T16:42:02.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:42:02.386 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836538 2021-01-15T16:42:02.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836536 -lt 21474836538 2021-01-15T16:42:02.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:42:03.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:42:03.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:42:04.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836538 -lt 21474836538 2021-01-15T16:42:04.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:04.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673016 2021-01-15T16:42:04.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:04.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:42:04.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673016 2021-01-15T16:42:04.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:04.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673016 2021-01-15T16:42:04.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2021-01-15T16:42:04.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:42:04.187 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673016 2021-01-15T16:42:04.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673016 -lt 42949673016 2021-01-15T16:42:04.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:04.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509495 2021-01-15T16:42:04.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:04.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:42:04.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509495 2021-01-15T16:42:04.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509495 2021-01-15T16:42:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509495' 2021-01-15T16:42:04.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:42:04.603 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509495 2021-01-15T16:42:04.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509495 -lt 64424509495 2021-01-15T16:42:04.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:04.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345973 2021-01-15T16:42:04.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:04.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:42:04.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345973 2021-01-15T16:42:04.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:04.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345973 2021-01-15T16:42:04.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345973' 2021-01-15T16:42:04.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:42:04.993 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345973 2021-01-15T16:42:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345974 -lt 85899345973 2021-01-15T16:42:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:05.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182451 2021-01-15T16:42:05.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:05.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:42:05.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182451 2021-01-15T16:42:05.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182451 2021-01-15T16:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182451' 2021-01-15T16:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:42:05.415 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182451 2021-01-15T16:42:05.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182452 -lt 107374182451 2021-01-15T16:42:05.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-01-15T16:42:06.185 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2021-01-15T16:42:06.186 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T16:42:06.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2021-01-15T16:42:08.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:42:08.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:42:08.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:42:08.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:42:08.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:42:08.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:42:08.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:42:08.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:42:08.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:42:08.317 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:08.317 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:08.317 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:08.318 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:08.318 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:08.318 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:08.318 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:08.319 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:08.319 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:08.319 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:08.319 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:08.319 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:08.320 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:08.320 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:08.320 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:08.320 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:08.321 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:08.321 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:08.321 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:08.321 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:08.322 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:08.322 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:08.322 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:08.322 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:08.322 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:08.323 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:08.323 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:08.323 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:08.323 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:08.324 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:08.324 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:08.324 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:08.324 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:08.324 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:08.325 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:08.325 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:08.325 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:08.325 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:08.326 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:08.326 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:08.326 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:08.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:42:08.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-01-15T16:42:08.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2021-01-15T16:42:08.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2021-01-15T16:42:08.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:42:08.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T16:42:08.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2021-01-15T16:42:08.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2021-01-15T16:42:08.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2021-01-15T16:42:08.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-01-15T16:42:08.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T16:42:08.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2021-01-15T16:42:08.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-01-15T16:42:08.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2021-01-15T16:42:09.028 INFO:tasks.workunit.client.0.smithi145.stderr:noout is unset 2021-01-15T16:42:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2021-01-15T16:42:09.781 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T16:42:09.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:42:09.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:42:09.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:42:09.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:42:09.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:42:09.796 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:42:09.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:42:09.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations' 2021-01-15T16:42:09.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations' 2021-01-15T16:42:09.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:42:09.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:42:09.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:42:09.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:42:09.800 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:42:09.800 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:42:09.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:42:09.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:42:09.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:42:09.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:42:09.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:42:09.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:42:09.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:42:09.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:42:10.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:42:10.278 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:42:10.278 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:42:10.279 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:42:10.279 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T16:42:10.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:42:10.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:10.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:42:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836541 2021-01-15T16:42:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836541 2021-01-15T16:42:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541' 2021-01-15T16:42:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:10.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:42:10.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673020 2021-01-15T16:42:10.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673020 2021-01-15T16:42:10.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-42949673020' 2021-01-15T16:42:10.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:10.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:42:10.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509498 2021-01-15T16:42:10.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509498 2021-01-15T16:42:10.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-42949673020 2-64424509498' 2021-01-15T16:42:10.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:10.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:42:10.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345977 2021-01-15T16:42:10.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345977 2021-01-15T16:42:10.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-42949673020 2-64424509498 3-85899345977' 2021-01-15T16:42:10.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:42:10.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:42:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182455 2021-01-15T16:42:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182455 2021-01-15T16:42:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836541 1-42949673020 2-64424509498 3-85899345977 4-107374182455' 2021-01-15T16:42:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:10.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836541 2021-01-15T16:42:10.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:10.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:42:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836541 2021-01-15T16:42:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:10.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836541 2021-01-15T16:42:10.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2021-01-15T16:42:10.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:42:10.999 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836541 2021-01-15T16:42:11.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836542 -lt 21474836541 2021-01-15T16:42:11.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:11.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673020 2021-01-15T16:42:11.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:11.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:42:11.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673020 2021-01-15T16:42:11.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:11.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673020 2021-01-15T16:42:11.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2021-01-15T16:42:11.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:42:11.416 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673020 2021-01-15T16:42:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673020 -lt 42949673020 2021-01-15T16:42:11.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:11.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509498 2021-01-15T16:42:11.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:11.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:42:11.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509498 2021-01-15T16:42:11.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:11.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509498 2021-01-15T16:42:11.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509498' 2021-01-15T16:42:11.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:42:11.897 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509498 2021-01-15T16:42:12.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509498 -lt 64424509498 2021-01-15T16:42:12.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:12.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345977 2021-01-15T16:42:12.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:12.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:42:12.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345977 2021-01-15T16:42:12.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:12.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345977 2021-01-15T16:42:12.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345977' 2021-01-15T16:42:12.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:42:12.340 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345977 2021-01-15T16:42:12.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345977 -lt 85899345977 2021-01-15T16:42:12.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:42:12.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182455 2021-01-15T16:42:12.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:42:12.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:42:12.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182455 2021-01-15T16:42:12.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:42:12.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182455 2021-01-15T16:42:12.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182455' 2021-01-15T16:42:12.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:42:12.745 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182455 2021-01-15T16:42:13.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182455 -lt 107374182455 2021-01-15T16:42:13.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:42:13.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:13.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:13.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T16:42:13.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:13.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:13.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:13.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:13.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:13.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:13.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:14.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:14.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:14.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:14.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:14.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:14.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:42:14.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:42:14.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:42:14.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:14.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:14.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:14.732 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:14.733 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:14.733 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:14.733 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:14.733 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:14.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:14.734 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:14.734 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:14.734 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:14.734 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:14.735 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:14.735 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:14.735 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:14.735 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:14.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:14.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:14.736 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:14.736 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:14.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:14.736 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:14.737 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:14.737 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:14.737 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:14.737 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:14.737 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:14.738 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:14.738 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:14.738 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:14.738 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:14.738 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:14.739 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:14.739 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:14.739 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:14.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:14.739 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:14.740 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:14.740 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:14.740 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:14.740 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:14.740 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:14.741 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:14.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:14.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:14.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:14.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:15.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:15.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:15.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:16.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:16.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:16.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:16.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6775351 2021-01-15T16:42:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6775351 '!=' null 2021-01-15T16:42:16.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:16.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:16.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:16.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:16.792 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:16.792 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:16.792 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:16.793 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:16.793 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:16.793 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:16.793 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:16.793 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:16.794 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:16.794 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:16.794 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:16.794 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:16.794 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:16.795 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:16.795 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:16.795 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:16.795 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:16.795 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:16.796 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:16.796 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:16.796 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:16.796 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:16.796 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:16.797 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:16.797 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:16.797 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:16.797 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:16.797 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:16.798 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:16.798 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:16.798 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:16.798 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:16.798 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:16.799 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:16.799 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:16.799 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:16.799 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:16.799 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:16.800 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:16.800 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:16.800 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:16.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:16.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:16.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:16.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:16.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:16.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:16.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:16.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:16.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:17.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:17.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:17.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:17.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:17.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:17.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:17.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:17.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:17.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:18.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5241948 2021-01-15T16:42:18.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5241948 '!=' null 2021-01-15T16:42:18.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:18.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:18.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:18.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:18.674 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:18.675 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:18.675 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:18.676 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:18.676 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:18.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:18.677 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:18.677 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:18.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:18.679 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:18.679 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:18.680 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:18.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:18.681 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:18.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:18.682 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:18.683 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:18.684 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:18.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:18.685 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:18.686 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:18.686 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:18.687 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:18.687 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:18.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:18.688 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:18.689 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:18.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:18.690 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:18.691 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:18.692 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:18.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:18.693 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:18.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:18.694 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:18.694 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:18.695 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:18.696 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:18.696 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:18.697 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:18.697 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:18.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:18.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:18.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:18.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:18.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:18.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:18.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:18.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:18.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:19.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:19.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:19.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:19.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:19.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:19.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:19.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:19.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:19.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:19.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:19.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:19.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:19.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:20.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12953970 2021-01-15T16:42:20.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12953970 '!=' null 2021-01-15T16:42:20.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:20.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:20.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:20.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:20.420 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:20.420 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:20.420 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:20.421 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:20.421 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:20.421 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:20.421 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:20.422 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:20.422 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:20.422 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:20.422 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:20.422 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:20.423 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:20.423 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:20.423 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:20.423 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:20.423 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:20.424 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:20.424 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:20.424 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:20.424 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:20.424 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:20.425 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:20.425 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:20.425 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:20.425 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:20.425 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:20.426 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:20.426 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:20.426 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:20.426 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:20.427 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:20.427 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:20.427 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:20.427 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:20.427 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:20.428 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:20.428 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:20.428 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:20.428 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:20.428 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:20.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:20.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:20.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:20.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:20.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:20.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:20.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:20.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:20.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:20.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:20.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:21.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:21.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:21.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:21.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:21.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:21.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:21.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:21.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:21.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11000743 2021-01-15T16:42:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11000743 '!=' null 2021-01-15T16:42:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:22.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:22.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:22.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:22.245 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:22.246 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:22.246 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:22.246 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:22.247 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:22.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:22.247 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:22.247 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:22.248 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:22.248 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:22.248 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:22.248 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:22.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:22.249 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:22.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:22.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:22.249 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:22.249 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:22.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:22.250 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:22.250 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:22.250 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:22.251 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:22.251 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:22.251 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:22.251 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:22.252 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:22.252 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:22.252 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:22.252 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:22.252 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:22.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:22.253 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:22.253 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:22.253 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:22.254 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:22.254 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:22.254 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:22.254 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:22.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:22.255 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:22.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:22.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:22.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:22.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:22.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:22.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:22.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:22.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:22.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:22.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:22.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:22.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:22.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:23.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:23.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:23.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:23.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:23.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:23.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:23.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483838 2021-01-15T16:42:23.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483838 '!=' null 2021-01-15T16:42:23.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:23.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:23.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:23.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:24.069 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:24.070 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:24.070 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:24.070 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:24.070 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:24.071 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:24.071 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:24.071 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:24.071 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:24.071 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:24.072 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:24.072 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:24.072 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:24.072 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:24.072 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:24.073 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:24.073 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:24.073 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:24.073 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:24.073 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:24.074 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:24.074 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:24.074 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:24.074 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:24.074 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:24.075 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:24.075 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:24.075 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:24.075 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:24.075 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:24.076 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:24.077 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:24.077 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:24.077 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:24.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:24.077 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:24.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:24.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:24.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:24.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:24.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:24.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:24.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:24.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:24.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:24.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:24.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:24.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:24.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:25.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:25.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:25.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:25.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:25.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:25.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:25.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:25.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:25.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220376 2021-01-15T16:42:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220376 '!=' null 2021-01-15T16:42:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:26.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:26.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:26.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:26.122 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:26.122 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:26.122 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:26.123 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:26.123 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:26.123 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:26.123 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:26.124 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:26.124 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:26.124 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:26.124 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:26.124 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:26.125 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:26.125 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:26.125 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:26.125 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:26.125 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:26.126 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:26.126 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:26.126 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:26.126 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:26.127 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:26.127 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:26.127 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:26.127 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:26.127 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:26.128 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:26.128 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:26.128 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:26.128 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:26.128 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:26.129 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:26.129 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:26.129 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:26.129 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:26.129 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:26.130 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:26.130 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:26.131 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:26.131 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:26.132 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:26.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:26.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:26.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:26.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:26.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:26.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:26.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:26.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:26.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:26.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:26.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:26.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:27.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:27.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:27.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:27.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:27.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:27.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:27.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:27.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725732 2021-01-15T16:42:27.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725732 '!=' null 2021-01-15T16:42:27.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:27.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:27.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:27.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:27.815 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:27.816 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:27.816 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:27.817 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:27.817 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:27.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:27.818 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:27.818 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:27.818 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:27.819 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:27.819 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:27.820 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:27.820 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:27.820 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:27.821 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:27.821 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:27.821 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:27.822 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:27.822 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:27.823 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:27.823 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:27.823 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:27.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:27.824 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:27.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:27.825 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:27.825 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:27.826 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:27.826 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:27.826 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:27.827 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:27.827 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:27.828 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:27.828 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:27.829 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:27.829 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:27.829 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:27.830 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:27.830 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:27.831 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:27.831 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:27.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:27.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:27.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:27.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:27.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:27.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:27.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:28.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:28.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:28.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:28.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:28.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:28.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:28.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:28.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:28.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:28.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:28.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:28.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:29.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725732 2021-01-15T16:42:29.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725732 '!=' null 2021-01-15T16:42:29.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:29.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:29.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:29.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:29.503 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:29.503 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:29.503 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:29.504 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:29.504 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:29.504 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:29.505 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:29.505 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:29.505 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:29.505 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:29.506 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:29.506 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:29.506 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:29.506 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:29.506 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:29.507 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:29.507 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:29.507 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:29.507 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:29.507 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:29.508 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:29.508 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:29.508 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:29.508 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:29.509 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:29.509 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:29.509 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:29.509 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:29.510 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:29.510 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:29.510 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:29.510 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:29.511 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:29.511 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:29.511 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:29.511 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:29.512 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:29.512 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:29.512 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:29.512 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:29.513 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:29.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:29.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:29.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:29.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:29.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:29.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:29.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:29.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:29.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:30.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:30.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:30.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:30.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:30.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:30.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:30.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:30.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:30.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:31.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714947 2021-01-15T16:42:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714947 '!=' null 2021-01-15T16:42:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:31.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:31.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:31.230 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:31.231 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:31.231 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:31.231 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:31.231 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:31.231 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:31.232 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:31.232 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:31.232 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:31.232 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:31.232 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:31.233 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:31.233 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:31.233 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:31.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:31.233 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:31.234 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:31.234 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:31.234 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:31.234 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:31.235 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:31.235 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:31.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:31.235 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:31.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:31.236 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:31.236 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:31.236 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:31.236 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:31.236 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:31.237 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:31.237 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:31.237 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:31.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:31.237 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:31.238 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:31.238 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:31.238 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:31.238 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:31.238 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:31.239 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:31.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:31.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:31.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:31.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:31.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:31.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:31.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:31.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:31.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:31.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:31.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:31.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:32.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:32.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:32.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:32.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:32.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:32.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:32.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:32.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:32.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:32.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725629 2021-01-15T16:42:32.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725629 '!=' null 2021-01-15T16:42:32.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:32.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:32.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:32.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:32.995 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:33.005 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:33.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:33.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:33.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:33.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:33.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:33.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:33.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:33.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:33.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:33.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:33.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:33.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:33.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:34.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:34.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:34.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:34.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:34.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:34.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:34.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:34.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:34.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:34.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725683 2021-01-15T16:42:34.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725683 '!=' null 2021-01-15T16:42:34.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:34.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:34.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:34.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:34.832 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:34.833 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:34.833 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:34.833 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:34.833 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:34.834 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:34.834 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:34.834 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:34.835 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:34.835 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:34.835 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:34.835 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:34.835 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:34.836 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:34.836 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:34.836 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:34.836 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:34.836 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:34.837 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:34.837 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:34.837 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:34.837 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:34.837 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:34.838 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:34.838 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:34.838 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:34.838 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:34.839 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:34.839 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:34.839 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:34.839 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:34.839 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:34.840 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:34.840 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:34.840 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:34.840 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:34.840 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:34.841 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:34.841 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:34.842 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:34.842 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:34.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:34.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:34.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:34.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:34.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:34.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:34.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:34.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:34.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:35.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:35.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:35.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:35.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:35.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:35.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588635 2021-01-15T16:42:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588635 '!=' null 2021-01-15T16:42:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:36.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:36.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:36.870 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:36.871 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:36.871 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:36.871 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:36.871 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:36.872 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:36.872 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:36.872 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:36.872 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:36.872 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:36.873 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:36.873 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:36.873 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:36.873 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:36.874 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:36.874 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:36.874 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:36.874 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:36.874 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:36.875 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:36.875 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:36.875 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:36.875 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:36.875 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:36.876 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:36.876 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:36.876 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:36.876 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:36.876 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:36.877 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:36.878 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:36.878 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:36.878 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:36.878 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:36.878 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:36.879 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:36.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:36.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:36.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:36.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:36.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:36.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:36.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:36.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:36.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:37.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:37.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:37.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:37.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:37.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:37.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:37.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:37.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:38.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852095 2021-01-15T16:42:38.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852095 '!=' null 2021-01-15T16:42:38.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:38.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:38.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:38.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:38.597 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:38.598 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:38.598 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:38.598 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:38.598 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:38.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:38.599 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:38.599 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:38.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:38.599 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:38.600 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:38.600 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:38.600 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:38.600 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:38.600 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:38.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:38.601 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 151, 2021-01-15T16:42:38.601 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:38.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:38.601 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:38.602 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:38.602 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:38.602 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:38.602 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:38.602 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:38.603 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:38.603 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:38.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:38.604 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T16:42:38.604 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 254, 2021-01-15T16:42:38.604 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:38.604 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:38.604 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:38.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:38.605 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:38.605 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:38.605 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:38.605 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:38.606 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:38.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:38.606 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:38.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:38.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:38.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:38.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:38.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:38.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:38.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:38.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:38.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:39.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:39.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:39.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:39.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:39.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:39.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:39.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:39.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:39.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:39.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:40.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714971 2021-01-15T16:42:40.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714971 '!=' null 2021-01-15T16:42:40.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:40.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:40.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:40.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:40.453 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:40.454 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:40.454 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:40.455 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:40.455 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:40.456 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:40.456 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:40.456 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:40.457 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:40.457 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:40.458 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:40.458 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:40.458 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:40.459 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:40.459 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:40.460 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:40.460 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:40.460 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:40.461 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:40.461 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:40.461 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:40.462 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:40.462 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:40.463 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:40.463 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:40.463 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:40.464 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:40.464 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:40.464 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:40.465 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:40.465 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:40.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:40.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:40.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:40.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:40.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:40.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:40.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:40.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:40.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:41.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:41.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:41.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:41.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:41.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:41.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:41.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:41.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:41.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:41.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:42.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588437 2021-01-15T16:42:42.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588437 '!=' null 2021-01-15T16:42:42.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:42.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:42.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:42.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:42.361 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:42.362 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:42.362 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:42.362 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:42.362 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:42.363 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:42.363 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:42.363 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:42.363 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:42.363 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:42.364 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:42.364 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:42.364 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:42.364 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:42.364 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:42.365 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:42.365 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:42.365 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:42.365 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:42.366 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:42.366 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:42.366 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:42.366 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:42.367 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:42.367 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:42.367 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:42.367 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:42.367 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:42.368 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:42.369 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:42.369 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:42.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:42.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:42.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:42.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:42.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:42.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:42.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:42.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:42.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:42.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:42.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:42.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:42.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:43.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:43.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:43.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:43.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:43.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:43.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:43.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:43.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:43.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:44.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11794315 2021-01-15T16:42:44.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11794315 '!=' null 2021-01-15T16:42:44.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:44.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:44.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:44.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:44.247 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:44.247 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:44.247 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:44.248 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:44.248 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:44.248 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:44.248 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:44.249 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:44.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:44.249 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:44.249 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:44.249 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:44.250 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:44.250 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:44.250 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:44.250 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:44.251 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:44.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:44.251 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:44.251 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:44.251 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:44.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:44.252 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:44.252 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:44.252 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:44.253 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:44.253 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:44.253 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:44.253 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:44.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:44.254 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:44.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:44.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:44.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:44.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:44.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:44.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:44.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:44.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:44.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:44.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:42:44.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:44.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:44.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:45.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T16:42:45.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:42:45.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:45.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:45.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:45.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19657172 2021-01-15T16:42:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19657172 '!=' null 2021-01-15T16:42:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:45.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:45.893 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:45.894 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:45.894 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:45.894 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:45.894 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:45.895 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:45.895 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:45.895 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:45.895 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:45.895 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:45.896 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:45.896 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:45.896 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:45.896 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:45.896 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:45.897 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:45.897 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:45.897 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:45.897 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:45.897 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:45.898 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:45.898 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:45.898 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:45.898 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:45.898 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:45.899 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:45.899 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:45.899 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:45.899 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:45.899 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:45.900 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:45.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:46.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:46.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:46.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:46.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:46.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:46.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:46.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:46.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:46.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:46.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:46.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:46.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:47.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:47.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2021-01-15T16:42:47.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:42:47.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2021-01-15T16:42:47.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:47.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:47.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:47.105 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:47.111 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:47.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:47.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:47.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:47.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:47.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:47.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:47.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:47.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:47.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:47.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:47.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:47.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:47.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:48.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:48.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:48.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:48.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:48.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:48.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7862793 2021-01-15T16:42:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7862793 '!=' null 2021-01-15T16:42:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:48.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:48.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:48.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:48.770 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:48.770 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:48.770 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:48.771 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:48.771 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:48.771 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:48.771 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:48.771 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:48.772 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:48.772 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:48.772 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:48.772 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:48.772 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:48.773 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:48.773 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:48.773 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:48.773 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:48.773 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:48.774 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:48.776 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:48.776 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:48.777 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:48.777 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:48.777 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:48.777 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:48.778 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:48.778 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:48.778 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:48.778 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:48.779 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:48.779 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:48.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:48.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:48.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:48.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:48.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:48.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:48.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:48.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:48.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:49.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:49.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:49.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:49.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:49.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:49.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:49.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:49.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:49.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:49.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:49.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:49.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:49.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:50.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18101478 2021-01-15T16:42:50.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18101478 '!=' null 2021-01-15T16:42:50.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:50.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:50.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:50.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:50.415 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:50.415 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:50.415 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:50.416 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:50.416 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:50.416 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:50.416 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:50.416 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:50.417 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:50.417 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:50.417 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:50.417 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:50.417 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:50.418 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:50.418 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:50.418 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:50.418 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:50.418 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:50.419 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:50.419 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:50.419 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:50.419 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:50.419 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:50.420 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:50.420 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:50.420 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:50.420 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:50.420 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:50.421 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:50.421 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:50.421 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:50.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:50.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:50.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:50.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:50.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:50.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:50.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:50.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:50.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:51.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:51.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:51.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:51.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:51.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:51.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:51.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:51.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:51.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15644211 2021-01-15T16:42:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15644211 '!=' null 2021-01-15T16:42:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:51.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:51.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:52.067 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:52.067 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:52.067 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:52.067 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:52.068 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:52.068 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:52.068 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:52.068 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:52.069 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:52.069 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:52.069 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:52.069 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:52.069 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:52.070 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:52.070 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:52.070 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:52.070 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:52.070 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:52.071 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:52.071 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:52.071 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:52.071 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:52.071 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:52.072 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:52.072 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:52.072 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:52.072 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:52.072 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:52.073 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:52.073 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:52.073 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:52.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:52.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:52.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:52.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:52.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:52.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:52.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:52.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:52.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:52.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:52.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:52.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:52.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:53.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:53.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:53.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104740 2021-01-15T16:42:53.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104740 '!=' null 2021-01-15T16:42:53.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:53.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:53.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:53.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:53.963 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:53.963 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:53.963 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:53.964 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:53.964 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:53.964 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:53.964 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:53.964 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:53.965 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:53.965 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:53.965 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:53.965 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:53.965 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:53.966 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:53.966 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:53.966 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:53.966 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:53.966 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:53.967 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:53.967 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:53.967 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:53.967 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:53.967 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:53.968 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:53.968 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:53.968 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:53.968 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:53.968 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:53.969 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:53.969 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:53.969 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:53.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:54.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:54.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:54.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:54.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:54.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:54.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:54.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:54.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:54.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:54.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:54.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:55.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:55.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:55.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:55.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:55.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:55.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:55.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967328 2021-01-15T16:42:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967328 '!=' null 2021-01-15T16:42:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:55.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:55.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:55.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:55.768 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:55.768 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:55.769 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:55.770 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:55.771 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:55.772 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:55.773 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:55.774 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:55.774 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:55.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:55.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:56.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:56.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:56.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:56.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:56.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:56.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:56.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:56.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:56.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:56.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:56.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:56.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:56.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:57.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967328 2021-01-15T16:42:57.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967328 '!=' null 2021-01-15T16:42:57.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:57.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:57.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:57.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:57.420 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:57.421 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:57.421 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:57.421 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:57.421 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:57.421 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:57.422 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:57.422 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:57.422 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:57.422 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:57.423 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:57.423 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:57.423 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:57.423 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:57.424 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:57.424 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:57.424 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:57.424 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:57.424 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:57.425 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:57.425 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:57.425 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:57.425 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:57.425 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:57.426 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:57.426 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:57.426 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:57.426 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:57.427 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:57.427 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:57.427 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:57.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:57.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:57.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:57.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:57.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:57.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:57.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:57.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:57.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:57.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:57.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:57.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:42:58.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:42:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:42:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:42:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:42:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:42:58.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:42:58.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:42:58.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:42:58.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:42:59.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725343 2021-01-15T16:42:59.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725343 '!=' null 2021-01-15T16:42:59.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:42:59.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:59.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:42:59.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:42:59.458 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:42:59.458 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:42:59.459 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:59.459 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:59.459 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:42:59.459 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:59.459 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:42:59.460 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:42:59.460 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:59.460 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:42:59.460 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:42:59.461 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:59.461 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:59.461 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:59.461 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:59.461 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:42:59.462 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:42:59.462 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:42:59.462 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:42:59.462 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:42:59.462 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:42:59.463 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:59.463 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:42:59.463 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:42:59.463 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:42:59.464 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:42:59.464 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:42:59.464 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:42:59.465 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:42:59.465 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:42:59.465 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:42:59.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:42:59.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:42:59.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:42:59.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:42:59.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:42:59.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:42:59.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:42:59.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:42:59.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:42:59.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:42:59.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:42:59.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:42:59.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:00.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:00.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:00.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:01.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714472 2021-01-15T16:43:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714472 '!=' null 2021-01-15T16:43:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:01.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:01.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:01.181 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:01.182 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:01.182 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:01.182 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:01.182 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:43:01.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:01.183 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:43:01.183 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:43:01.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:01.183 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:01.184 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:01.184 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:01.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:01.184 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:01.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:01.185 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:43:01.185 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:01.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:01.185 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:43:01.185 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:43:01.186 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:01.186 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:01.186 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:01.186 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:01.186 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:01.187 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:01.187 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:01.187 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:01.187 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:01.187 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:01.188 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:01.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:01.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:01.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:01.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:01.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:01.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:01.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:01.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:01.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:01.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:01.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:01.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:02.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:02.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:02.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:02.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:02.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:02.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:02.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:02.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:02.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:02.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851685 2021-01-15T16:43:02.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851685 '!=' null 2021-01-15T16:43:02.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:02.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:02.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:02.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:02.999 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:03.000 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:03.000 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:03.000 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:03.000 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:43:03.000 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:03.001 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:43:03.001 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:43:03.001 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:03.001 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:03.001 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:03.002 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:03.002 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:03.002 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:03.002 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:03.002 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:43:03.003 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:03.003 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:03.003 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:43:03.003 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:43:03.003 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:03.004 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:03.004 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:03.004 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:03.004 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:03.004 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:03.005 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:03.005 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:03.005 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:03.005 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:03.005 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:03.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:03.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:03.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:03.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:03.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:03.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:03.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:03.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:03.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:03.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:03.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:03.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:03.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:04.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:04.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:04.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:04.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:04.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:04.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:04.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:04.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:04.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:04.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851466 2021-01-15T16:43:04.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851466 '!=' null 2021-01-15T16:43:04.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:04.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:04.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:04.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:04.742 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:04.742 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:04.743 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:04.743 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:04.743 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:43:04.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:04.744 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:43:04.744 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:43:04.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:04.744 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:04.745 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:04.745 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:04.745 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:04.745 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:04.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:04.746 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:43:04.746 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:04.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:04.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:43:04.747 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:43:04.747 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:04.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:04.747 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:04.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:04.748 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:04.748 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:04.748 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:04.749 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:04.749 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:04.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:04.749 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:04.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:04.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:04.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:04.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:04.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:04.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:04.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:04.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:04.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:05.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:05.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:05.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:05.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:05.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:05.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:05.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:05.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:05.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:05.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:05.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:05.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:05.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714023 2021-01-15T16:43:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714023 '!=' null 2021-01-15T16:43:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:06.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:06.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:06.650 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:06.651 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:06.651 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:06.651 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:06.651 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:43:06.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:06.652 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:43:06.652 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:43:06.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:06.652 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:06.653 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:06.653 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:06.653 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:06.653 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:06.654 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:06.654 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:43:06.654 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:06.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:06.655 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:43:06.655 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:43:06.655 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:06.655 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:06.656 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:06.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:06.656 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:06.656 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:06.657 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:06.657 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:06.657 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:06.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:06.657 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:06.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:06.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:06.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:06.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:06.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:06.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:06.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:06.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:06.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:07.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:07.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:07.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:07.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:07.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:07.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:08.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851474 2021-01-15T16:43:08.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851474 '!=' null 2021-01-15T16:43:08.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:08.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:08.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:08.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:08.562 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:08.562 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:08.562 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:08.563 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:08.563 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T16:43:08.563 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:08.563 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 110, 2021-01-15T16:43:08.563 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T16:43:08.564 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:08.564 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:08.564 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:08.564 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:08.564 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:08.565 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:08.565 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:08.565 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:43:08.565 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:08.565 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:08.566 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T16:43:08.566 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 151, 2021-01-15T16:43:08.566 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:08.566 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:08.566 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:08.567 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:08.567 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:08.567 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:08.567 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:08.567 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:08.568 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:08.568 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:08.568 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:08.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:09.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:09.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:09.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:09.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:09.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:09.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:09.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:09.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:09.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:10.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714253 2021-01-15T16:43:10.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714253 '!=' null 2021-01-15T16:43:10.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:10.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:10.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:10.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:10.301 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:10.302 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:10.302 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:10.302 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:10.302 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:10.302 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:10.303 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:10.303 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:10.303 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:10.303 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:10.303 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:10.304 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:10.304 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:10.304 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:10.304 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:10.304 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:10.305 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:10.305 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:10.305 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:10.305 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:10.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:10.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:10.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:10.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:10.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:10.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:10.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:10.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:10.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:10.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:10.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:11.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:11.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:11.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:11.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:11.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:12.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588255 2021-01-15T16:43:12.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588255 '!=' null 2021-01-15T16:43:12.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:12.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:12.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:12.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:12.310 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:12.310 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:12.311 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:12.311 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:12.311 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:12.311 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:12.311 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:12.312 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:12.312 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:12.312 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:12.312 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:12.312 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:12.313 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:12.313 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:12.313 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:12.313 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:12.313 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:12.314 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:12.314 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:12.314 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:12.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:12.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:12.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:12.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:12.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:12.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:12.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:12.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:12.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:12.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:12.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:12.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:13.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:13.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:13.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:13.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:13.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:13.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:13.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:13.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:13.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:14.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11793866 2021-01-15T16:43:14.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11793866 '!=' null 2021-01-15T16:43:14.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:14.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:14.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:14.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:14.159 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:14.159 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:14.159 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:14.160 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:14.160 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:14.160 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:14.160 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:14.160 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:14.161 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:14.161 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:14.161 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:14.161 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:14.161 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:14.162 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:14.162 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:14.162 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:14.162 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:14.163 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:14.163 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:14.163 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:14.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:14.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:14.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:14.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:14.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:14.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:14.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:14.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:14.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:14.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:43:14.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:14.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:14.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:15.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T16:43:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T16:43:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:15.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:15.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:15.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:15.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967219 2021-01-15T16:43:15.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967219 '!=' null 2021-01-15T16:43:15.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:15.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:15.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:15.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:16.041 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:16.042 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:16.042 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:16.042 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:16.042 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:16.042 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:16.043 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:16.043 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:16.043 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:16.043 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:16.043 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:16.044 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:16.044 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:16.044 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:16.044 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:16.045 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:16.045 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:16.045 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:16.045 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:16.045 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:16.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:16.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:16.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:16.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:16.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:16.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:16.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:16.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:16.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:16.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:16.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 1 2021-01-15T16:43:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:43:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=2 2021-01-15T16:43:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:17.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:17.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:17.166 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:17.166 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:17.166 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:17.167 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:17.168 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:17.169 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:17.170 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:17.170 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:17.170 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:17.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:17.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:17.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:17.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:17.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:17.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:17.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:18.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:18.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:18.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:18.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:18.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:18.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:18.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:18.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:18.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:18.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:18.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:18.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:18.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:19.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9173167 2021-01-15T16:43:19.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9173167 '!=' null 2021-01-15T16:43:19.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:19.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:19.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:19.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:19.180 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:19.181 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:19.181 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:19.181 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:19.181 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:19.182 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:19.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:19.182 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:19.182 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:19.182 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:19.183 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:19.183 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:19.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:19.184 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:19.184 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:19.184 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:19.184 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:19.184 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:19.185 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:19.185 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:19.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:19.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:19.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:19.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:19.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:19.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:19.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:19.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:19.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:19.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:19.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:19.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:19.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:20.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:20.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:20.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:20.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:20.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:20.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:20.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:20.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18439968 2021-01-15T16:43:20.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18439968 '!=' null 2021-01-15T16:43:20.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:20.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:20.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:20.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:20.989 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:20.989 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:20.990 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:20.990 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:20.990 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:20.990 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:20.991 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:20.991 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:20.991 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:20.991 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:20.991 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:20.992 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:20.992 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:20.992 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:20.992 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:20.992 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:20.993 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:20.993 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:20.993 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:20.993 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:20.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:21.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:21.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:21.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:21.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:21.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:21.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:21.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:21.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:21.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:21.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:21.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:21.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:22.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:22.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:22.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15819422 2021-01-15T16:43:22.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15819422 '!=' null 2021-01-15T16:43:22.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:22.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:22.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:22.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:22.717 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:22.718 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:22.718 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:22.719 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:22.719 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:22.719 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:22.720 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:22.720 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:22.720 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:22.720 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:22.721 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:22.721 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:22.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:22.721 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:22.721 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:22.722 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:22.722 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:22.722 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:22.722 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:22.722 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:22.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:22.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:22.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:22.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:22.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:22.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:23.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:23.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:23.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:23.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:24.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:24.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:24.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:24.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:24.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:24.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:24.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:24.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:24.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:24.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978478 2021-01-15T16:43:24.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978478 '!=' null 2021-01-15T16:43:24.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:24.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:24.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:24.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:24.799 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:24.799 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:24.800 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:24.800 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:24.800 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:24.800 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:24.800 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:24.801 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:24.801 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:24.801 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:24.801 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:24.801 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:24.802 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:24.802 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:24.802 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:24.802 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:24.802 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:24.803 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:24.803 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:24.803 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:24.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:24.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:24.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:24.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:24.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:25.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:25.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:25.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:25.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:25.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:25.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:26.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20968145 2021-01-15T16:43:26.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20968145 '!=' null 2021-01-15T16:43:26.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:26.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:26.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:26.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:26.860 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:26.860 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:26.860 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:26.860 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:26.861 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:26.861 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:26.861 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:26.861 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:26.861 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:26.862 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:26.862 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:26.862 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:26.862 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:26.862 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:26.863 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:26.863 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:26.863 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:26.863 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:26.863 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:26.864 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:26.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:26.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:26.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:26.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:26.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:26.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:26.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:26.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:26.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:27.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:27.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:27.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:27.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:27.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:27.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:27.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:27.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:28.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852428 2021-01-15T16:43:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852428 '!=' null 2021-01-15T16:43:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:28.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:28.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:28.612 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:28.612 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:28.612 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:28.612 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:28.613 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:28.613 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:28.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:28.613 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:28.613 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:28.614 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:28.614 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:28.614 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:28.614 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:28.614 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:28.615 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:28.615 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:28.615 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:28.615 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:28.615 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:28.616 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:28.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:28.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:28.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:28.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:28.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:28.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:28.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:29.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:29.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:29.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:29.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:30.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:30.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:30.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:30.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:30.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:30.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:30.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:30.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:30.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841843 2021-01-15T16:43:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841843 '!=' null 2021-01-15T16:43:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:30.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:30.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:30.743 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:30.744 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:30.744 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:30.745 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:30.745 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:30.745 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:30.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:30.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:30.746 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:30.746 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:30.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:30.747 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:30.747 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:30.747 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:30.747 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:30.747 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:30.748 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:30.748 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:30.748 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:30.748 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:30.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:30.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:30.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:30.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:30.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:30.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:30.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:30.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:30.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:31.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:31.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:31.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:31.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:31.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:31.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:31.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:31.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:31.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:31.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:31.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:31.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:31.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:32.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978496 2021-01-15T16:43:32.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978496 '!=' null 2021-01-15T16:43:32.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:32.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:32.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:32.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:32.444 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:32.445 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:32.445 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:32.445 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:32.446 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:32.446 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:32.447 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:32.447 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:32.447 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:32.448 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:32.448 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:32.448 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:32.449 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:32.449 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:32.449 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:32.450 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:32.450 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:32.451 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:32.451 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:32.451 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:32.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:32.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:32.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:32.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:32.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:32.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:32.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:32.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:33.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:33.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:33.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:33.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:33.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:33.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:33.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:33.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:33.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:33.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:33.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:33.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:33.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:34.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978562 2021-01-15T16:43:34.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978562 '!=' null 2021-01-15T16:43:34.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:34.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:34.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:34.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:34.322 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:34.326 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:34.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:34.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:34.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:34.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:34.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:34.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:34.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:34.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:34.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:34.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:34.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:34.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:34.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:35.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:35.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:35.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:35.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841432 2021-01-15T16:43:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841432 '!=' null 2021-01-15T16:43:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:36.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:36.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:36.463 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:36.464 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:36.464 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:36.464 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:36.464 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:36.464 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:36.465 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:36.465 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:36.465 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:36.465 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:36.465 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:36.466 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:36.466 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:36.466 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:36.466 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:36.467 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:36.467 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:36.467 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:36.467 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:36.467 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:36.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:36.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:36.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:36.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:36.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:36.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:36.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:36.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:36.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:36.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:36.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:36.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:36.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:37.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:37.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:37.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:37.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:37.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:37.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:37.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:37.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:37.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:38.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852134 2021-01-15T16:43:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852134 '!=' null 2021-01-15T16:43:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:38.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:38.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:38.240 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:38.241 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:38.241 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:38.241 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:38.241 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:38.242 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:38.242 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:38.242 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:38.242 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:38.243 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:38.243 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:38.243 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:38.243 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:38.243 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:38.244 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:38.244 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:38.244 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:38.244 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:38.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:38.245 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:38.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:38.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:38.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:38.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:38.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:38.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:38.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:38.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:38.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:38.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:38.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:38.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:38.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:39.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:39.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:39.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:39.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:39.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841356 2021-01-15T16:43:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841356 '!=' null 2021-01-15T16:43:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:39.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:39.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:40.088 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:40.089 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:40.089 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:40.089 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:40.089 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:40.090 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:40.090 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:40.090 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:40.090 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:40.090 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:40.091 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:40.091 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:40.091 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:40.091 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:40.092 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:40.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:40.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:40.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:40.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:40.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:40.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:40.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:40.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:40.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:40.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:40.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:40.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:40.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:41.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:41.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:41.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:41.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:41.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:41.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:41.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852047 2021-01-15T16:43:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852047 '!=' null 2021-01-15T16:43:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:41.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:41.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:41.898 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T16:43:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T16:43:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 110, 2021-01-15T16:43:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T16:43:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:41.903 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:41.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:42.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:42.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:42.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:42.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:42.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:42.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:42.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:42.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:42.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:42.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:42.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:42.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:43.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:43.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:43.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:43.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:43.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:43.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:43.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483919 2021-01-15T16:43:43.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483919 '!=' null 2021-01-15T16:43:43.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:43.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:43.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:43.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:43.689 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:43.689 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:43.689 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:43.690 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:43.690 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:43.690 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:43.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:43.691 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:43.691 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:43.691 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:43.691 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:43.692 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:43.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:43.692 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:43.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:43.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:43.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:43.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:43.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:43.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:43.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:43.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:43.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:43:44.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:44.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:44.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:44.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:43:44.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T16:43:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:43:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:43:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:43:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:43:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:43:44.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:43:44.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:43:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483919 2021-01-15T16:43:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483919 '!=' null 2021-01-15T16:43:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:43:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:45.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T16:43:45.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T16:43:45.328 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:45.328 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T16:43:45.328 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:45.329 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:45.329 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:45.329 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:45.329 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:45.330 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T16:43:45.330 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T16:43:45.330 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T16:43:45.330 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T16:43:45.330 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T16:43:45.331 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:45.331 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:45.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:43:45.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:43:45.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:43:45.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:43:45.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:43:45.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:43:45.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:43:45.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:43:45.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:43:45.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T16:43:45.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:43:45.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:43:45.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:43:46.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T16:43:46.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:43:46.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:43:46.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2021-01-15T16:43:46.728 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:43:46.729 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:42:41.512646+0000 92'150 117:344 [1,4] 1 [1,4] 1 0'0 2021-01-15T16:39:12.788297+0000 0'0 2021-01-15T16:39:12.788297+0000 0 2021-01-15T16:43:46.729 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:43:42.982604+0000 92'150 118:356 [1,2] 1 [1,2] 1 0'0 2021-01-15T16:38:57.619195+0000 0'0 2021-01-15T16:38:57.619195+0000 0 2021-01-15T16:43:46.729 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-01-15T16:43:11.736094+0000 92'150 117:357 [1,2] 1 [1,2] 1 0'0 2021-01-15T16:39:02.666268+0000 0'0 2021-01-15T16:39:02.666268+0000 0 2021-01-15T16:43:46.729 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:43:46.730 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:43:46.730 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:43:46.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2021-01-15T16:43:46.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T16:43:46.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T16:43:46.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:43:46.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:46.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:43:46.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.92827/ceph-osd.1.asok 2021-01-15T16:43:46.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2021-01-15T16:43:46.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.92827/ceph-osd.1.asok dump_pgstate_history 2021-01-15T16:43:46.842 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:43:46.843 INFO:tasks.workunit.client.0.smithi145.stdout: "pgs": [ 2021-01-15T16:43:46.843 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.844 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "7.0", 2021-01-15T16:43:46.844 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.844 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T16:43:46.845 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.845 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "62", 2021-01-15T16:43:46.845 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.846 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.846 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.846 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.389457+0000", 2021-01-15T16:43:46.847 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.389466+0000" 2021-01-15T16:43:46.847 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.847 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.848 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.848 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.389472+0000", 2021-01-15T16:43:46.848 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.390093+0000" 2021-01-15T16:43:46.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.849 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.849 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.849 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.390094+0000", 2021-01-15T16:43:46.850 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.390163+0000" 2021-01-15T16:43:46.850 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.853 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.853 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.853 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.390163+0000", 2021-01-15T16:43:46.853 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.390184+0000" 2021-01-15T16:43:46.854 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.854 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.854 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.855 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.390185+0000", 2021-01-15T16:43:46.855 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.732940+0000" 2021-01-15T16:43:46.855 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.856 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.856 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.856 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.389469+0000", 2021-01-15T16:43:46.857 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:13.732962+0000" 2021-01-15T16:43:46.857 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.857 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.858 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.858 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.733172+0000", 2021-01-15T16:43:46.859 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.669516+0000" 2021-01-15T16:43:46.859 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.859 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.859 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.860 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.669517+0000", 2021-01-15T16:43:46.860 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.669562+0000" 2021-01-15T16:43:46.860 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.860 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.861 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.861 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.669562+0000", 2021-01-15T16:43:46.861 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.978801+0000" 2021-01-15T16:43:46.861 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.861 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.862 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.862 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.732963+0000", 2021-01-15T16:43:46.862 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.978820+0000" 2021-01-15T16:43:46.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.863 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.863 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.864 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.389466+0000", 2021-01-15T16:43:46.864 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.978840+0000" 2021-01-15T16:43:46.864 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.865 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.865 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.865 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:13.389453+0000", 2021-01-15T16:43:46.865 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.978843+0000" 2021-01-15T16:43:46.865 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.866 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.866 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.866 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.866 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "63", 2021-01-15T16:43:46.867 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.867 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.867 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.868 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.978844+0000", 2021-01-15T16:43:46.868 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.979140+0000" 2021-01-15T16:43:46.868 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.868 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.869 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.869 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.869 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "90", 2021-01-15T16:43:46.869 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.870 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.870 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.979144+0000", 2021-01-15T16:43:46.870 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:14.979156+0000" 2021-01-15T16:43:46.871 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.871 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.871 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.871 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.979163+0000", 2021-01-15T16:43:46.872 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.100720+0000" 2021-01-15T16:43:46.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.872 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.873 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.873 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.100721+0000", 2021-01-15T16:43:46.873 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.100778+0000" 2021-01-15T16:43:46.873 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.874 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.874 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.874 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.100778+0000", 2021-01-15T16:43:46.875 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.100789+0000" 2021-01-15T16:43:46.875 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.876 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.876 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.877 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.100789+0000", 2021-01-15T16:43:46.877 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.305672+0000" 2021-01-15T16:43:46.878 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.878 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.878 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.878 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.979160+0000", 2021-01-15T16:43:46.879 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.305681+0000" 2021-01-15T16:43:46.879 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.879 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.879 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.879 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.305835+0000", 2021-01-15T16:43:46.880 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.700082+0000" 2021-01-15T16:43:46.880 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.880 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.881 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.881 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.700082+0000", 2021-01-15T16:43:46.881 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:15.700132+0000" 2021-01-15T16:43:46.882 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.882 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.882 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.883 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.700132+0000", 2021-01-15T16:43:46.883 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.190985+0000" 2021-01-15T16:43:46.883 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.883 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.884 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.884 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:15.305681+0000", 2021-01-15T16:43:46.884 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191001+0000" 2021-01-15T16:43:46.884 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.885 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.885 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.885 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.979157+0000", 2021-01-15T16:43:46.886 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191014+0000" 2021-01-15T16:43:46.886 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.886 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.887 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.887 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:14.979140+0000", 2021-01-15T16:43:46.887 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191018+0000" 2021-01-15T16:43:46.888 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.888 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.888 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.889 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.889 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "90", 2021-01-15T16:43:46.889 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.890 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.890 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.890 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191018+0000", 2021-01-15T16:43:46.891 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191252+0000" 2021-01-15T16:43:46.891 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.891 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.891 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.891 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.892 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "97", 2021-01-15T16:43:46.892 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.892 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.892 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.893 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191255+0000", 2021-01-15T16:43:46.893 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191266+0000" 2021-01-15T16:43:46.893 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.893 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.894 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.894 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191273+0000", 2021-01-15T16:43:46.894 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191382+0000" 2021-01-15T16:43:46.894 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.895 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.895 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.895 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191382+0000", 2021-01-15T16:43:46.895 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191436+0000" 2021-01-15T16:43:46.895 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.896 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.896 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.896 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191436+0000", 2021-01-15T16:43:46.897 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.191449+0000" 2021-01-15T16:43:46.897 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.897 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.897 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.898 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191449+0000", 2021-01-15T16:43:46.898 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.743997+0000" 2021-01-15T16:43:46.898 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.899 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.899 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.899 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191270+0000", 2021-01-15T16:43:46.899 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.744008+0000" 2021-01-15T16:43:46.900 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.900 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.900 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.744073+0000", 2021-01-15T16:43:46.901 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.777489+0000" 2021-01-15T16:43:46.901 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.901 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.902 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.902 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.777490+0000", 2021-01-15T16:43:46.902 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:43.777528+0000" 2021-01-15T16:43:46.902 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.903 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.904 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.904 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.777528+0000", 2021-01-15T16:43:46.904 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.701553+0000" 2021-01-15T16:43:46.905 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.905 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.905 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.905 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.744009+0000", 2021-01-15T16:43:46.906 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.701573+0000" 2021-01-15T16:43:46.906 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.906 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.906 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.906 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191267+0000", 2021-01-15T16:43:46.907 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.701622+0000" 2021-01-15T16:43:46.907 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.907 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.907 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.907 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:43.191252+0000", 2021-01-15T16:43:46.908 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.701626+0000" 2021-01-15T16:43:46.908 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.908 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.909 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.909 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.909 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "97", 2021-01-15T16:43:46.910 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.910 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.911 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.911 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.701627+0000", 2021-01-15T16:43:46.911 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.702012+0000" 2021-01-15T16:43:46.912 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.912 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.912 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.913 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.913 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "99", 2021-01-15T16:43:46.914 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.914 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.914 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.915 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.702017+0000", 2021-01-15T16:43:46.915 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.702031+0000" 2021-01-15T16:43:46.916 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.916 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.916 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.917 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.702041+0000", 2021-01-15T16:43:46.917 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.810577+0000" 2021-01-15T16:43:46.917 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.918 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.918 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.919 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.810577+0000", 2021-01-15T16:43:46.919 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.810635+0000" 2021-01-15T16:43:46.919 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.920 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.920 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.921 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.702037+0000", 2021-01-15T16:43:46.921 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:17.810643+0000" 2021-01-15T16:43:46.921 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.922 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.922 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T16:43:46.922 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.810644+0000", 2021-01-15T16:43:46.923 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.195579+0000" 2021-01-15T16:43:46.923 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.924 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.924 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.924 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.702031+0000", 2021-01-15T16:43:46.925 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.195602+0000" 2021-01-15T16:43:46.925 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.926 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.926 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.926 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:17.702012+0000", 2021-01-15T16:43:46.927 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.195607+0000" 2021-01-15T16:43:46.927 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.927 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.928 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.928 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.929 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "99", 2021-01-15T16:43:46.929 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.929 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.930 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.930 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195607+0000", 2021-01-15T16:43:46.931 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.195892+0000" 2021-01-15T16:43:46.931 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.932 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.932 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.932 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.933 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "113", 2021-01-15T16:43:46.933 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.934 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.934 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195897+0000", 2021-01-15T16:43:46.935 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.195910+0000" 2021-01-15T16:43:46.935 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.935 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.936 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.936 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195918+0000", 2021-01-15T16:43:46.937 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.196415+0000" 2021-01-15T16:43:46.937 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.937 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.938 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.938 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.196415+0000", 2021-01-15T16:43:46.939 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.196485+0000" 2021-01-15T16:43:46.939 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.939 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.940 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.940 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.196486+0000", 2021-01-15T16:43:46.940 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.196503+0000" 2021-01-15T16:43:46.941 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.941 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.941 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.942 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.196504+0000", 2021-01-15T16:43:46.942 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.497806+0000" 2021-01-15T16:43:46.943 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.943 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.943 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.944 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195915+0000", 2021-01-15T16:43:46.944 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.497826+0000" 2021-01-15T16:43:46.945 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.945 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.945 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.946 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.498205+0000", 2021-01-15T16:43:46.946 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.900079+0000" 2021-01-15T16:43:46.947 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.947 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.948 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-01-15T16:43:46.948 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.900080+0000", 2021-01-15T16:43:46.949 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.900204+0000" 2021-01-15T16:43:46.949 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.949 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.950 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-01-15T16:43:46.950 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.900204+0000", 2021-01-15T16:43:46.950 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:18.900580+0000" 2021-01-15T16:43:46.951 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.951 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.952 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2021-01-15T16:43:46.952 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.900580+0000", 2021-01-15T16:43:46.952 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:53.504476+0000" 2021-01-15T16:43:46.953 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.953 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.954 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotBackfilling", 2021-01-15T16:43:46.954 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:53.504476+0000", 2021-01-15T16:43:46.954 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:53.504782+0000" 2021-01-15T16:43:46.955 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.955 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.955 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-01-15T16:43:46.956 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:53.504783+0000", 2021-01-15T16:43:46.956 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:06.078199+0000" 2021-01-15T16:43:46.957 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.957 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.957 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-01-15T16:43:46.958 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:06.078200+0000", 2021-01-15T16:43:46.958 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:06.078612+0000" 2021-01-15T16:43:46.959 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.959 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.959 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2021-01-15T16:43:46.960 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:06.078612+0000", 2021-01-15T16:43:46.960 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.349606+0000" 2021-01-15T16:43:46.961 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.961 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.961 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.962 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:40.349607+0000", 2021-01-15T16:43:46.962 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.349767+0000" 2021-01-15T16:43:46.963 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.963 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.963 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.964 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:40.349767+0000", 2021-01-15T16:43:46.964 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.743157+0000" 2021-01-15T16:43:46.964 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.965 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.965 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.965 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.497827+0000", 2021-01-15T16:43:46.965 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.743185+0000" 2021-01-15T16:43:46.965 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.966 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.966 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.966 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195911+0000", 2021-01-15T16:43:46.966 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.743216+0000" 2021-01-15T16:43:46.966 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.967 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.967 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.967 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:18.195893+0000", 2021-01-15T16:43:46.967 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.743226+0000" 2021-01-15T16:43:46.967 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "113", 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.968 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.969 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.969 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:40.743227+0000", 2021-01-15T16:43:46.969 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.743596+0000" 2021-01-15T16:43:46.969 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.969 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.970 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.970 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.970 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.970 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.970 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "3.0", 2021-01-15T16:43:46.971 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.971 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T16:43:46.971 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.971 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "42", 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:02.947061+0000", 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:02.947076+0000" 2021-01-15T16:43:46.972 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.973 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.973 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.973 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:02.947089+0000", 2021-01-15T16:43:46.973 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.139945+0000" 2021-01-15T16:43:46.973 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.974 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.974 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.974 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.139946+0000", 2021-01-15T16:43:46.974 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.140013+0000" 2021-01-15T16:43:46.974 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.975 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.975 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.975 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.140013+0000", 2021-01-15T16:43:46.975 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.140024+0000" 2021-01-15T16:43:46.975 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.976 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.976 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.976 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.140025+0000", 2021-01-15T16:43:46.976 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.432761+0000" 2021-01-15T16:43:46.976 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.977 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.977 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.977 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:02.947083+0000", 2021-01-15T16:43:46.977 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.432770+0000" 2021-01-15T16:43:46.977 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.978 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.978 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.978 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.432945+0000", 2021-01-15T16:43:46.978 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.826978+0000" 2021-01-15T16:43:46.978 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.979 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.979 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.979 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.826979+0000", 2021-01-15T16:43:46.979 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:03.827027+0000" 2021-01-15T16:43:46.979 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.980 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.980 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.980 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.827027+0000", 2021-01-15T16:43:46.980 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438069+0000" 2021-01-15T16:43:46.980 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.981 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.981 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.981 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:03.432770+0000", 2021-01-15T16:43:46.981 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438088+0000" 2021-01-15T16:43:46.981 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.982 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.982 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.982 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:02.947076+0000", 2021-01-15T16:43:46.982 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438106+0000" 2021-01-15T16:43:46.982 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:02.947058+0000", 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438109+0000" 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.983 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.984 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.984 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.984 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "42", 2021-01-15T16:43:46.984 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.984 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.985 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.985 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438109+0000", 2021-01-15T16:43:46.985 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438363+0000" 2021-01-15T16:43:46.985 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.985 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.986 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.986 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.986 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "87", 2021-01-15T16:43:46.986 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.986 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.987 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:46.987 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438367+0000", 2021-01-15T16:43:46.987 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.438377+0000" 2021-01-15T16:43:46.987 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.987 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.988 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:46.988 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438386+0000", 2021-01-15T16:43:46.988 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.571283+0000" 2021-01-15T16:43:46.988 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.988 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.571283+0000", 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.571345+0000" 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.989 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:46.990 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.571345+0000", 2021-01-15T16:43:46.990 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.571357+0000" 2021-01-15T16:43:46.990 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.990 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.990 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:46.991 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.571357+0000", 2021-01-15T16:43:46.991 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.807170+0000" 2021-01-15T16:43:46.991 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.991 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.991 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:46.992 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438382+0000", 2021-01-15T16:43:46.992 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:04.807189+0000" 2021-01-15T16:43:46.992 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.992 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.992 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:46.993 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.807347+0000", 2021-01-15T16:43:46.993 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:05.159036+0000" 2021-01-15T16:43:46.993 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.993 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.993 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:46.994 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:05.159036+0000", 2021-01-15T16:43:46.994 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:05.159080+0000" 2021-01-15T16:43:46.994 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.994 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.994 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:46.995 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:05.159080+0000", 2021-01-15T16:43:46.995 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236039+0000" 2021-01-15T16:43:46.995 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.995 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.995 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:46.996 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.807190+0000", 2021-01-15T16:43:46.996 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236055+0000" 2021-01-15T16:43:46.996 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.996 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.996 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:46.997 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438378+0000", 2021-01-15T16:43:46.997 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236073+0000" 2021-01-15T16:43:46.997 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.997 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.997 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:46.998 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:04.438363+0000", 2021-01-15T16:43:46.998 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236078+0000" 2021-01-15T16:43:46.998 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:46.998 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:46.998 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "87", 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:46.999 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236078+0000", 2021-01-15T16:43:47.000 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236352+0000" 2021-01-15T16:43:47.000 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.000 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.000 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.000 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.001 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "105", 2021-01-15T16:43:47.001 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.001 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.001 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.001 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236357+0000", 2021-01-15T16:43:47.002 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.236369+0000" 2021-01-15T16:43:47.002 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.002 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.002 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.002 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236377+0000", 2021-01-15T16:43:47.003 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.420410+0000" 2021-01-15T16:43:47.003 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.003 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.003 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.003 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.420410+0000", 2021-01-15T16:43:47.004 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.420465+0000" 2021-01-15T16:43:47.004 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.004 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.004 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:47.004 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.420466+0000", 2021-01-15T16:43:47.005 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.420478+0000" 2021-01-15T16:43:47.005 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.005 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.005 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:47.005 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.420478+0000", 2021-01-15T16:43:47.006 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.843095+0000" 2021-01-15T16:43:47.006 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.006 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.006 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.006 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236374+0000", 2021-01-15T16:43:47.007 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.843103+0000" 2021-01-15T16:43:47.007 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.007 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.007 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:47.007 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.843192+0000", 2021-01-15T16:43:47.008 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.908345+0000" 2021-01-15T16:43:47.008 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.008 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.008 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:47.008 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.908345+0000", 2021-01-15T16:43:47.009 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:42.908389+0000" 2021-01-15T16:43:47.009 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.009 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.009 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.009 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.908389+0000", 2021-01-15T16:43:47.010 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950196+0000" 2021-01-15T16:43:47.010 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.010 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.010 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:47.010 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.843104+0000", 2021-01-15T16:43:47.011 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950226+0000" 2021-01-15T16:43:47.011 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.011 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.011 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.012 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236370+0000", 2021-01-15T16:43:47.012 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950289+0000" 2021-01-15T16:43:47.012 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.012 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.013 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.013 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:42.236352+0000", 2021-01-15T16:43:47.013 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950293+0000" 2021-01-15T16:43:47.014 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.014 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.014 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.015 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.015 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "105", 2021-01-15T16:43:47.016 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.016 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.016 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.017 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950293+0000", 2021-01-15T16:43:47.017 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950656+0000" 2021-01-15T16:43:47.018 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.018 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.018 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.019 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.019 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "107", 2021-01-15T16:43:47.020 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.020 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.020 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.021 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950660+0000", 2021-01-15T16:43:47.021 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.950671+0000" 2021-01-15T16:43:47.022 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.022 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.022 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.023 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950678+0000", 2021-01-15T16:43:47.023 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.951178+0000" 2021-01-15T16:43:47.024 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.024 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.024 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.025 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.951179+0000", 2021-01-15T16:43:47.025 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.951229+0000" 2021-01-15T16:43:47.025 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.026 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.026 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.027 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950675+0000", 2021-01-15T16:43:47.027 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:40.951237+0000" 2021-01-15T16:43:47.027 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.028 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.028 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T16:43:47.029 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.951237+0000", 2021-01-15T16:43:47.029 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.218127+0000" 2021-01-15T16:43:47.029 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.030 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.030 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.031 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950672+0000", 2021-01-15T16:43:47.031 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.218146+0000" 2021-01-15T16:43:47.031 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.032 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.032 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.033 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:40.950656+0000", 2021-01-15T16:43:47.033 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.218150+0000" 2021-01-15T16:43:47.033 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.034 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.034 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.035 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.035 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "107", 2021-01-15T16:43:47.035 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.036 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.036 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218150+0000", 2021-01-15T16:43:47.037 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.218392+0000" 2021-01-15T16:43:47.037 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.037 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.038 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.038 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.038 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "115", 2021-01-15T16:43:47.038 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.039 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.039 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.039 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218396+0000", 2021-01-15T16:43:47.039 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.218408+0000" 2021-01-15T16:43:47.039 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.040 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.040 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.040 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218433+0000", 2021-01-15T16:43:47.040 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.352098+0000" 2021-01-15T16:43:47.040 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.041 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.041 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.041 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.352099+0000", 2021-01-15T16:43:47.041 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.352179+0000" 2021-01-15T16:43:47.041 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.042 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.042 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:47.042 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.352180+0000", 2021-01-15T16:43:47.042 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.352200+0000" 2021-01-15T16:43:47.042 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.043 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.043 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:47.043 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.352200+0000", 2021-01-15T16:43:47.043 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.546553+0000" 2021-01-15T16:43:47.043 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.044 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.044 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.044 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218429+0000", 2021-01-15T16:43:47.044 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:41.546566+0000" 2021-01-15T16:43:47.044 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.045 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.045 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:47.045 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.546856+0000", 2021-01-15T16:43:47.045 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:42.647125+0000" 2021-01-15T16:43:47.045 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.046 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.046 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-01-15T16:43:47.046 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:42.647126+0000", 2021-01-15T16:43:47.046 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:53.504611+0000" 2021-01-15T16:43:47.046 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.047 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.047 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-01-15T16:43:47.047 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:53.504612+0000", 2021-01-15T16:43:47.047 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:53.505057+0000" 2021-01-15T16:43:47.047 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:53.505058+0000", 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:06.078129+0000" 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.048 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.049 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotBackfilling", 2021-01-15T16:43:47.049 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:06.078130+0000", 2021-01-15T16:43:47.049 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:06.078305+0000" 2021-01-15T16:43:47.049 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.049 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.050 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-01-15T16:43:47.050 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:06.078309+0000", 2021-01-15T16:43:47.050 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.349951+0000" 2021-01-15T16:43:47.050 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.050 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.051 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-01-15T16:43:47.051 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:40.349951+0000", 2021-01-15T16:43:47.051 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:42:40.350354+0000" 2021-01-15T16:43:47.051 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.051 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.052 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2021-01-15T16:43:47.052 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:42:40.350354+0000", 2021-01-15T16:43:47.052 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:09.997585+0000" 2021-01-15T16:43:47.052 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.052 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.053 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:47.053 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:09.997586+0000", 2021-01-15T16:43:47.053 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:09.997742+0000" 2021-01-15T16:43:47.053 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.053 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.054 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.054 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:09.997743+0000", 2021-01-15T16:43:47.054 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:10.331366+0000" 2021-01-15T16:43:47.054 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.054 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.055 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:47.055 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.546566+0000", 2021-01-15T16:43:47.055 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:10.331393+0000" 2021-01-15T16:43:47.055 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.055 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.056 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.056 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218409+0000", 2021-01-15T16:43:47.056 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:10.331427+0000" 2021-01-15T16:43:47.056 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.056 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:41.218392+0000", 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:10.331433+0000" 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.057 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.058 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.058 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "115", 2021-01-15T16:43:47.058 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.058 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.058 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.059 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:10.331434+0000", 2021-01-15T16:43:47.059 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:10.331777+0000" 2021-01-15T16:43:47.059 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.059 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.059 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.060 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.060 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.060 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.060 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "1.0", 2021-01-15T16:43:47.060 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.061 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T16:43:47.061 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.061 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "27", 2021-01-15T16:43:47.061 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.061 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.062 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Initial", 2021-01-15T16:43:47.062 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.920846+0000", 2021-01-15T16:43:47.062 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:57.920991+0000" 2021-01-15T16:43:47.062 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.062 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.063 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.063 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.063 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "27", 2021-01-15T16:43:47.063 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.063 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.064 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.064 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.920992+0000", 2021-01-15T16:43:47.064 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:57.921023+0000" 2021-01-15T16:43:47.064 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.064 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.065 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.065 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.065 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "30", 2021-01-15T16:43:47.065 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.065 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.066 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.066 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.921026+0000", 2021-01-15T16:43:47.066 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:57.921053+0000" 2021-01-15T16:43:47.066 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.066 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.067 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.067 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.921059+0000", 2021-01-15T16:43:47.067 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.021674+0000" 2021-01-15T16:43:47.067 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.067 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.068 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.068 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.021675+0000", 2021-01-15T16:43:47.068 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.021787+0000" 2021-01-15T16:43:47.068 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.068 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.069 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:47.069 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.021787+0000", 2021-01-15T16:43:47.069 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.021810+0000" 2021-01-15T16:43:47.069 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.069 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.070 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:47.070 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.021811+0000", 2021-01-15T16:43:47.070 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.193394+0000" 2021-01-15T16:43:47.070 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.070 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.071 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.071 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.921056+0000", 2021-01-15T16:43:47.071 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.193408+0000" 2021-01-15T16:43:47.071 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.071 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.072 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:47.072 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.193722+0000", 2021-01-15T16:43:47.072 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.575054+0000" 2021-01-15T16:43:47.072 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.072 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.575054+0000", 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:58.575100+0000" 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.073 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.074 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.575100+0000", 2021-01-15T16:43:47.074 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469488+0000" 2021-01-15T16:43:47.074 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.074 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.075 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:47.075 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:58.193409+0000", 2021-01-15T16:43:47.075 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469507+0000" 2021-01-15T16:43:47.075 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.075 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.076 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.076 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.921053+0000", 2021-01-15T16:43:47.076 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469527+0000" 2021-01-15T16:43:47.076 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.076 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.077 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.077 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:57.921023+0000", 2021-01-15T16:43:47.077 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469530+0000" 2021-01-15T16:43:47.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.077 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.078 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.078 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "30", 2021-01-15T16:43:47.078 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469530+0000", 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469834+0000" 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.079 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.080 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2021-01-15T16:43:47.080 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.080 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.081 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469839+0000", 2021-01-15T16:43:47.081 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.469850+0000" 2021-01-15T16:43:47.081 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.081 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.082 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469858+0000", 2021-01-15T16:43:47.082 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.612553+0000" 2021-01-15T16:43:47.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.082 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.083 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.612554+0000", 2021-01-15T16:43:47.083 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.612680+0000" 2021-01-15T16:43:47.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.083 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.083 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:47.084 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.612681+0000", 2021-01-15T16:43:47.084 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.612705+0000" 2021-01-15T16:43:47.084 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.084 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.612705+0000", 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.988958+0000" 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.085 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469855+0000", 2021-01-15T16:43:47.086 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:38:59.988966+0000" 2021-01-15T16:43:47.086 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.086 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.086 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:47.086 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.989116+0000", 2021-01-15T16:43:47.087 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:00.442521+0000" 2021-01-15T16:43:47.087 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.087 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.087 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:47.087 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:00.442522+0000", 2021-01-15T16:43:47.088 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:39:00.442567+0000" 2021-01-15T16:43:47.088 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.088 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.088 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.088 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:39:00.442567+0000", 2021-01-15T16:43:47.089 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.666838+0000" 2021-01-15T16:43:47.089 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.089 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.089 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:47.089 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.988966+0000", 2021-01-15T16:43:47.090 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.666863+0000" 2021-01-15T16:43:47.090 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.090 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.090 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.090 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469851+0000", 2021-01-15T16:43:47.091 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.666915+0000" 2021-01-15T16:43:47.091 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.091 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.091 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.091 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:38:59.469834+0000", 2021-01-15T16:43:47.092 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.666919+0000" 2021-01-15T16:43:47.092 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.092 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.092 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.092 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.093 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2021-01-15T16:43:47.093 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.093 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.093 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.093 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.666919+0000", 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.667289+0000" 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.094 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "103", 2021-01-15T16:43:47.095 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.095 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.095 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.095 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.667294+0000", 2021-01-15T16:43:47.095 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.667308+0000" 2021-01-15T16:43:47.096 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.096 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.096 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.096 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.667318+0000", 2021-01-15T16:43:47.096 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.818045+0000" 2021-01-15T16:43:47.097 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.097 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.097 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.097 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.818046+0000", 2021-01-15T16:43:47.097 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.818106+0000" 2021-01-15T16:43:47.098 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.098 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.098 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.098 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.667313+0000", 2021-01-15T16:43:47.098 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:32.818115+0000" 2021-01-15T16:43:47.099 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.099 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.099 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T16:43:47.099 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.818115+0000", 2021-01-15T16:43:47.099 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.245161+0000" 2021-01-15T16:43:47.100 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.100 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.100 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.100 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.667309+0000", 2021-01-15T16:43:47.100 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.245183+0000" 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:32.667289+0000", 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.245187+0000" 2021-01-15T16:43:47.101 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.102 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.102 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.102 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "103", 2021-01-15T16:43:47.102 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.103 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.103 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.103 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245187+0000", 2021-01-15T16:43:47.103 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.245479+0000" 2021-01-15T16:43:47.103 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.104 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.104 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.104 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.104 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "117", 2021-01-15T16:43:47.104 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T16:43:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245483+0000", 2021-01-15T16:43:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.245498+0000" 2021-01-15T16:43:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T16:43:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245507+0000", 2021-01-15T16:43:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.367675+0000" 2021-01-15T16:43:47.106 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T16:43:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.367676+0000", 2021-01-15T16:43:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.367752+0000" 2021-01-15T16:43:47.107 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T16:43:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.367752+0000", 2021-01-15T16:43:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.367771+0000" 2021-01-15T16:43:47.108 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T16:43:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.367771+0000", 2021-01-15T16:43:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.639262+0000" 2021-01-15T16:43:47.109 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T16:43:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245504+0000", 2021-01-15T16:43:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:33.639275+0000" 2021-01-15T16:43:47.110 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T16:43:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.639598+0000", 2021-01-15T16:43:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:41:34.158411+0000" 2021-01-15T16:43:47.111 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:34.158412+0000", 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:09.997926+0000" 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.112 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.113 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-01-15T16:43:47.113 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:09.997926+0000", 2021-01-15T16:43:47.113 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:09.998414+0000" 2021-01-15T16:43:47.113 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.113 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.114 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Backfilling", 2021-01-15T16:43:47.114 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:09.998418+0000", 2021-01-15T16:43:47.114 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.220927+0000" 2021-01-15T16:43:47.114 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.114 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.115 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T16:43:47.115 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:42.220927+0000", 2021-01-15T16:43:47.115 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.221216+0000" 2021-01-15T16:43:47.115 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.115 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.116 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T16:43:47.116 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:42.221216+0000", 2021-01-15T16:43:47.116 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.412537+0000" 2021-01-15T16:43:47.116 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.116 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.117 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T16:43:47.117 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.639275+0000", 2021-01-15T16:43:47.117 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.412573+0000" 2021-01-15T16:43:47.117 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.117 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.118 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T16:43:47.118 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245498+0000", 2021-01-15T16:43:47.118 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.412615+0000" 2021-01-15T16:43:47.118 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.118 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.119 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T16:43:47.119 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:41:33.245479+0000", 2021-01-15T16:43:47.119 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.412625+0000" 2021-01-15T16:43:47.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.119 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "117", 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:43:47.120 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T16:43:47.121 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T16:43:42.412625+0000", 2021-01-15T16:43:47.121 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T16:43:42.413292+0000" 2021-01-15T16:43:47.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.121 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.121 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.122 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.122 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:43:47.122 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:43:47.122 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:43:47.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2021-01-15T16:43:47.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2021-01-15T16:43:47.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2021-01-15T16:43:47.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T16:43:47.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T16:43:47.124 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2021-01-15T16:43:47.360 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T16:43:47.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2021-01-15T16:43:47.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T16:43:47.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T16:43:47.972 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T16:43:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2021-01-15T16:43:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2021-01-15T16:43:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-01-15T16:43:48.582 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' does not exist 2021-01-15T16:43:48.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2021-01-15T16:43:48.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:43:48.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:43:48.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:43:48.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:43:48.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:43:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:43:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2021-01-15T16:43:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2021-01-15T16:43:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-01-15T16:43:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:43:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-01-15T16:43:48.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:43:48.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:43:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:43:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:43:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:43:48.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:43:48.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:43:48.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:43:48.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:43:48.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:43:48.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:43:48.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:43:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:43:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:43:48.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:43:48.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:43:48.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-01-15T16:43:48.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:43:48.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:48.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:43:48.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92827 2021-01-15T16:43:48.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:43:48.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:43:48.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T16:43:48.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-prio 0 2021-01-15T16:43:48.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-01-15T16:43:48.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T16:43:48.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-01-15T16:43:48.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:43:48.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:43:48.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:43:48.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:43:48.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:43:48.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:43:48.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:43:48.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:43:48.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:43:48.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:43:48.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:43:48.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:43:48.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:43:48.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:43:48.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:43:48.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T16:43:48.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-01-15T16:43:48.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:43:48.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:48.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.92827 2021-01-15T16:43:48.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.92827 2021-01-15T16:43:48.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:43:48.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:43:48.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T16:43:48.753 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T16:43:48.753 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T16:43:48.785 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2021-01-15T16:43:48.786 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2021-01-15T16:43:48.845 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:43:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:43:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:43:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:43:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:43:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:43:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:43:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:43:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:43:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:43:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-recovery-log 2021-01-15T16:43:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2021-01-15T16:43:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2021-01-15T16:43:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2021-01-15T16:43:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2021-01-15T16:43:48.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2021-01-15T16:43:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none ' 2021-01-15T16:43:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2021-01-15T16:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2021-01-15T16:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_log_2 2021-01-15T16:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_1 2021-01-15T16:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_2' 2021-01-15T16:43:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-01-15T16:43:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-01-15T16:43:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-01-15T16:43:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:43:48.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:43:48.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:43:48.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:43:48.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:43:48.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:43:48.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:43:48.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:43:48.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:43:48.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:43:48.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:43:48.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:43:48.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:43:48.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:43:48.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:43:48.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:43:48.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:43:48.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:43:48.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:43:48.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:43:48.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:43:48.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:48.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:48.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:43:48.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:43:48.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:43:48.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-01-15T16:43:48.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:43:48.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:48.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:48.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119796 2021-01-15T16:43:48.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:43:48.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:43:48.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:43:48.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:43:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-01-15T16:43:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2021-01-15T16:43:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:107: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2021-01-15T16:43:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:43:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2021-01-15T16:43:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2021-01-15T16:43:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2021-01-15T16:43:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-01-15T16:43:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-01-15T16:43:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-01-15T16:43:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:43:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:43:48.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:43:48.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-01-15T16:43:48.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-01-15T16:43:49.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:43:49.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:43:49.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:43:49.194 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:43:49.194 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:49.194 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:49.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:49.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:43:49.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:43:49.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:43:49.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:43:49.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:43:49.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:43:49.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:43:49.389 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:43:49.389 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:43:49.389 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:43:49.390 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:43:49.390 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:49.391 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:49.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:43:49.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:43:49.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get fsid 2021-01-15T16:43:49.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:43:49.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:43:49.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:43:49.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:43:49.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:43:49.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:43:49.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:43:49.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:43:49.490 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:43:49.490 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:49.490 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:49.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:43:49.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:43:49.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get mon_host 2021-01-15T16:43:49.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-01-15T16:43:49.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:49.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:43:49.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:43:49.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:43:49.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-01-15T16:43:49.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:43:50.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:43:50.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:43:50.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:43:50.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:43:50.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:50.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:50.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:50.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:43:50.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:43:50.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-01-15T16:43:50.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:43:50.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:43:50.511 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-01-15T16:43:50.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-01-15T16:43:50.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:43:50.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-01-15T16:43:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:43:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:43:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:43:50.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-01-15T16:43:50.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:43:50.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:43:50.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:43:50.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:43:50.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-01-15T16:43:50.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-01-15T16:43:50.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:43:50.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:43:50.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:43:50.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:43:50.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:43:50.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:43:50.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:43:50.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:50.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:50.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:50.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:50.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:43:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:43:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:43:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:43:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:43:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:43:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:43:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:43:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:43:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:43:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-01-15T16:43:50.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:43:50.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fb969791-a661-46dd-8347-2412a3e7e347 2021-01-15T16:43:50.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 fb969791-a661-46dd-8347-2412a3e7e347' 2021-01-15T16:43:50.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:43:50.523 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 fb969791-a661-46dd-8347-2412a3e7e347 2021-01-15T16:43:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBGxgFghWaVHxAACuRZ0LDDpP9aAFPv6Ip7LA== 2021-01-15T16:43:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBGxgFghWaVHxAACuRZ0LDDpP9aAFPv6Ip7LA=="}' 2021-01-15T16:43:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fb969791-a661-46dd-8347-2412a3e7e347 -i td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:43:50.869 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:43:50.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:43:50.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBGxgFghWaVHxAACuRZ0LDDpP9aAFPv6Ip7LA== --osd-uuid fb969791-a661-46dd-8347-2412a3e7e347 2021-01-15T16:43:50.907 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.906+0000 7faa603fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:50.908 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.908+0000 7faa603fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:50.909 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.909+0000 7faa603fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.909+0000 7faa603fe080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:43:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.909+0000 7faa603fe080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:43:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.909+0000 7faa603fe080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:43:50.951 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:50.951+0000 7faa603fe080 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-01-15T16:43:55.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-01-15T16:43:55.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:43:55.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:43:55.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:43:55.478 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:43:56.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:43:56.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:43:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:43:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:43:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:43:56.004 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:43:56.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:56.026+0000 7f87b2a8e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:56.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:56.060+0000 7f87b2a8e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:56.062 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:56.062+0000 7f87b2a8e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:56.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:43:56.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:43:56.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:43:56.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:43:56.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:43:56.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:43:56.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:43:56.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:43:56.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:43:56.398 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:43:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:56.580+0000 7f87b2a8e080 -1 Falling back to public interface 2021-01-15T16:43:56.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:43:57.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:57.429+0000 7f87b2a8e080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:43:57.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:43:57.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:43:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:43:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:43:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:43:57.787 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:43:58.241 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/120838,v1:127.0.0.1:6803/120838] [v2:127.0.0.1:6804/120838,v1:127.0.0.1:6805/120838] exists,up fb969791-a661-46dd-8347-2412a3e7e347 2021-01-15T16:43:58.241 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 ebb5cd35-d98b-4a22-9cb2-821b9236416e 2021-01-15T16:43:58.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:43:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:43:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:43:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:43:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-01-15T16:43:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:43:58.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:43:58.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:43:58.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:43:58.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-01-15T16:43:58.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:43:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:43:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:43:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:43:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-01-15T16:43:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-01-15T16:43:58.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:43:58.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:43:58.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:43:58.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:43:58.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:43:58.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:43:58.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:43:58.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:43:58.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:43:58.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:58.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:43:58.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:43:58.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:43:58.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:43:58.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:43:58.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:43:58.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:43:58.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:43:58.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:43:58.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:43:58.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:43:58.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-01-15T16:43:58.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:43:58.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ebb5cd35-d98b-4a22-9cb2-821b9236416e 2021-01-15T16:43:58.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ebb5cd35-d98b-4a22-9cb2-821b9236416e' 2021-01-15T16:43:58.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:43:58.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBOxgFg4Nw0DxAAeQR5/uieF5sXhCcmmnLFxw== 2021-01-15T16:43:58.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBOxgFg4Nw0DxAAeQR5/uieF5sXhCcmmnLFxw=="}' 2021-01-15T16:43:58.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ebb5cd35-d98b-4a22-9cb2-821b9236416e -i td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:43:58.820 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:43:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:43:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOxgFg4Nw0DxAAeQR5/uieF5sXhCcmmnLFxw== --osd-uuid ebb5cd35-d98b-4a22-9cb2-821b9236416e 2021-01-15T16:43:58.857 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.857+0000 7f13b858e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:58.859 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.859+0000 7f13b858e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:58.860 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.860+0000 7f13b858e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:43:58.860 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.860+0000 7f13b858e080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:43:58.861 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.860+0000 7f13b858e080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:43:58.861 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.860+0000 7f13b858e080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:43:58.893 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:43:58.892+0000 7f13b858e080 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-01-15T16:44:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-01-15T16:44:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:44:03.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:44:03.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:44:03.771 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:44:04.296 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:44:04.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:44:04.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:04.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:44:04.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:44:04.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:44:04.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:04.319+0000 7f423cc6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:04.341 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:04.341+0000 7f423cc6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:04.343 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:04.343+0000 7f423cc6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:04.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:44:04.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:44:04.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:44:04.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:44:04.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:44:04.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:04.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:44:04.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:04.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:44:04.728 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:44:04.859 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:04.859+0000 7f423cc6a080 -1 Falling back to public interface 2021-01-15T16:44:05.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:44:05.676 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:05.675+0000 7f423cc6a080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:44:06.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:44:06.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:06.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:44:06.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:06.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:44:06.115 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:44:06.550 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/122102,v1:127.0.0.1:6811/122102] [v2:127.0.0.1:6812/122102,v1:127.0.0.1:6813/122102] exists,up ebb5cd35-d98b-4a22-9cb2-821b9236416e 2021-01-15T16:44:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:44:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:44:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:44:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:44:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-01-15T16:44:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:44:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:44:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:44:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:44:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-01-15T16:44:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:44:06.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:44:06.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:44:06.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:44:06.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-01-15T16:44:06.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-01-15T16:44:06.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:44:06.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:44:06.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:44:06.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:44:06.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:44:06.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:44:06.555 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:44:06.555 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:44:06.555 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:06.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:44:06.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:44:06.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-01-15T16:44:06.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:44:06.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5ab8f00a-1a83-43fe-8b36-84ca96eff75d 2021-01-15T16:44:06.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 5ab8f00a-1a83-43fe-8b36-84ca96eff75d' 2021-01-15T16:44:06.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:44:06.560 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 5ab8f00a-1a83-43fe-8b36-84ca96eff75d 2021-01-15T16:44:06.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBWxgFgQ1LGIRAAl0ZQzQkb9OA0rxCtOiGvNQ== 2021-01-15T16:44:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBWxgFgQ1LGIRAAl0ZQzQkb9OA0rxCtOiGvNQ=="}' 2021-01-15T16:44:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5ab8f00a-1a83-43fe-8b36-84ca96eff75d -i td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:44:07.111 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:44:07.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:44:07.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWxgFgQ1LGIRAAl0ZQzQkb9OA0rxCtOiGvNQ== --osd-uuid 5ab8f00a-1a83-43fe-8b36-84ca96eff75d 2021-01-15T16:44:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.148+0000 7f3d450fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:07.150 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.150+0000 7f3d450fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:07.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.151+0000 7f3d450fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:07.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.152+0000 7f3d450fe080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:44:07.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.152+0000 7f3d450fe080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:44:07.153 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.152+0000 7f3d450fe080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:44:07.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:07.202+0000 7f3d450fe080 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-01-15T16:44:11.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-01-15T16:44:11.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:44:11.700 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:44:11.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:44:11.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:44:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:44:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:44:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:44:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:44:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:44:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:12.246+0000 7f1fca4e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:12.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:12.273+0000 7f1fca4e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:12.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:12.274+0000 7f1fca4e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:12.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:44:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:44:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:44:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:44:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:44:12.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:44:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:44:12.630 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:44:12.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:12.789+0000 7f1fca4e6080 -1 Falling back to public interface 2021-01-15T16:44:13.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:44:13.583 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:13.583+0000 7f1fca4e6080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:44:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:44:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:44:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:44:14.021 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:44:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/123366,v1:127.0.0.1:6819/123366] [v2:127.0.0.1:6820/123366,v1:127.0.0.1:6821/123366] exists,up 5ab8f00a-1a83-43fe-8b36-84ca96eff75d 2021-01-15T16:44:14.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:44:14.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:44:14.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:44:14.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:44:14.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-01-15T16:44:14.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:44:14.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:44:14.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:44:14.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:44:14.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-01-15T16:44:14.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:44:14.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:44:14.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:44:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:44:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-01-15T16:44:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-01-15T16:44:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:44:14.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:44:14.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:44:14.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:44:14.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:44:14.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:44:14.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:44:14.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:44:14.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:44:14.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:14.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:44:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:44:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:44:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:44:14.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:44:14.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:44:14.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:44:14.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:14.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:44:14.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:44:14.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-01-15T16:44:14.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:44:14.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d7844e3d-6311-4cae-84bd-b5fde50eb984 2021-01-15T16:44:14.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 d7844e3d-6311-4cae-84bd-b5fde50eb984' 2021-01-15T16:44:14.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:44:14.451 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 d7844e3d-6311-4cae-84bd-b5fde50eb984 2021-01-15T16:44:14.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBexgFgOR9XGxAAK1X5WuVHK2FVwQuDWhkIdw== 2021-01-15T16:44:14.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBexgFgOR9XGxAAK1X5WuVHK2FVwQuDWhkIdw=="}' 2021-01-15T16:44:14.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d7844e3d-6311-4cae-84bd-b5fde50eb984 -i td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:44:15.018 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:44:15.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:44:15.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBexgFgOR9XGxAAK1X5WuVHK2FVwQuDWhkIdw== --osd-uuid d7844e3d-6311-4cae-84bd-b5fde50eb984 2021-01-15T16:44:15.054 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.054+0000 7f1fe4c44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:15.056 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.056+0000 7f1fe4c44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:15.057 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.057+0000 7f1fe4c44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:15.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.057+0000 7f1fe4c44080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:44:15.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.057+0000 7f1fe4c44080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:44:15.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.057+0000 7f1fe4c44080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:44:15.106 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:15.106+0000 7f1fe4c44080 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-01-15T16:44:19.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-01-15T16:44:19.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:44:19.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:44:19.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:44:19.825 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:44:20.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:44:20.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:44:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:44:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:44:20.360 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:44:20.383 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:20.382+0000 7fa332fa8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:20.399 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:20.399+0000 7fa332fa8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:20.400 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:20.399+0000 7fa332fa8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:20.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:44:20.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:44:20.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:44:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:44:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:44:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:44:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:20.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:44:20.756 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:44:20.916 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:20.916+0000 7fa332fa8080 -1 Falling back to public interface 2021-01-15T16:44:21.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:44:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:21.733+0000 7fa332fa8080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:44:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:44:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:44:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:44:22.150 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:44:22.583 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/124630,v1:127.0.0.1:6827/124630] [v2:127.0.0.1:6828/124630,v1:127.0.0.1:6829/124630] exists,up d7844e3d-6311-4cae-84bd-b5fde50eb984 2021-01-15T16:44:22.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:44:22.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:44:22.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:44:22.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:44:22.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-01-15T16:44:22.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:44:22.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:44:22.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T16:44:22.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:44:22.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:44:22.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:44:22.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:44:22.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:44:22.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:44:22.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-01-15T16:44:22.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-01-15T16:44:22.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:44:22.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:44:22.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:44:22.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:44:22.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:44:22.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:44:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:44:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:44:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:44:22.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:22.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:44:22.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:44:22.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:44:22.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:44:22.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:44:22.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:44:22.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:44:22.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:44:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:44:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-01-15T16:44:22.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:44:22.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8b1cc73c-2211-446f-a657-a9923a26fe68 2021-01-15T16:44:22.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 8b1cc73c-2211-446f-a657-a9923a26fe68' 2021-01-15T16:44:22.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:44:22.593 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 8b1cc73c-2211-446f-a657-a9923a26fe68 2021-01-15T16:44:22.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBmxgFgARW7IxAAJ1eiA1fttYNwD0nI1hHw8g== 2021-01-15T16:44:22.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBmxgFgARW7IxAAJ1eiA1fttYNwD0nI1hHw8g=="}' 2021-01-15T16:44:22.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8b1cc73c-2211-446f-a657-a9923a26fe68 -i td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:44:23.135 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T16:44:23.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:44:23.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBmxgFgARW7IxAAJ1eiA1fttYNwD0nI1hHw8g== --osd-uuid 8b1cc73c-2211-446f-a657-a9923a26fe68 2021-01-15T16:44:23.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.172+0000 7f82ebbdc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:23.174 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.174+0000 7f82ebbdc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:23.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.175+0000 7f82ebbdc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.176+0000 7f82ebbdc080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:44:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.176+0000 7f82ebbdc080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:44:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.176+0000 7f82ebbdc080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:44:23.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:23.358+0000 7f82ebbdc080 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-01-15T16:44:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-01-15T16:44:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:44:28.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T16:44:28.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:44:28.193 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T16:44:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T16:44:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:44:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:44:28.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:44:28.742 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T16:44:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:28.763+0000 7f9c284eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:28.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:28.775+0000 7f9c284eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:28.777 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:28.776+0000 7f9c284eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T16:44:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:44:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T16:44:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:44:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:44:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:44:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:29.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:44:29.133 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:44:29.296 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:29.295+0000 7f9c284eb080 -1 Falling back to public interface 2021-01-15T16:44:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:44:30.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:30.127+0000 7f9c284eb080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T16:44:30.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:44:30.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:30.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:44:30.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:30.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:44:30.547 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:44:30.996 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/125894,v1:127.0.0.1:6835/125894] [v2:127.0.0.1:6836/125894,v1:127.0.0.1:6837/125894] exists,up 8b1cc73c-2211-446f-a657-a9923a26fe68 2021-01-15T16:44:30.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:44:30.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:44:30.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:44:30.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:44:30.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-01-15T16:44:30.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:44:30.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:44:30.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T16:44:30.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:44:30.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-01-15T16:44:30.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:44:30.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:44:30.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:44:30.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:44:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-01-15T16:44:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-01-15T16:44:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:44:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-01-15T16:44:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:44:31.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:44:31.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:44:31.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:44:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:44:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:44:31.002 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:44:31.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:31.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:44:31.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:44:31.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:44:31.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:44:31.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:44:31.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:44:31.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:44:31.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:44:31.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:31.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:44:31.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:44:31.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-01-15T16:44:31.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:44:31.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2391acf1-b516-43a4-b922-d3770d3c6ee8 2021-01-15T16:44:31.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 2391acf1-b516-43a4-b922-d3770d3c6ee8' 2021-01-15T16:44:31.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:44:31.006 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 2391acf1-b516-43a4-b922-d3770d3c6ee8 2021-01-15T16:44:31.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBvxgFgudzBABAAXAmZyqh02bOCf1gQcBscfA== 2021-01-15T16:44:31.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBvxgFgudzBABAAXAmZyqh02bOCf1gQcBscfA=="}' 2021-01-15T16:44:31.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2391acf1-b516-43a4-b922-d3770d3c6ee8 -i td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:44:31.564 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T16:44:31.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:44:31.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvxgFgudzBABAAXAmZyqh02bOCf1gQcBscfA== --osd-uuid 2391acf1-b516-43a4-b922-d3770d3c6ee8 2021-01-15T16:44:31.602 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.601+0000 7f919a14b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:31.603 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.603+0000 7f919a14b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:31.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.605+0000 7f919a14b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:31.606 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.606+0000 7f919a14b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:44:31.606 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.606+0000 7f919a14b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:44:31.606 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.606+0000 7f919a14b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:44:31.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:31.692+0000 7f919a14b080 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-01-15T16:44:36.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-01-15T16:44:36.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:44:36.487 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T16:44:36.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T16:44:36.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:44:37.018 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T16:44:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T16:44:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:44:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:44:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:44:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:44:37.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:37.041+0000 7f59f6f1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:37.043 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:37.043+0000 7f59f6f1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:37.044+0000 7f59f6f1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:44:37.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T16:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T16:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:44:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:44:37.413 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:44:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:37.571+0000 7f59f6f1f080 -1 Falling back to public interface 2021-01-15T16:44:37.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:44:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:44:38.463+0000 7f59f6f1f080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T16:44:38.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:44:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:44:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:44:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:44:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:44:38.822 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:44:39.274 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/127166,v1:127.0.0.1:6843/127166] [v2:127.0.0.1:6844/127166,v1:127.0.0.1:6845/127166] exists,up 2391acf1-b516-43a4-b922-d3770d3c6ee8 2021-01-15T16:44:39.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:44:39.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:44:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:44:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-01-15T16:44:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:44:39.968 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:44:39.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:44:40.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2021-01-15T16:44:40.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:40.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-01-15T16:44:41.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:41.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-01-15T16:44:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-01-15T16:44:41.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:41.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-01-15T16:44:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-01-15T16:44:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-01-15T16:44:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-01-15T16:44:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-01-15T16:44:42.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-01-15T16:44:42.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-01-15T16:44:42.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-01-15T16:44:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-01-15T16:44:43.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-01-15T16:44:43.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-01-15T16:44:43.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-01-15T16:44:43.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-01-15T16:44:43.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-01-15T16:44:43.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:43.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-01-15T16:44:44.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-01-15T16:44:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-01-15T16:44:44.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-01-15T16:44:44.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-01-15T16:44:44.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-01-15T16:44:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-01-15T16:44:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-01-15T16:44:45.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-01-15T16:44:45.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-01-15T16:44:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-01-15T16:44:45.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-01-15T16:44:45.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-01-15T16:44:45.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:45.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-01-15T16:44:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-01-15T16:44:46.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-01-15T16:44:46.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-01-15T16:44:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-01-15T16:44:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-01-15T16:44:46.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-01-15T16:44:46.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:46.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-01-15T16:44:47.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-01-15T16:44:47.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-01-15T16:44:47.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2021-01-15T16:44:47.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2021-01-15T16:44:47.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2021-01-15T16:44:47.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:47.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2021-01-15T16:44:48.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2021-01-15T16:44:48.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2021-01-15T16:44:48.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2021-01-15T16:44:48.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2021-01-15T16:44:48.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2021-01-15T16:44:48.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2021-01-15T16:44:48.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:48.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2021-01-15T16:44:49.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2021-01-15T16:44:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2021-01-15T16:44:49.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2021-01-15T16:44:49.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2021-01-15T16:44:49.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2021-01-15T16:44:49.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2021-01-15T16:44:49.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2021-01-15T16:44:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2021-01-15T16:44:50.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2021-01-15T16:44:50.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2021-01-15T16:44:50.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2021-01-15T16:44:50.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2021-01-15T16:44:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2021-01-15T16:44:51.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2021-01-15T16:44:51.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2021-01-15T16:44:51.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2021-01-15T16:44:51.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2021-01-15T16:44:51.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2021-01-15T16:44:51.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:51.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2021-01-15T16:44:52.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2021-01-15T16:44:52.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2021-01-15T16:44:52.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2021-01-15T16:44:52.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2021-01-15T16:44:52.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2021-01-15T16:44:52.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2021-01-15T16:44:52.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:52.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2021-01-15T16:44:53.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2021-01-15T16:44:53.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2021-01-15T16:44:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2021-01-15T16:44:53.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2021-01-15T16:44:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2021-01-15T16:44:53.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:53.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2021-01-15T16:44:54.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2021-01-15T16:44:54.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2021-01-15T16:44:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2021-01-15T16:44:54.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2021-01-15T16:44:54.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2021-01-15T16:44:54.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:54.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2021-01-15T16:44:55.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2021-01-15T16:44:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2021-01-15T16:44:55.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2021-01-15T16:44:55.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2021-01-15T16:44:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2021-01-15T16:44:55.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:55.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2021-01-15T16:44:56.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2021-01-15T16:44:56.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2021-01-15T16:44:56.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2021-01-15T16:44:56.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2021-01-15T16:44:56.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2021-01-15T16:44:56.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2021-01-15T16:44:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2021-01-15T16:44:57.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2021-01-15T16:44:57.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2021-01-15T16:44:57.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2021-01-15T16:44:57.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2021-01-15T16:44:57.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2021-01-15T16:44:57.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2021-01-15T16:44:57.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:57.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2021-01-15T16:44:58.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2021-01-15T16:44:58.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2021-01-15T16:44:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2021-01-15T16:44:58.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2021-01-15T16:44:58.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2021-01-15T16:44:58.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2021-01-15T16:44:58.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2021-01-15T16:44:59.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2021-01-15T16:44:59.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2021-01-15T16:44:59.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2021-01-15T16:44:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2021-01-15T16:44:59.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2021-01-15T16:44:59.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:44:59.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2021-01-15T16:45:00.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2021-01-15T16:45:00.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2021-01-15T16:45:00.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2021-01-15T16:45:00.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2021-01-15T16:45:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2021-01-15T16:45:00.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2021-01-15T16:45:00.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:00.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2021-01-15T16:45:01.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2021-01-15T16:45:01.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2021-01-15T16:45:01.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2021-01-15T16:45:01.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2021-01-15T16:45:01.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2021-01-15T16:45:01.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2021-01-15T16:45:01.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:01.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2021-01-15T16:45:02.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2021-01-15T16:45:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2021-01-15T16:45:02.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2021-01-15T16:45:02.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2021-01-15T16:45:02.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2021-01-15T16:45:02.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2021-01-15T16:45:02.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:02.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2021-01-15T16:45:03.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2021-01-15T16:45:03.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2021-01-15T16:45:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2021-01-15T16:45:03.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2021-01-15T16:45:03.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2021-01-15T16:45:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2021-01-15T16:45:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:45:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2021-01-15T16:45:04.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:45:04.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-01-15T16:45:04.419 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:45:04.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-01-15T16:45:05.030 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-01-15T16:45:05.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:45:05.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-01-15T16:45:06.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-01-15T16:45:06.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:45:06.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:45:06.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:45:06.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:45:06.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:45:06.044 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:45:06.044 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:45:06.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:45:06.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:45:06.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:45:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:45:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:45:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:45:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:45:06.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:45:06.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:45:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:45:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:45:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:45:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:45:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:45:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:45:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:06.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:45:06.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2021-01-15T16:45:06.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2021-01-15T16:45:06.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2021-01-15T16:45:06.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:06.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:45:06.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:45:06.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:45:06.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-01-15T16:45:06.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:06.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:45:07.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509453 2021-01-15T16:45:07.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509453 2021-01-15T16:45:07.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453' 2021-01-15T16:45:07.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:07.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:45:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345931 2021-01-15T16:45:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345931 2021-01-15T16:45:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931' 2021-01-15T16:45:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:07.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:45:07.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182410 2021-01-15T16:45:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182410 2021-01-15T16:45:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931 4-107374182410' 2021-01-15T16:45:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:07.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018888 2021-01-15T16:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018888 2021-01-15T16:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931 4-107374182410 5-128849018888' 2021-01-15T16:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:07.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2021-01-15T16:45:07.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:07.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:45:07.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-01-15T16:45:07.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:07.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2021-01-15T16:45:07.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-01-15T16:45:07.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:45:07.427 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836496 2021-01-15T16:45:07.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 0 -lt 21474836496 2021-01-15T16:45:07.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:45:08.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:45:08.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:45:09.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2021-01-15T16:45:09.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:09.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:45:09.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:09.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:45:09.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:45:09.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:09.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:45:09.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:45:09.224 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:45:09.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:45:09.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672975 -lt 42949672974 2021-01-15T16:45:09.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:09.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2021-01-15T16:45:09.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:09.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:45:09.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-01-15T16:45:09.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:09.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509453 2021-01-15T16:45:09.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-01-15T16:45:09.664 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509453 2021-01-15T16:45:09.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:45:10.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509453 2021-01-15T16:45:10.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:10.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345931 2021-01-15T16:45:10.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:10.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:45:10.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345931 2021-01-15T16:45:10.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:10.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345931 2021-01-15T16:45:10.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2021-01-15T16:45:10.090 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345931 2021-01-15T16:45:10.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:45:10.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345932 -lt 85899345931 2021-01-15T16:45:10.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:10.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182410 2021-01-15T16:45:10.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:10.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:45:10.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182410 2021-01-15T16:45:10.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:10.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182410 2021-01-15T16:45:10.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2021-01-15T16:45:10.488 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182410 2021-01-15T16:45:10.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:45:10.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182410 -lt 107374182410 2021-01-15T16:45:10.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:10.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018888 2021-01-15T16:45:10.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:45:10.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018888 2021-01-15T16:45:10.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:10.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018888 2021-01-15T16:45:10.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2021-01-15T16:45:10.950 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018888 2021-01-15T16:45:10.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:45:11.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018889 -lt 128849018888 2021-01-15T16:45:11.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:45:11.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:11.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:11.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:45:11.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:11.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:11.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:11.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:11.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:11.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:11.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:12.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:12.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:12.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:12.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:12.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:12.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:45:12.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:45:12.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:45:12.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:12.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:12.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:12.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:12.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:12.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:12.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:12.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:12.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:12.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:13.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:13.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:13.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:13.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:13.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:13.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:14.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6797 2021-01-15T16:45:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6797 '!=' null 2021-01-15T16:45:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:14.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:14.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:14.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:14.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:14.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:14.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:14.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:14.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:14.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:14.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:14.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:14.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:15.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:15.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:15.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:15.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:15.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24914 2021-01-15T16:45:15.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24914 '!=' null 2021-01-15T16:45:15.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:15.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:15.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:15.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:15.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:15.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:15.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:15.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:15.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:15.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:15.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:16.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:16.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:16.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:16.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:16.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:16.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:16.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:16.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20802 2021-01-15T16:45:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20802 '!=' null 2021-01-15T16:45:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:17.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:17.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:17.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:17.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:17.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:17.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:17.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:17.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:17.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:17.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:17.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:17.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:18.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:18.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:18.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:18.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:18.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:18.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:18.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17340 2021-01-15T16:45:18.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17340 '!=' null 2021-01-15T16:45:18.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:18.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:18.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:19.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:19.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:19.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:19.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:19.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:19.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:19.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:19.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:19.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:19.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:19.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:19.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:20.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:20.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:20.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:20.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:20.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=30511 2021-01-15T16:45:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 30511 '!=' null 2021-01-15T16:45:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:20.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:20.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:20.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:20.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:20.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:20.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:20.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:20.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:21.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:21.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:21.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:21.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:21.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:21.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:21.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:21.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:21.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:21.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:21.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:21.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:22.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=30511 2021-01-15T16:45:22.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 30511 '!=' null 2021-01-15T16:45:22.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:22.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:22.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:22.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:22.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:22.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:22.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:22.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:22.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:22.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:22.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:22.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:22.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:22.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:22.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:23.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:23.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:23.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26351 2021-01-15T16:45:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26351 '!=' null 2021-01-15T16:45:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:23.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:23.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:23.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:23.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:23.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:23.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:23.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:23.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:24.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:24.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:24.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:24.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:24.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:24.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:24.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:24.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:24.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:25.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38140 2021-01-15T16:45:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38140 '!=' null 2021-01-15T16:45:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:25.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:25.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:25.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:25.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:25.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:25.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:25.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:26.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:26.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:26.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24964 2021-01-15T16:45:26.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24964 '!=' null 2021-01-15T16:45:26.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:26.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:26.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:27.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:27.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:27.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:27.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:27.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:27.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:27.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:27.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:27.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:27.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:27.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:27.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:27.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:27.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:27.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:27.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:27.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:28.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24964 2021-01-15T16:45:28.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24964 '!=' null 2021-01-15T16:45:28.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:28.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:28.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:28.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:28.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:28.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:28.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:28.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:28.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:28.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:28.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:28.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:28.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:28.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:28.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:29.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:29.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:29.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:29.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24964 2021-01-15T16:45:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24964 '!=' null 2021-01-15T16:45:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:29.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:30.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:30.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:30.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:30.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:30.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:30.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:30.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:30.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:30.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:30.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:30.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:30.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:31.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:31.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:31.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:31.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:31.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:31.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:31.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:31.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:31.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:31.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38833 2021-01-15T16:45:31.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38833 '!=' null 2021-01-15T16:45:31.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:31.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:31.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:31.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:32.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:32.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:32.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:32.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:32.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:32.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:32.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:32.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:32.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:32.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:32.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:32.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25658 2021-01-15T16:45:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25658 '!=' null 2021-01-15T16:45:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:33.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:33.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:33.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:33.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:33.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:33.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:33.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:33.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:33.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:33.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:33.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:34.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:34.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:34.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39527 2021-01-15T16:45:34.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39527 '!=' null 2021-01-15T16:45:34.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:34.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:34.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:34.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:34.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:34.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:34.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:34.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:34.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:35.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:35.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:35.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:35.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:35.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:35.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:35.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:35.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:35.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:36.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39527 2021-01-15T16:45:36.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39527 '!=' null 2021-01-15T16:45:36.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:36.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:36.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:36.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:36.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:36.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:36.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:36.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:36.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:36.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:36.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:36.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:36.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:36.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:36.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:37.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:37.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:37.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:37.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27738 2021-01-15T16:45:37.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27738 '!=' null 2021-01-15T16:45:37.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:37.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:37.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:37.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:37.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:37.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:37.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:37.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:37.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:37.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:37.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:38.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:38.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:38.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:38.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:38.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:38.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:38.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:38.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:38.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:38.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27738 2021-01-15T16:45:39.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27738 '!=' null 2021-01-15T16:45:39.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:39.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:39.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:39.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:39.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:39.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:39.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:39.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:39.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:39.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:39.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:39.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:39.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:39.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:40.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:40.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:40.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:40.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:40.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:40.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:40.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:40.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:40.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:40.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41607 2021-01-15T16:45:40.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41607 '!=' null 2021-01-15T16:45:40.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:40.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:40.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:40.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:40.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:40.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:40.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:40.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:40.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:40.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:40.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:41.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:41.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:41.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:41.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:41.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:41.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:41.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:41.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:41.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:42.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41607 2021-01-15T16:45:42.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41607 '!=' null 2021-01-15T16:45:42.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:42.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:42.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:42.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:42.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:42.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:42.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:42.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:42.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:42.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:43.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:43.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:43.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:43.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:43.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27738 2021-01-15T16:45:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27738 '!=' null 2021-01-15T16:45:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:43.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:44.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:44.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:44.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:44.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:44.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:44.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:44.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:44.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:44.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:44.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:44.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:44.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:45.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:45.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:45.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40913 2021-01-15T16:45:45.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40913 '!=' null 2021-01-15T16:45:45.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:45.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:45.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:45.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:45.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:45.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:45.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:45.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:45.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:46.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:46.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:46.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:46.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:46.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:46.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:46.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:46.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:46.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:46.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:46.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:46.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:47.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27044 2021-01-15T16:45:47.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27044 '!=' null 2021-01-15T16:45:47.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:47.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:47.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:47.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:47.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:47.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:47.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:47.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:47.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:47.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:47.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:47.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:47.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:47.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:47.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:48.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:48.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:48.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:48.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:48.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:48.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:48.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:48.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:48.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19763 2021-01-15T16:45:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19763 '!=' null 2021-01-15T16:45:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:48.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:48.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:48.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:48.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:48.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:48.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:48.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:48.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:48.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:49.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:49.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:49.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:49.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:49.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:49.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:49.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:49.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:49.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:49.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:49.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:49.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:49.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19763 2021-01-15T16:45:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19763 '!=' null 2021-01-15T16:45:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:50.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:50.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:50.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:50.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:50.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:50.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:50.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:50.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:51.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:51.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:51.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:51.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:51.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:51.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:51.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19763 2021-01-15T16:45:51.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19763 '!=' null 2021-01-15T16:45:51.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:51.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:51.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:51.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:51.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:52.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:45:52.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:52.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:52.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:52.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:45:52.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:45:52.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:45:52.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:45:52.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:45:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:45:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:45:52.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:45:52.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:45:53.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10401 2021-01-15T16:45:53.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10401 '!=' null 2021-01-15T16:45:53.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:45:53.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:45:53.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:45:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:45:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:45:53.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:45:53.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:45:53.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:45:53.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:45:53.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:45:53.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:45:53.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:45:53.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:45:53.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:45:53.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:45:54.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:45:54.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:45:54.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:45:54.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-01-15T16:45:54.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:45:54.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:45:54.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:45:54.554 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:45:54.554 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:45:54.554 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:45:54.554 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:45:54.555 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:45:54.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:45:54.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:54.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:45:54.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836508 2021-01-15T16:45:54.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836508 2021-01-15T16:45:54.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508' 2021-01-15T16:45:54.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:54.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:45:54.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672986 2021-01-15T16:45:54.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672986 2021-01-15T16:45:54.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986' 2021-01-15T16:45:54.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:54.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:45:54.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509464 2021-01-15T16:45:54.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509464 2021-01-15T16:45:54.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464' 2021-01-15T16:45:54.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:54.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:45:55.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345943 2021-01-15T16:45:55.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345943 2021-01-15T16:45:55.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943' 2021-01-15T16:45:55.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:55.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:45:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182421 2021-01-15T16:45:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182421 2021-01-15T16:45:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943 4-107374182421' 2021-01-15T16:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:45:55.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:45:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018900 2021-01-15T16:45:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018900 2021-01-15T16:45:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943 4-107374182421 5-128849018900' 2021-01-15T16:45:55.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:55.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836508 2021-01-15T16:45:55.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:55.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:45:55.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836508 2021-01-15T16:45:55.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:55.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836508 2021-01-15T16:45:55.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2021-01-15T16:45:55.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:45:55.378 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836508 2021-01-15T16:45:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836506 -lt 21474836508 2021-01-15T16:45:55.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:45:56.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:45:56.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:45:57.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836508 -lt 21474836508 2021-01-15T16:45:57.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:57.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672986 2021-01-15T16:45:57.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:57.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:45:57.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672986 2021-01-15T16:45:57.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:57.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672986 2021-01-15T16:45:57.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2021-01-15T16:45:57.166 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672986 2021-01-15T16:45:57.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:45:57.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672986 -lt 42949672986 2021-01-15T16:45:57.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:57.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509464 2021-01-15T16:45:57.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:57.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:45:57.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509464 2021-01-15T16:45:57.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:57.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509464 2021-01-15T16:45:57.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2021-01-15T16:45:57.566 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509464 2021-01-15T16:45:57.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:45:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509465 -lt 64424509464 2021-01-15T16:45:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:57.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345943 2021-01-15T16:45:57.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:57.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:45:57.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345943 2021-01-15T16:45:57.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:57.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345943 2021-01-15T16:45:57.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2021-01-15T16:45:57.974 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345943 2021-01-15T16:45:57.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:45:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345944 -lt 85899345943 2021-01-15T16:45:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:58.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182421 2021-01-15T16:45:58.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:58.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:45:58.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182421 2021-01-15T16:45:58.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:58.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182421 2021-01-15T16:45:58.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2021-01-15T16:45:58.422 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182421 2021-01-15T16:45:58.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:45:58.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182422 -lt 107374182421 2021-01-15T16:45:58.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:45:58.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018900 2021-01-15T16:45:58.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:45:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:45:58.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018900 2021-01-15T16:45:58.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:45:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018900 2021-01-15T16:45:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2021-01-15T16:45:58.834 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018900 2021-01-15T16:45:58.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:45:59.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018900 -lt 128849018900 2021-01-15T16:45:59.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:45:59.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-01-15T16:45:59.592 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:45:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-01-15T16:45:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-01-15T16:45:59.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:45:59.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:45:59.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:45:59.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:45:59.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:45:59.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:45:59.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-01-15T16:45:59.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:45:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:45:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:45:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-01-15T16:45:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2021-01-15T16:45:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:45:59.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:45:59.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:46:00.965 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:46:00.966 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_log_t": { 2021-01-15T16:46:00.966 INFO:tasks.workunit.client.0.smithi145.stdout: "head": "34'150", 2021-01-15T16:46:00.966 INFO:tasks.workunit.client.0.smithi145.stdout: "tail": "34'148", 2021-01-15T16:46:00.966 INFO:tasks.workunit.client.0.smithi145.stdout: "log": [ 2021-01-15T16:46:00.967 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.967 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:46:00.967 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e9c179f9:::obj-149:head", 2021-01-15T16:46:00.967 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'149", 2021-01-15T16:46:00.967 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:46:00.968 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4631.0:1", 2021-01-15T16:46:00.968 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:46:00.968 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:45:03.753055+0000", 2021-01-15T16:46:00.968 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:46:00.968 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:46:00.969 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:46:00.969 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:46:00.969 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:46:00.969 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:46:00.969 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.970 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.970 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:46:00.970 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:46:00.970 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:46:00.970 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:46:00.971 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:46:00.971 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.971 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.971 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.971 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.972 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:46:00.972 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cb39fd60:::obj-150:head", 2021-01-15T16:46:00.972 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'150", 2021-01-15T16:46:00.972 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:46:00.973 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4634.0:1", 2021-01-15T16:46:00.973 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:46:00.973 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:45:03.928954+0000", 2021-01-15T16:46:00.973 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:46:00.973 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:46:00.974 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:46:00.974 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:46:00.974 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:46:00.974 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:46:00.974 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.975 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.975 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:46:00.975 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:46:00.975 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:46:00.976 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:46:00.976 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:46:00.976 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.976 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.977 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.977 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:46:00.977 INFO:tasks.workunit.client.0.smithi145.stdout: "dups": [ 2021-01-15T16:46:00.977 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.978 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4607.0:1", 2021-01-15T16:46:00.978 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'141", 2021-01-15T16:46:00.978 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "141", 2021-01-15T16:46:00.978 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.979 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.979 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.979 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4610.0:1", 2021-01-15T16:46:00.979 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'142", 2021-01-15T16:46:00.979 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "142", 2021-01-15T16:46:00.980 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.980 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.980 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.980 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4613.0:1", 2021-01-15T16:46:00.980 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'143", 2021-01-15T16:46:00.981 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "143", 2021-01-15T16:46:00.981 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.981 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.981 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.981 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4616.0:1", 2021-01-15T16:46:00.982 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'144", 2021-01-15T16:46:00.982 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "144", 2021-01-15T16:46:00.982 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.982 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.982 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.983 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4619.0:1", 2021-01-15T16:46:00.983 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'145", 2021-01-15T16:46:00.983 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "145", 2021-01-15T16:46:00.983 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.983 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.984 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.984 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4622.0:1", 2021-01-15T16:46:00.984 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'146", 2021-01-15T16:46:00.984 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "146", 2021-01-15T16:46:00.984 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.985 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.985 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.985 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4625.0:1", 2021-01-15T16:46:00.985 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'147", 2021-01-15T16:46:00.985 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "147", 2021-01-15T16:46:00.986 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.986 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.986 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:46:00.986 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4628.0:1", 2021-01-15T16:46:00.986 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'148", 2021-01-15T16:46:00.987 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "148", 2021-01-15T16:46:00.987 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:46:00.987 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.987 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:46:00.987 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:46:00.988 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_missing_t": { 2021-01-15T16:46:00.988 INFO:tasks.workunit.client.0.smithi145.stdout: "missing": [], 2021-01-15T16:46:00.988 INFO:tasks.workunit.client.0.smithi145.stdout: "may_include_deletes": true 2021-01-15T16:46:00.988 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:46:00.988 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:46:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:46:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2021-01-15T16:46:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2021-01-15T16:46:00.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:46:00.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=8 2021-01-15T16:46:00.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 8 '!=' 8 ']' 2021-01-15T16:46:00.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-01-15T16:46:00.991 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to max 2 2021-01-15T16:46:00.991 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'150 2021-01-15T16:46:00.992 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'149 2021-01-15T16:46:00.992 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after earliest_dup_version 141 2021-01-15T16:46:00.992 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:46:00.992 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:46:00.992 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:46:00.993 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to max 2 2021-01-15T16:46:00.994 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'150 2021-01-15T16:46:00.994 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'149 2021-01-15T16:46:00.994 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after earliest_dup_version 141 2021-01-15T16:46:00.994 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:46:00.995 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:46:00.995 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:46:00.995 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:46:00.995 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:46:00.995 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:46:00.996 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:46:00.996 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:46:00.996 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to max 2 2021-01-15T16:46:00.996 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'150 2021-01-15T16:46:00.996 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to copy log version 34'149 2021-01-15T16:46:00.997 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after earliest_dup_version 141 2021-01-15T16:46:00.997 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:46:00.997 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:46:00.997 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:46:00.998 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:46:00.998 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:46:00.998 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:46:00.999 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:46:00.999 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:06.299+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:46:00.999 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_after v 34'150 2021-01-15T16:46:00.999 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after earliest_dup_version 141 2021-01-15T16:46:00.999 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:46:01.000 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:46:01.000 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:46:01.000 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:46:01.000 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:46:01.001 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:46:01.001 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:46:01.001 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:46:01.001 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'149 2021-01-15T16:46:01.001 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:45:46.837+0000 7f878ff62700 20 copy_up_to/copy_after copy dup from log version 34'150 2021-01-15T16:46:01.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-01-15T16:46:01.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:46:01.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-01-15T16:46:01.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:46:01.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:46:01.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:46:01.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:46:01.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:46:01.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:46:01.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:46:01.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:46:01.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:46:01.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:46:01.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:46:01.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:46:01.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:46:01.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:46:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:46:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:46:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:46:01.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:46:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:46:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:46:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:46:01.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:46:01.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:46:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:46:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:46:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-01-15T16:46:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-01-15T16:46:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-01-15T16:46:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:46:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:46:01.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:46:01.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:46:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:46:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:46:01.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:46:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:46:01.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:46:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:46:01.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:46:01.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:46:01.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:46:01.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:46:01.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:46:01.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:46:01.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:46:01.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:46:01.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:46:01.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:46:01.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:46:01.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:46:01.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:46:01.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:46:01.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:46:01.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-01-15T16:46:01.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:46:01.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119796 2021-01-15T16:46:01.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:46:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:46:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:46:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-01-15T16:46:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2021-01-15T16:46:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:115: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2021-01-15T16:46:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2021-01-15T16:46:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2021-01-15T16:46:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2021-01-15T16:46:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-01-15T16:46:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-01-15T16:46:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-01-15T16:46:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:46:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:46:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:46:01.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-01-15T16:46:01.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:46:01.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:01.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:01.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:46:01.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:46:01.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:46:01.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:46:01.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:46:01.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:46:01.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:46:01.552 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:46:01.552 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:46:01.552 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:46:01.552 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:46:01.553 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.553 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.553 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:46:01.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:46:01.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get fsid 2021-01-15T16:46:01.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:46:01.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:46:01.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:46:01.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:46:01.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:46:01.663 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:46:01.663 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:46:01.663 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:46:01.664 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:46:01.664 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:01.664 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:01.664 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:46:01.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:46:01.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get mon_host 2021-01-15T16:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-01-15T16:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:46:01.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:46:01.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:46:01.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-01-15T16:46:01.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:46:02.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:46:02.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:02.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:02.250 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:02.251 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:02.251 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:02.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:02.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:46:02.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:46:02.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-01-15T16:46:02.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:02.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:02.275 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-01-15T16:46:02.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-01-15T16:46:02.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:02.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-01-15T16:46:02.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:02.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:02.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:46:02.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:02.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-01-15T16:46:02.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-01-15T16:46:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-01-15T16:46:02.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:02.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:02.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:02.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:02.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:02.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:02.283 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:02.283 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:02.284 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:02.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:02.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:02.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:02.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:02.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:02.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:02.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:02.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-01-15T16:46:02.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:02.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=08900651-c997-482b-a564-a147680d65e7 2021-01-15T16:46:02.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 08900651-c997-482b-a564-a147680d65e7' 2021-01-15T16:46:02.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:02.288 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 08900651-c997-482b-a564-a147680d65e7 2021-01-15T16:46:02.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDKxgFg/suJERAAl/c43eBl5C4atc3gamfhhQ== 2021-01-15T16:46:02.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDKxgFg/suJERAAl/c43eBl5C4atc3gamfhhQ=="}' 2021-01-15T16:46:02.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 08900651-c997-482b-a564-a147680d65e7 -i td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:46:02.632 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:02.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:46:02.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDKxgFg/suJERAAl/c43eBl5C4atc3gamfhhQ== --osd-uuid 08900651-c997-482b-a564-a147680d65e7 2021-01-15T16:46:02.667 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.667+0000 7f6feb143080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:02.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.669+0000 7f6feb143080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.670+0000 7f6feb143080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.671+0000 7f6feb143080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:46:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.671+0000 7f6feb143080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:46:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.671+0000 7f6feb143080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:46:02.764 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:02.763+0000 7f6feb143080 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-01-15T16:46:07.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-01-15T16:46:07.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:46:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:07.324 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:46:07.846 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:46:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:46:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:07.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:07.869 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:07.869+0000 7f3754f35080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:07.896 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:07.895+0000 7f3754f35080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:07.897 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:07.897+0000 7f3754f35080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:46:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:46:08.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:08.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:08.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:08.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:08.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:08.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:46:08.242 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:08.413 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:08.412+0000 7f3754f35080 -1 Falling back to public interface 2021-01-15T16:46:08.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:09.223 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:09.222+0000 7f3754f35080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:46:09.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:09.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:09.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:09.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:09.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:46:09.638 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:10.053 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/135259,v1:127.0.0.1:6803/135259] [v2:127.0.0.1:6804/135259,v1:127.0.0.1:6805/135259] exists,up 08900651-c997-482b-a564-a147680d65e7 2021-01-15T16:46:10.054 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 d73b5928-6d7d-419f-ac97-e2811a5736e7 2021-01-15T16:46:10.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:10.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:10.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-01-15T16:46:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:10.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:46:10.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:10.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-01-15T16:46:10.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:10.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:10.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:10.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:10.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-01-15T16:46:10.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-01-15T16:46:10.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:10.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:10.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:10.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:10.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:10.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:10.060 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:10.060 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:10.060 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:10.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:10.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:10.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:10.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:10.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:10.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:10.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:10.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-01-15T16:46:10.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:10.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d73b5928-6d7d-419f-ac97-e2811a5736e7 2021-01-15T16:46:10.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 d73b5928-6d7d-419f-ac97-e2811a5736e7' 2021-01-15T16:46:10.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:10.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDSxgFgRqD2AxAAB4L2FEbosj5/zaleuDvMaQ== 2021-01-15T16:46:10.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDSxgFgRqD2AxAAB4L2FEbosj5/zaleuDvMaQ=="}' 2021-01-15T16:46:10.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d73b5928-6d7d-419f-ac97-e2811a5736e7 -i td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:46:10.579 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:10.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:46:10.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDSxgFgRqD2AxAAB4L2FEbosj5/zaleuDvMaQ== --osd-uuid d73b5928-6d7d-419f-ac97-e2811a5736e7 2021-01-15T16:46:10.616 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.615+0000 7f572623d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:10.617 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.617+0000 7f572623d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:10.618 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.618+0000 7f572623d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.618+0000 7f572623d080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:46:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.618+0000 7f572623d080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:46:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.618+0000 7f572623d080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:46:10.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:10.677+0000 7f572623d080 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-01-15T16:46:15.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-01-15T16:46:15.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:15.444 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:46:15.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:46:15.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:15.968 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:46:15.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:46:15.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:15.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:15.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:15.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:15.991 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:15.990+0000 7f2a10991080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:16.018 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:16.017+0000 7f2a10991080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:16.019 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:16.018+0000 7f2a10991080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:16.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:46:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:46:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:46:16.387 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:16.535 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:16.535+0000 7f2a10991080 -1 Falling back to public interface 2021-01-15T16:46:16.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:17.311 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:17.310+0000 7f2a10991080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:46:17.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:17.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:17.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:17.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:17.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:46:17.781 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:18.191 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/136523,v1:127.0.0.1:6811/136523] [v2:127.0.0.1:6812/136523,v1:127.0.0.1:6813/136523] exists,up d73b5928-6d7d-419f-ac97-e2811a5736e7 2021-01-15T16:46:18.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-01-15T16:46:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:18.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:18.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:46:18.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:18.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-01-15T16:46:18.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:18.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:18.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:18.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:18.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-01-15T16:46:18.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-01-15T16:46:18.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:18.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:18.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:18.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:18.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:18.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:18.196 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:18.196 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:18.196 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:18.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:18.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:18.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:18.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:18.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:18.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:18.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:18.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:18.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-01-15T16:46:18.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:18.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cb49f81c-f340-4a44-ada9-4a98bfd3cfa0 2021-01-15T16:46:18.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 cb49f81c-f340-4a44-ada9-4a98bfd3cfa0' 2021-01-15T16:46:18.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:18.201 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 cb49f81c-f340-4a44-ada9-4a98bfd3cfa0 2021-01-15T16:46:18.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDaxgFgT4tIDBAA0UGEGePPJHlpXh5R7MY54Q== 2021-01-15T16:46:18.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDaxgFgT4tIDBAA0UGEGePPJHlpXh5R7MY54Q=="}' 2021-01-15T16:46:18.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cb49f81c-f340-4a44-ada9-4a98bfd3cfa0 -i td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:46:18.753 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:46:18.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:46:18.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDaxgFgT4tIDBAA0UGEGePPJHlpXh5R7MY54Q== --osd-uuid cb49f81c-f340-4a44-ada9-4a98bfd3cfa0 2021-01-15T16:46:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.790+0000 7fc6cb8d9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:18.792 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.791+0000 7fc6cb8d9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:18.793 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.793+0000 7fc6cb8d9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:18.795 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.794+0000 7fc6cb8d9080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:46:18.795 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.794+0000 7fc6cb8d9080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:46:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.794+0000 7fc6cb8d9080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:46:18.825 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:18.825+0000 7fc6cb8d9080 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-01-15T16:46:23.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-01-15T16:46:23.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:23.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:46:23.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:23.385 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:46:23.912 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:46:23.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:46:23.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:23.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:23.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:23.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:23.936 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:23.935+0000 7f24dba89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:23.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:23.937+0000 7f24dba89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:23.939 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:23.939+0000 7f24dba89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:24.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:46:24.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:24.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:46:24.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:46:24.308 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:24.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:24.458+0000 7f24dba89080 -1 Falling back to public interface 2021-01-15T16:46:24.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:25.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:25.291+0000 7f24dba89080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:46:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:25.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:25.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:46:25.696 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:26.100 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/137787,v1:127.0.0.1:6819/137787] [v2:127.0.0.1:6820/137787,v1:127.0.0.1:6821/137787] exists,up cb49f81c-f340-4a44-ada9-4a98bfd3cfa0 2021-01-15T16:46:26.101 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca 2021-01-15T16:46:26.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:26.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-01-15T16:46:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:46:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-01-15T16:46:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-01-15T16:46:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-01-15T16:46:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:26.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:26.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:26.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:26.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:26.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:26.106 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:26.106 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:26.106 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:26.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca' 2021-01-15T16:46:26.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:26.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDixgFgcBPcBhAAiTOecihEOCSOAvdOzEU9RQ== 2021-01-15T16:46:26.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDixgFgcBPcBhAAiTOecihEOCSOAvdOzEU9RQ=="}' 2021-01-15T16:46:26.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca -i td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:46:26.733 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:46:26.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:46:26.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDixgFgcBPcBhAAiTOecihEOCSOAvdOzEU9RQ== --osd-uuid dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca 2021-01-15T16:46:26.771 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.770+0000 7fc8a82dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:26.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.772+0000 7fc8a82dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:26.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.773+0000 7fc8a82dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:26.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.773+0000 7fc8a82dd080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:46:26.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.773+0000 7fc8a82dd080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:46:26.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.773+0000 7fc8a82dd080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:46:26.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:26.815+0000 7fc8a82dd080 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-01-15T16:46:31.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-01-15T16:46:31.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:31.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:46:31.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:31.342 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:46:31.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:46:31.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:31.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:31.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:31.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:31.867 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:46:31.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:31.888+0000 7fc74cd77080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:31.890 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:31.890+0000 7fc74cd77080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:31.892 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:31.892+0000 7fc74cd77080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:32.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:46:32.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:32.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:46:32.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:46:32.264 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:32.410 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:32.410+0000 7fc74cd77080 -1 Falling back to public interface 2021-01-15T16:46:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:33.200+0000 7fc74cd77080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:46:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:46:33.665 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:34.065 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/139051,v1:127.0.0.1:6827/139051] [v2:127.0.0.1:6828/139051,v1:127.0.0.1:6829/139051] exists,up dd6159fe-e0e5-40b7-99b9-3a0f04d2a8ca 2021-01-15T16:46:34.066 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 033d9994-708c-4324-9215-4ad85b197b23 2021-01-15T16:46:34.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:34.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:34.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-01-15T16:46:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T16:46:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:46:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-01-15T16:46:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-01-15T16:46:34.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:34.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:34.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:34.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:34.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:34.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:34.071 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:34.071 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:34.071 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:34.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:34.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:34.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:34.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:34.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:34.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:34.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:34.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:34.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:34.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:34.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:34.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:34.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-01-15T16:46:34.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:34.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=033d9994-708c-4324-9215-4ad85b197b23 2021-01-15T16:46:34.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 033d9994-708c-4324-9215-4ad85b197b23' 2021-01-15T16:46:34.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:34.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDqxgFgO4jDBBAAJ/lcz8O0yzwf9eKcxrUXeg== 2021-01-15T16:46:34.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDqxgFgO4jDBBAAJ/lcz8O0yzwf9eKcxrUXeg=="}' 2021-01-15T16:46:34.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 033d9994-708c-4324-9215-4ad85b197b23 -i td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:46:34.619 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T16:46:34.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:46:34.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDqxgFgO4jDBBAAJ/lcz8O0yzwf9eKcxrUXeg== --osd-uuid 033d9994-708c-4324-9215-4ad85b197b23 2021-01-15T16:46:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.655+0000 7fdedfe94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:34.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.656+0000 7fdedfe94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:34.658 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.658+0000 7fdedfe94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:34.660 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.659+0000 7fdedfe94080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:46:34.660 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.659+0000 7fdedfe94080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:46:34.660 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.659+0000 7fdedfe94080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:46:34.707 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:34.707+0000 7fdedfe94080 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-01-15T16:46:39.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-01-15T16:46:39.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:39.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T16:46:39.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:39.518 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T16:46:40.042 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T16:46:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T16:46:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:40.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:40.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:40.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:40.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:40.064+0000 7f781748b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:40.091 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:40.091+0000 7f781748b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:40.093 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:40.093+0000 7f781748b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:40.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T16:46:40.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:40.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T16:46:40.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:40.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:40.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:40.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:40.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:40.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:46:40.439 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:40.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:40.612+0000 7f781748b080 -1 Falling back to public interface 2021-01-15T16:46:40.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:41.426 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:41.426+0000 7f781748b080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T16:46:41.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:46:41.833 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:42.282 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/140323,v1:127.0.0.1:6835/140323] [v2:127.0.0.1:6836/140323,v1:127.0.0.1:6837/140323] exists,up 033d9994-708c-4324-9215-4ad85b197b23 2021-01-15T16:46:42.282 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 12de2165-c50c-40af-813b-81125affd114 2021-01-15T16:46:42.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:42.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:42.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:42.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:46:42.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-01-15T16:46:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:46:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:46:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T16:46:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:46:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-01-15T16:46:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:46:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:46:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:46:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:46:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-01-15T16:46:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-01-15T16:46:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:46:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-01-15T16:46:42.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:46:42.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:46:42.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:46:42.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:46:42.288 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:46:42.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:46:42.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:46:42.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:42.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:46:42.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:46:42.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:46:42.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:46:42.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:46:42.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:46:42.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:46:42.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:46:42.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:46:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:46:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-01-15T16:46:42.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:46:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=12de2165-c50c-40af-813b-81125affd114 2021-01-15T16:46:42.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 12de2165-c50c-40af-813b-81125affd114' 2021-01-15T16:46:42.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:46:42.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDyxgFgDiuDERAA7i7/U/Ym2MT/KlghPNgvKQ== 2021-01-15T16:46:42.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDyxgFgDiuDERAA7i7/U/Ym2MT/KlghPNgvKQ=="}' 2021-01-15T16:46:42.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 12de2165-c50c-40af-813b-81125affd114 -i td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:46:42.847 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T16:46:42.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:46:42.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDyxgFgDiuDERAA7i7/U/Ym2MT/KlghPNgvKQ== --osd-uuid 12de2165-c50c-40af-813b-81125affd114 2021-01-15T16:46:42.884 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.883+0000 7f450340b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:42.885 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.885+0000 7f450340b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:42.886 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.886+0000 7f450340b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:42.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.886+0000 7f450340b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:46:42.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.886+0000 7f450340b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:46:42.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.886+0000 7f450340b080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:46:42.926 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:42.926+0000 7f450340b080 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-01-15T16:46:47.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-01-15T16:46:47.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:46:47.771 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T16:46:47.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T16:46:47.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:46:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T16:46:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:46:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:46:48.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:46:48.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:46:48.310 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T16:46:48.332 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:48.332+0000 7f1fffc83080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:48.334+0000 7f1fffc83080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:48.338 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:48.336+0000 7f1fffc83080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:46:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T16:46:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:46:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T16:46:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:46:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:46:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:46:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:46:48.705 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:46:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:48.851+0000 7f1fffc83080 -1 Falling back to public interface 2021-01-15T16:46:49.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:46:49.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:46:49.687+0000 7f1fffc83080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T16:46:50.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:46:50.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:46:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:46:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:46:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:46:50.103 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:46:50.515 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/141587,v1:127.0.0.1:6843/141587] [v2:127.0.0.1:6844/141587,v1:127.0.0.1:6845/141587] exists,up 12de2165-c50c-40af-813b-81125affd114 2021-01-15T16:46:50.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:46:50.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:46:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:46:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-01-15T16:46:50.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:46:51.194 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:46:51.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:46:52.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2021-01-15T16:46:52.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:52.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-01-15T16:46:52.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:52.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-01-15T16:46:52.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:52.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-01-15T16:46:52.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:52.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-01-15T16:46:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-01-15T16:46:53.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-01-15T16:46:53.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-01-15T16:46:53.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-01-15T16:46:53.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-01-15T16:46:53.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-01-15T16:46:53.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:53.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-01-15T16:46:54.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-01-15T16:46:54.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-01-15T16:46:54.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-01-15T16:46:54.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-01-15T16:46:54.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-01-15T16:46:54.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:54.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-01-15T16:46:55.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-01-15T16:46:55.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-01-15T16:46:55.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-01-15T16:46:55.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-01-15T16:46:55.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-01-15T16:46:55.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-01-15T16:46:55.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:55.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-01-15T16:46:56.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-01-15T16:46:56.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-01-15T16:46:56.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-01-15T16:46:56.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-01-15T16:46:56.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-01-15T16:46:56.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:56.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-01-15T16:46:57.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-01-15T16:46:57.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-01-15T16:46:57.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-01-15T16:46:57.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-01-15T16:46:57.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-01-15T16:46:57.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:57.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-01-15T16:46:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-01-15T16:46:58.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-01-15T16:46:58.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-01-15T16:46:58.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-01-15T16:46:58.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2021-01-15T16:46:58.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2021-01-15T16:46:58.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:58.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2021-01-15T16:46:59.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2021-01-15T16:46:59.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2021-01-15T16:46:59.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2021-01-15T16:46:59.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2021-01-15T16:46:59.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2021-01-15T16:46:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:46:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2021-01-15T16:47:00.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2021-01-15T16:47:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2021-01-15T16:47:00.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2021-01-15T16:47:00.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2021-01-15T16:47:00.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2021-01-15T16:47:00.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:00.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2021-01-15T16:47:01.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2021-01-15T16:47:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2021-01-15T16:47:01.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2021-01-15T16:47:01.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2021-01-15T16:47:01.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2021-01-15T16:47:01.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2021-01-15T16:47:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2021-01-15T16:47:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2021-01-15T16:47:02.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2021-01-15T16:47:02.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2021-01-15T16:47:02.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2021-01-15T16:47:02.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2021-01-15T16:47:02.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:02.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2021-01-15T16:47:03.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2021-01-15T16:47:03.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2021-01-15T16:47:03.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2021-01-15T16:47:03.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2021-01-15T16:47:03.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2021-01-15T16:47:03.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:03.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2021-01-15T16:47:04.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2021-01-15T16:47:04.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2021-01-15T16:47:04.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2021-01-15T16:47:04.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2021-01-15T16:47:04.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2021-01-15T16:47:04.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:04.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2021-01-15T16:47:05.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2021-01-15T16:47:05.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2021-01-15T16:47:05.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2021-01-15T16:47:05.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2021-01-15T16:47:05.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2021-01-15T16:47:05.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2021-01-15T16:47:05.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:05.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2021-01-15T16:47:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2021-01-15T16:47:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2021-01-15T16:47:06.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2021-01-15T16:47:06.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2021-01-15T16:47:06.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2021-01-15T16:47:06.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:06.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2021-01-15T16:47:07.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2021-01-15T16:47:07.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2021-01-15T16:47:07.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2021-01-15T16:47:07.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2021-01-15T16:47:07.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2021-01-15T16:47:07.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:07.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2021-01-15T16:47:08.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2021-01-15T16:47:08.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2021-01-15T16:47:08.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2021-01-15T16:47:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2021-01-15T16:47:08.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2021-01-15T16:47:08.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:08.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2021-01-15T16:47:09.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2021-01-15T16:47:09.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2021-01-15T16:47:09.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2021-01-15T16:47:09.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2021-01-15T16:47:09.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2021-01-15T16:47:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2021-01-15T16:47:09.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:09.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2021-01-15T16:47:10.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2021-01-15T16:47:10.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2021-01-15T16:47:10.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2021-01-15T16:47:10.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2021-01-15T16:47:10.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2021-01-15T16:47:10.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2021-01-15T16:47:10.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:10.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2021-01-15T16:47:11.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2021-01-15T16:47:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2021-01-15T16:47:11.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2021-01-15T16:47:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2021-01-15T16:47:11.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2021-01-15T16:47:11.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2021-01-15T16:47:11.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:11.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2021-01-15T16:47:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2021-01-15T16:47:12.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2021-01-15T16:47:12.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2021-01-15T16:47:12.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2021-01-15T16:47:12.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2021-01-15T16:47:12.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2021-01-15T16:47:12.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:12.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2021-01-15T16:47:13.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2021-01-15T16:47:13.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2021-01-15T16:47:13.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2021-01-15T16:47:13.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2021-01-15T16:47:13.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2021-01-15T16:47:13.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:13.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2021-01-15T16:47:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2021-01-15T16:47:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2021-01-15T16:47:14.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2021-01-15T16:47:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2021-01-15T16:47:14.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2021-01-15T16:47:14.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2021-01-15T16:47:14.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:14.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2021-01-15T16:47:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2021-01-15T16:47:15.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:15.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2021-01-15T16:47:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2021-01-15T16:47:15.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:47:15.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2021-01-15T16:47:15.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:47:15.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-01-15T16:47:15.962 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:47:15.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-01-15T16:47:16.535 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-01-15T16:47:16.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:47:16.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-01-15T16:47:17.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-01-15T16:47:17.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:47:17.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:47:17.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:47:17.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:47:17.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:47:17.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:47:17.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:47:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:47:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:47:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:47:17.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:47:17.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:47:17.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:47:17.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:47:17.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:47:18.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:47:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:47:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:47:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:47:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:47:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:47:18.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:47:18.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:47:18.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2021-01-15T16:47:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2021-01-15T16:47:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2021-01-15T16:47:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:47:18.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T16:47:18.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T16:47:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-01-15T16:47:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:47:18.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509453 2021-01-15T16:47:18.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509453 2021-01-15T16:47:18.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453' 2021-01-15T16:47:18.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:47:18.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345931 2021-01-15T16:47:18.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345931 2021-01-15T16:47:18.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931' 2021-01-15T16:47:18.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:47:18.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182410 2021-01-15T16:47:18.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182410 2021-01-15T16:47:18.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931 4-107374182410' 2021-01-15T16:47:18.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:47:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:47:18.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018888 2021-01-15T16:47:18.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018888 2021-01-15T16:47:18.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509453 3-85899345931 4-107374182410 5-128849018888' 2021-01-15T16:47:18.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:18.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2021-01-15T16:47:18.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:18.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:47:18.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-01-15T16:47:18.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:18.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2021-01-15T16:47:18.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-01-15T16:47:18.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:47:18.905 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836496 2021-01-15T16:47:19.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 0 -lt 21474836496 2021-01-15T16:47:19.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:47:20.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:47:20.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:47:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836496 2021-01-15T16:47:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:20.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T16:47:20.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:20.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:47:20.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T16:47:20.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:20.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T16:47:20.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T16:47:20.701 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T16:47:20.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:47:21.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672975 -lt 42949672974 2021-01-15T16:47:21.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:21.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2021-01-15T16:47:21.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:21.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:47:21.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-01-15T16:47:21.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:21.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509453 2021-01-15T16:47:21.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-01-15T16:47:21.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:47:21.123 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509453 2021-01-15T16:47:21.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509453 2021-01-15T16:47:21.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:21.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345931 2021-01-15T16:47:21.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:21.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:47:21.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345931 2021-01-15T16:47:21.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:21.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345931 2021-01-15T16:47:21.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2021-01-15T16:47:21.540 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345931 2021-01-15T16:47:21.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:47:21.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345932 -lt 85899345931 2021-01-15T16:47:21.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:21.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182410 2021-01-15T16:47:21.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:21.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:47:21.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182410 2021-01-15T16:47:21.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:21.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182410 2021-01-15T16:47:21.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2021-01-15T16:47:21.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:47:21.941 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182410 2021-01-15T16:47:22.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182410 -lt 107374182410 2021-01-15T16:47:22.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:47:22.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018888 2021-01-15T16:47:22.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:47:22.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:47:22.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018888 2021-01-15T16:47:22.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:47:22.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018888 2021-01-15T16:47:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2021-01-15T16:47:22.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:47:22.374 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018888 2021-01-15T16:47:22.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018889 -lt 128849018888 2021-01-15T16:47:22.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:47:22.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:22.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:23.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:47:23.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:23.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:23.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:23.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:23.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:23.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:23.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:23.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:23.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:23.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:23.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:24.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:24.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:47:24.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:47:24.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:47:24.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:24.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:24.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:24.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:24.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:24.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:24.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:25.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:25.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:25.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:25.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:25.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:25.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:25.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:25.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:25.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5433 2021-01-15T16:47:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5433 '!=' null 2021-01-15T16:47:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:25.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:25.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:25.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:25.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:25.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:25.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:26.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:26.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:26.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:26.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:26.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:26.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:26.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:26.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:26.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:26.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:26.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:26.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:26.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23923 2021-01-15T16:47:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23923 '!=' null 2021-01-15T16:47:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:27.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:27.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:27.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:27.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:27.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:27.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:27.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:27.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:27.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:27.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:27.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:27.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:27.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:27.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:28.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:28.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:28.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:28.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:28.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:28.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19445 2021-01-15T16:47:28.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19445 '!=' null 2021-01-15T16:47:28.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:28.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:28.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:28.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:28.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:28.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:28.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:28.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:28.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:28.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:28.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:29.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:29.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:29.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:29.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:29.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:29.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:29.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:29.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:29.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:29.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:29.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:29.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:29.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16643 2021-01-15T16:47:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16643 '!=' null 2021-01-15T16:47:30.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:30.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:30.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:30.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:30.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:30.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:30.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:30.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:30.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:30.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:30.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:30.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:30.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:30.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:30.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:31.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:31.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:31.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:31.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:31.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16643 2021-01-15T16:47:31.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16643 '!=' null 2021-01-15T16:47:31.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:31.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:31.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:31.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:31.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:31.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:31.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:31.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:31.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:31.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:31.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:32.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:32.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:32.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:32.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:32.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:32.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:32.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:32.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:32.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:33.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28431 2021-01-15T16:47:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28431 '!=' null 2021-01-15T16:47:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:33.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:33.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:33.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:33.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:33.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:33.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:33.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:33.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:33.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:33.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:33.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:34.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:34.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:34.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:34.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:34.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25658 2021-01-15T16:47:34.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25658 '!=' null 2021-01-15T16:47:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:35.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:35.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:35.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:35.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:35.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:35.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:35.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:35.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:35.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:35.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:35.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:35.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:36.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:36.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:36.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:36.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:36.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:36.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:36.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:36.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:36.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:36.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38140 2021-01-15T16:47:36.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38140 '!=' null 2021-01-15T16:47:36.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:36.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:36.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:36.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:36.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:36.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:36.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:36.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:36.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:37.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:37.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:37.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:37.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:37.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:37.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38140 2021-01-15T16:47:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38140 '!=' null 2021-01-15T16:47:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:38.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:38.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:38.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:38.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:38.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:38.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:38.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:38.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:38.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:39.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:39.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:39.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:39.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:39.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:39.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:39.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:39.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:39.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:39.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24963 2021-01-15T16:47:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24963 '!=' null 2021-01-15T16:47:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:39.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:39.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:40.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:40.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:40.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:40.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:40.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:40.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:41.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24963 2021-01-15T16:47:41.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24963 '!=' null 2021-01-15T16:47:41.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:41.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:41.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:41.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:41.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:41.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:41.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:41.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:41.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:41.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:41.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:41.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:41.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:41.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:41.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:42.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:42.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:42.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:42.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:42.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:42.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:42.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:42.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:42.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=38140 2021-01-15T16:47:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 38140 '!=' null 2021-01-15T16:47:42.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:42.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:42.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:42.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:42.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:42.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:42.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:42.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:42.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:42.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:42.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:43.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:43.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:43.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:43.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:43.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:43.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:43.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:43.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:43.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:44.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25658 2021-01-15T16:47:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25658 '!=' null 2021-01-15T16:47:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:44.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:44.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:44.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:44.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:44.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:44.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:44.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:44.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:45.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:45.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:45.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:45.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:45.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25658 2021-01-15T16:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25658 '!=' null 2021-01-15T16:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:45.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:45.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:45.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:45.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:45.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:45.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:45.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:45.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:45.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:46.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:46.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:46.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:46.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:46.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:46.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:46.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:46.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:46.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:46.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=39526 2021-01-15T16:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 39526 '!=' null 2021-01-15T16:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:47.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:47.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:47.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:47.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:47.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:47.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:47.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:47.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:47.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:47.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:47.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:47.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:48.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:48.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:48.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:48.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:48.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:48.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:48.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:48.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:48.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27045 2021-01-15T16:47:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27045 '!=' null 2021-01-15T16:47:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:48.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:48.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:48.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:48.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:48.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:48.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:48.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:48.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:48.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:49.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:49.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:49.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:49.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:49.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:49.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:49.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:49.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:49.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:49.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:49.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:49.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:49.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:50.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27045 2021-01-15T16:47:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27045 '!=' null 2021-01-15T16:47:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:50.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:50.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:50.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:50.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:50.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:50.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:50.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:50.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:50.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:50.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:50.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:50.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:50.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:51.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:51.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:51.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:51.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:51.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:51.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27045 2021-01-15T16:47:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27045 '!=' null 2021-01-15T16:47:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:52.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:52.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:52.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:52.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:52.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:52.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:52.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:52.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:52.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:52.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:52.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:52.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:52.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:52.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:52.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:52.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:52.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:52.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:52.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:52.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:52.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:53.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40913 2021-01-15T16:47:53.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40913 '!=' null 2021-01-15T16:47:53.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:53.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:53.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:53.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:53.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:53.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:53.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:53.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:53.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:53.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:53.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:53.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:54.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:54.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:54.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:54.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:54.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:54.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:54.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:54.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:54.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:54.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27738 2021-01-15T16:47:54.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27738 '!=' null 2021-01-15T16:47:54.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:54.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:54.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:55.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:55.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:55.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:55.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:55.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:55.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:55.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:55.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:55.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:55.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:55.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:55.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:55.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:55.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:55.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:55.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:55.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:56.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40220 2021-01-15T16:47:56.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40220 '!=' null 2021-01-15T16:47:56.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:56.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:56.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:56.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:56.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:56.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:56.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:56.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:57.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:57.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:57.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:57.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:57.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:57.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:57.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:57.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:57.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:58.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40220 2021-01-15T16:47:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40220 '!=' null 2021-01-15T16:47:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:58.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:58.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:58.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:58.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:58.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:58.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:58.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:58.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:47:58.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:47:58.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:47:58.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:47:58.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:47:59.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:47:59.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:47:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:47:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:47:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:47:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:47:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:47:59.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:47:59.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:47:59.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26351 2021-01-15T16:47:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26351 '!=' null 2021-01-15T16:47:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:47:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:47:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:47:59.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:47:59.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:47:59.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:47:59.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:47:59.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:47:59.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:47:59.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:47:59.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:48:00.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:48:00.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:48:00.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:48:00.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:48:00.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:48:00.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:48:00.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:48:00.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:48:00.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:48:00.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:48:00.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:48:00.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:48:00.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:48:01.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18723 2021-01-15T16:48:01.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18723 '!=' null 2021-01-15T16:48:01.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:48:01.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:48:01.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:48:01.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:48:01.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:48:01.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:48:01.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:48:01.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:48:01.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:48:01.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:48:01.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:48:01.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:48:01.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:48:01.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:48:01.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:48:02.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:48:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:48:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:48:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:48:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:48:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:48:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:48:02.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:48:02.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:48:02.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18723 2021-01-15T16:48:02.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18723 '!=' null 2021-01-15T16:48:02.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:48:02.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:48:02.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:48:02.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:48:02.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:48:02.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:48:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:48:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:48:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:48:02.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:48:02.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:48:03.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:48:03.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:48:03.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:48:03.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:48:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:48:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:48:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:48:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:48:03.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:48:03.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:48:03.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:48:03.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:48:03.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:48:04.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14562 2021-01-15T16:48:04.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14562 '!=' null 2021-01-15T16:48:04.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:48:04.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:48:04.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:48:04.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:48:04.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:48:04.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:48:04.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:48:04.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:48:04.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:48:04.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:48:04.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:48:04.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:48:04.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:48:04.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:48:04.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:48:05.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:48:05.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:48:05.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:48:05.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-01-15T16:48:05.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:48:05.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:48:05.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:48:05.667 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:48:05.667 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:48:05.667 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:48:05.667 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:48:05.667 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:48:05.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:48:05.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:05.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:48:05.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836508 2021-01-15T16:48:05.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836508 2021-01-15T16:48:05.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508' 2021-01-15T16:48:05.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:05.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:48:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672986 2021-01-15T16:48:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672986 2021-01-15T16:48:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986' 2021-01-15T16:48:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:05.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:48:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509464 2021-01-15T16:48:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509464 2021-01-15T16:48:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464' 2021-01-15T16:48:06.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:06.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:48:06.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345943 2021-01-15T16:48:06.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345943 2021-01-15T16:48:06.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943' 2021-01-15T16:48:06.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:06.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:48:06.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182421 2021-01-15T16:48:06.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182421 2021-01-15T16:48:06.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943 4-107374182421' 2021-01-15T16:48:06.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:48:06.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:48:06.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018900 2021-01-15T16:48:06.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018900 2021-01-15T16:48:06.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509464 3-85899345943 4-107374182421 5-128849018900' 2021-01-15T16:48:06.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:06.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836508 2021-01-15T16:48:06.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:06.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:48:06.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836508 2021-01-15T16:48:06.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:06.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836508 2021-01-15T16:48:06.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2021-01-15T16:48:06.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:48:06.507 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836508 2021-01-15T16:48:06.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836508 -lt 21474836508 2021-01-15T16:48:06.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:06.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672986 2021-01-15T16:48:06.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:06.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:48:06.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672986 2021-01-15T16:48:06.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:06.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672986 2021-01-15T16:48:06.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2021-01-15T16:48:06.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:48:06.897 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672986 2021-01-15T16:48:07.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672986 -lt 42949672986 2021-01-15T16:48:07.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:07.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509464 2021-01-15T16:48:07.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:07.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:48:07.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509464 2021-01-15T16:48:07.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:07.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509464 2021-01-15T16:48:07.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2021-01-15T16:48:07.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:48:07.338 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509464 2021-01-15T16:48:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509462 -lt 64424509464 2021-01-15T16:48:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:48:08.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:48:08.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:48:09.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509465 -lt 64424509464 2021-01-15T16:48:09.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:09.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345943 2021-01-15T16:48:09.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:09.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:48:09.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345943 2021-01-15T16:48:09.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:09.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345943 2021-01-15T16:48:09.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2021-01-15T16:48:09.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:48:09.141 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345943 2021-01-15T16:48:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345943 -lt 85899345943 2021-01-15T16:48:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:09.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182421 2021-01-15T16:48:09.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:09.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:48:09.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182421 2021-01-15T16:48:09.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:09.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182421 2021-01-15T16:48:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2021-01-15T16:48:09.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:48:09.550 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182421 2021-01-15T16:48:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182422 -lt 107374182421 2021-01-15T16:48:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:48:09.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018900 2021-01-15T16:48:09.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:48:09.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:48:09.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018900 2021-01-15T16:48:09.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:48:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018900 2021-01-15T16:48:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2021-01-15T16:48:09.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:48:09.958 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018900 2021-01-15T16:48:10.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018900 -lt 128849018900 2021-01-15T16:48:10.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:48:10.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-01-15T16:48:10.728 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:48:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-01-15T16:48:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-01-15T16:48:10.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:48:10.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:48:10.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:48:10.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:48:10.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:48:10.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:48:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-01-15T16:48:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:48:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-01-15T16:48:10.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:48:10.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2021-01-15T16:48:10.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:48:10.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:48:10.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:48:11.819 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:48:11.819 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_log_t": { 2021-01-15T16:48:11.820 INFO:tasks.workunit.client.0.smithi145.stdout: "head": "34'150", 2021-01-15T16:48:11.820 INFO:tasks.workunit.client.0.smithi145.stdout: "tail": "34'148", 2021-01-15T16:48:11.820 INFO:tasks.workunit.client.0.smithi145.stdout: "log": [ 2021-01-15T16:48:11.820 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.820 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:48:11.821 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e9c179f9:::obj-149:head", 2021-01-15T16:48:11.821 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'149", 2021-01-15T16:48:11.821 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:48:11.821 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4631.0:1", 2021-01-15T16:48:11.821 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:48:11.822 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:47:15.347446+0000", 2021-01-15T16:48:11.822 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:48:11.822 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:48:11.822 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:48:11.822 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:48:11.823 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:48:11.823 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:48:11.823 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.823 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.823 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:48:11.824 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:48:11.824 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:48:11.824 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:48:11.824 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:48:11.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.825 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.825 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.825 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.825 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:48:11.825 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cb39fd60:::obj-150:head", 2021-01-15T16:48:11.826 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'150", 2021-01-15T16:48:11.826 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:48:11.826 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4634.0:1", 2021-01-15T16:48:11.826 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:48:11.826 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:47:15.481402+0000", 2021-01-15T16:48:11.827 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:48:11.827 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:48:11.827 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:48:11.827 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:48:11.827 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:48:11.828 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:48:11.828 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.828 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.828 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:48:11.828 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.829 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:11.830 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:48:11.830 INFO:tasks.workunit.client.0.smithi145.stdout: "dups": [ 2021-01-15T16:48:11.830 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.830 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4187.0:1", 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'1", 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "1", 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.831 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4190.0:1", 2021-01-15T16:48:11.832 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'2", 2021-01-15T16:48:11.832 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "2", 2021-01-15T16:48:11.832 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.832 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.833 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.833 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4193.0:1", 2021-01-15T16:48:11.833 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'3", 2021-01-15T16:48:11.833 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "3", 2021-01-15T16:48:11.834 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.834 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.834 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4196.0:1", 2021-01-15T16:48:11.834 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'4", 2021-01-15T16:48:11.835 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "4", 2021-01-15T16:48:11.835 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.835 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.835 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.836 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4199.0:1", 2021-01-15T16:48:11.836 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'5", 2021-01-15T16:48:11.836 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "5", 2021-01-15T16:48:11.836 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.837 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.837 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.837 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4202.0:1", 2021-01-15T16:48:11.838 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'6", 2021-01-15T16:48:11.838 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "6", 2021-01-15T16:48:11.838 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.838 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.839 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.839 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4205.0:1", 2021-01-15T16:48:11.839 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'7", 2021-01-15T16:48:11.839 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "7", 2021-01-15T16:48:11.840 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.840 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.840 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.840 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4208.0:1", 2021-01-15T16:48:11.841 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'8", 2021-01-15T16:48:11.841 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "8", 2021-01-15T16:48:11.841 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.841 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.841 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.842 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4211.0:1", 2021-01-15T16:48:11.842 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'9", 2021-01-15T16:48:11.842 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "9", 2021-01-15T16:48:11.842 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.842 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.843 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.843 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4214.0:1", 2021-01-15T16:48:11.843 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'10", 2021-01-15T16:48:11.843 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "10", 2021-01-15T16:48:11.843 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.844 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.844 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.844 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4217.0:1", 2021-01-15T16:48:11.844 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'11", 2021-01-15T16:48:11.844 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "11", 2021-01-15T16:48:11.845 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.845 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.845 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.845 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4220.0:1", 2021-01-15T16:48:11.845 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'12", 2021-01-15T16:48:11.846 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "12", 2021-01-15T16:48:11.846 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.846 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.846 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.846 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4223.0:1", 2021-01-15T16:48:11.847 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'13", 2021-01-15T16:48:11.847 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "13", 2021-01-15T16:48:11.847 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.847 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.847 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.848 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4226.0:1", 2021-01-15T16:48:11.848 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'14", 2021-01-15T16:48:11.848 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "14", 2021-01-15T16:48:11.848 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.850 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.851 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.851 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4229.0:1", 2021-01-15T16:48:11.851 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'15", 2021-01-15T16:48:11.852 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "15", 2021-01-15T16:48:11.852 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.852 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.852 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.852 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4232.0:1", 2021-01-15T16:48:11.853 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'16", 2021-01-15T16:48:11.853 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "16", 2021-01-15T16:48:11.853 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.853 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.853 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.854 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4235.0:1", 2021-01-15T16:48:11.854 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'17", 2021-01-15T16:48:11.854 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "17", 2021-01-15T16:48:11.854 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.854 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.855 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.855 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4238.0:1", 2021-01-15T16:48:11.855 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'18", 2021-01-15T16:48:11.855 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "18", 2021-01-15T16:48:11.855 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.856 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.856 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.856 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4241.0:1", 2021-01-15T16:48:11.856 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'19", 2021-01-15T16:48:11.857 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "19", 2021-01-15T16:48:11.857 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.857 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.857 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.857 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4244.0:1", 2021-01-15T16:48:11.858 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'20", 2021-01-15T16:48:11.858 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "20", 2021-01-15T16:48:11.858 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.858 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.858 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.859 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4247.0:1", 2021-01-15T16:48:11.859 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'21", 2021-01-15T16:48:11.859 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "21", 2021-01-15T16:48:11.859 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.859 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.860 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.860 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4250.0:1", 2021-01-15T16:48:11.860 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'22", 2021-01-15T16:48:11.860 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "22", 2021-01-15T16:48:11.860 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.861 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.861 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.861 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4253.0:1", 2021-01-15T16:48:11.861 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'23", 2021-01-15T16:48:11.861 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "23", 2021-01-15T16:48:11.862 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.862 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.862 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.862 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4256.0:1", 2021-01-15T16:48:11.862 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'24", 2021-01-15T16:48:11.863 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "24", 2021-01-15T16:48:11.863 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.863 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.863 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4259.0:1", 2021-01-15T16:48:11.864 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'25", 2021-01-15T16:48:11.864 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "25", 2021-01-15T16:48:11.864 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.864 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.865 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.865 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4262.0:1", 2021-01-15T16:48:11.865 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'26", 2021-01-15T16:48:11.865 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "26", 2021-01-15T16:48:11.866 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.866 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.866 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.866 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4265.0:1", 2021-01-15T16:48:11.866 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'27", 2021-01-15T16:48:11.867 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "27", 2021-01-15T16:48:11.867 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.867 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.867 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.867 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4268.0:1", 2021-01-15T16:48:11.868 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'28", 2021-01-15T16:48:11.868 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "28", 2021-01-15T16:48:11.868 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.868 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.868 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4271.0:1", 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'29", 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "29", 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.869 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.870 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4274.0:1", 2021-01-15T16:48:11.870 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'30", 2021-01-15T16:48:11.870 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "30", 2021-01-15T16:48:11.870 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4277.0:1", 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'31", 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "31", 2021-01-15T16:48:11.871 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.872 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.872 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4280.0:1", 2021-01-15T16:48:11.872 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'32", 2021-01-15T16:48:11.872 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "32", 2021-01-15T16:48:11.873 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.873 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.873 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.873 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4283.0:1", 2021-01-15T16:48:11.874 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'33", 2021-01-15T16:48:11.874 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "33", 2021-01-15T16:48:11.874 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.874 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.874 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.875 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4286.0:1", 2021-01-15T16:48:11.875 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'34", 2021-01-15T16:48:11.875 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "34", 2021-01-15T16:48:11.875 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.875 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.876 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.876 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4289.0:1", 2021-01-15T16:48:11.876 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'35", 2021-01-15T16:48:11.876 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "35", 2021-01-15T16:48:11.876 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.877 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.877 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.877 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4292.0:1", 2021-01-15T16:48:11.877 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'36", 2021-01-15T16:48:11.877 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "36", 2021-01-15T16:48:11.878 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.878 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.878 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.878 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4295.0:1", 2021-01-15T16:48:11.878 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'37", 2021-01-15T16:48:11.879 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "37", 2021-01-15T16:48:11.879 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.879 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.879 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.879 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4298.0:1", 2021-01-15T16:48:11.880 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'38", 2021-01-15T16:48:11.880 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "38", 2021-01-15T16:48:11.880 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.880 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.880 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.881 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4301.0:1", 2021-01-15T16:48:11.881 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'39", 2021-01-15T16:48:11.881 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "39", 2021-01-15T16:48:11.881 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.881 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.882 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.882 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4304.0:1", 2021-01-15T16:48:11.882 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'40", 2021-01-15T16:48:11.882 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "40", 2021-01-15T16:48:11.882 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.883 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.883 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.883 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4307.0:1", 2021-01-15T16:48:11.883 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'41", 2021-01-15T16:48:11.883 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "41", 2021-01-15T16:48:11.884 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.884 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.884 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.884 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4310.0:1", 2021-01-15T16:48:11.884 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'42", 2021-01-15T16:48:11.885 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "42", 2021-01-15T16:48:11.885 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.885 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.885 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.885 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4313.0:1", 2021-01-15T16:48:11.886 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'43", 2021-01-15T16:48:11.886 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "43", 2021-01-15T16:48:11.886 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.886 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.886 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.887 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4316.0:1", 2021-01-15T16:48:11.887 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'44", 2021-01-15T16:48:11.887 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "44", 2021-01-15T16:48:11.887 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.887 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.888 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.888 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4319.0:1", 2021-01-15T16:48:11.888 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'45", 2021-01-15T16:48:11.888 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "45", 2021-01-15T16:48:11.888 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.889 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.889 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.889 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4322.0:1", 2021-01-15T16:48:11.889 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'46", 2021-01-15T16:48:11.889 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "46", 2021-01-15T16:48:11.890 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.890 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.890 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.890 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4325.0:1", 2021-01-15T16:48:11.890 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'47", 2021-01-15T16:48:11.891 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "47", 2021-01-15T16:48:11.891 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.891 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.891 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.892 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4328.0:1", 2021-01-15T16:48:11.892 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'48", 2021-01-15T16:48:11.892 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "48", 2021-01-15T16:48:11.892 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.892 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.893 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.893 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4331.0:1", 2021-01-15T16:48:11.893 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'49", 2021-01-15T16:48:11.893 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "49", 2021-01-15T16:48:11.893 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.894 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.894 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.894 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4334.0:1", 2021-01-15T16:48:11.894 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'50", 2021-01-15T16:48:11.894 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "50", 2021-01-15T16:48:11.895 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.895 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.895 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.895 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4337.0:1", 2021-01-15T16:48:11.895 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'51", 2021-01-15T16:48:11.896 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "51", 2021-01-15T16:48:11.896 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.896 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.896 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.896 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4340.0:1", 2021-01-15T16:48:11.897 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'52", 2021-01-15T16:48:11.897 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "52", 2021-01-15T16:48:11.897 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.897 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.897 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.898 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4343.0:1", 2021-01-15T16:48:11.898 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'53", 2021-01-15T16:48:11.898 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "53", 2021-01-15T16:48:11.898 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.898 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.899 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.899 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4346.0:1", 2021-01-15T16:48:11.899 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'54", 2021-01-15T16:48:11.899 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "54", 2021-01-15T16:48:11.899 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.900 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.900 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4349.0:1", 2021-01-15T16:48:11.900 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'55", 2021-01-15T16:48:11.900 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "55", 2021-01-15T16:48:11.901 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.901 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.901 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.901 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4352.0:1", 2021-01-15T16:48:11.901 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'56", 2021-01-15T16:48:11.902 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "56", 2021-01-15T16:48:11.902 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.902 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.902 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.903 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4355.0:1", 2021-01-15T16:48:11.903 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'57", 2021-01-15T16:48:11.903 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "57", 2021-01-15T16:48:11.903 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.904 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.904 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.904 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4358.0:1", 2021-01-15T16:48:11.904 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'58", 2021-01-15T16:48:11.904 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "58", 2021-01-15T16:48:11.905 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.905 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.905 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.905 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4361.0:1", 2021-01-15T16:48:11.905 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'59", 2021-01-15T16:48:11.906 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "59", 2021-01-15T16:48:11.906 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.906 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.906 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.906 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4364.0:1", 2021-01-15T16:48:11.907 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'60", 2021-01-15T16:48:11.907 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "60", 2021-01-15T16:48:11.907 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.907 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.907 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.908 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4367.0:1", 2021-01-15T16:48:11.908 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'61", 2021-01-15T16:48:11.908 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "61", 2021-01-15T16:48:11.908 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.909 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.909 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.909 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4370.0:1", 2021-01-15T16:48:11.909 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'62", 2021-01-15T16:48:11.909 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "62", 2021-01-15T16:48:11.910 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.910 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.910 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.910 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4373.0:1", 2021-01-15T16:48:11.910 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'63", 2021-01-15T16:48:11.911 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "63", 2021-01-15T16:48:11.911 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.911 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.911 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.911 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4376.0:1", 2021-01-15T16:48:11.912 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'64", 2021-01-15T16:48:11.912 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "64", 2021-01-15T16:48:11.912 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.912 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.912 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4379.0:1", 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'65", 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "65", 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.913 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.914 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4382.0:1", 2021-01-15T16:48:11.914 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'66", 2021-01-15T16:48:11.914 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "66", 2021-01-15T16:48:11.914 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4385.0:1", 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'67", 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "67", 2021-01-15T16:48:11.915 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.916 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.916 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.916 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4388.0:1", 2021-01-15T16:48:11.916 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'68", 2021-01-15T16:48:11.916 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "68", 2021-01-15T16:48:11.917 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.917 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.917 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.917 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4391.0:1", 2021-01-15T16:48:11.917 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'69", 2021-01-15T16:48:11.918 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "69", 2021-01-15T16:48:11.918 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.918 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.918 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.918 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4394.0:1", 2021-01-15T16:48:11.919 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'70", 2021-01-15T16:48:11.919 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "70", 2021-01-15T16:48:11.919 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.919 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.919 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.920 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4397.0:1", 2021-01-15T16:48:11.920 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'71", 2021-01-15T16:48:11.920 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "71", 2021-01-15T16:48:11.920 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.920 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.921 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.921 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4400.0:1", 2021-01-15T16:48:11.921 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'72", 2021-01-15T16:48:11.921 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "72", 2021-01-15T16:48:11.921 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.922 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.922 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.922 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4403.0:1", 2021-01-15T16:48:11.922 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'73", 2021-01-15T16:48:11.922 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "73", 2021-01-15T16:48:11.923 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.923 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.923 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.923 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4406.0:1", 2021-01-15T16:48:11.923 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'74", 2021-01-15T16:48:11.924 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "74", 2021-01-15T16:48:11.924 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.924 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.924 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.925 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4409.0:1", 2021-01-15T16:48:11.925 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'75", 2021-01-15T16:48:11.925 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "75", 2021-01-15T16:48:11.925 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.925 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.926 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.926 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4412.0:1", 2021-01-15T16:48:11.926 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'76", 2021-01-15T16:48:11.926 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "76", 2021-01-15T16:48:11.926 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.927 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.927 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.927 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4415.0:1", 2021-01-15T16:48:11.927 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'77", 2021-01-15T16:48:11.928 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "77", 2021-01-15T16:48:11.928 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.928 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.928 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.929 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4418.0:1", 2021-01-15T16:48:11.929 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'78", 2021-01-15T16:48:11.929 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "78", 2021-01-15T16:48:11.929 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.929 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.930 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.930 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4421.0:1", 2021-01-15T16:48:11.930 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'79", 2021-01-15T16:48:11.930 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "79", 2021-01-15T16:48:11.930 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.931 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.931 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.931 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4424.0:1", 2021-01-15T16:48:11.931 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'80", 2021-01-15T16:48:11.932 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "80", 2021-01-15T16:48:11.932 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.932 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.932 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.932 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4427.0:1", 2021-01-15T16:48:11.933 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'81", 2021-01-15T16:48:11.933 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "81", 2021-01-15T16:48:11.933 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.933 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.934 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4430.0:1", 2021-01-15T16:48:11.934 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'82", 2021-01-15T16:48:11.934 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "82", 2021-01-15T16:48:11.934 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.935 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.935 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.935 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4433.0:1", 2021-01-15T16:48:11.935 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'83", 2021-01-15T16:48:11.935 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "83", 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4436.0:1", 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'84", 2021-01-15T16:48:11.936 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "84", 2021-01-15T16:48:11.937 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.937 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.937 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.937 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4439.0:1", 2021-01-15T16:48:11.938 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'85", 2021-01-15T16:48:11.938 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "85", 2021-01-15T16:48:11.938 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.938 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.938 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.939 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4442.0:1", 2021-01-15T16:48:11.939 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'86", 2021-01-15T16:48:11.939 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "86", 2021-01-15T16:48:11.939 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.939 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.940 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.940 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4445.0:1", 2021-01-15T16:48:11.940 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'87", 2021-01-15T16:48:11.940 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "87", 2021-01-15T16:48:11.940 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.941 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.941 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.941 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4448.0:1", 2021-01-15T16:48:11.941 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'88", 2021-01-15T16:48:11.941 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "88", 2021-01-15T16:48:11.942 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.942 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.942 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.942 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4451.0:1", 2021-01-15T16:48:11.942 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'89", 2021-01-15T16:48:11.943 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "89", 2021-01-15T16:48:11.943 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.943 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.943 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.943 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4454.0:1", 2021-01-15T16:48:11.944 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'90", 2021-01-15T16:48:11.944 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "90", 2021-01-15T16:48:11.944 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.944 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.944 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.945 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4457.0:1", 2021-01-15T16:48:11.945 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'91", 2021-01-15T16:48:11.945 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "91", 2021-01-15T16:48:11.945 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.945 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.946 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.946 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4460.0:1", 2021-01-15T16:48:11.946 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'92", 2021-01-15T16:48:11.946 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "92", 2021-01-15T16:48:11.947 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.947 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.947 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.947 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4463.0:1", 2021-01-15T16:48:11.947 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'93", 2021-01-15T16:48:11.948 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "93", 2021-01-15T16:48:11.948 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.948 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.948 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.948 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4466.0:1", 2021-01-15T16:48:11.949 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'94", 2021-01-15T16:48:11.949 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "94", 2021-01-15T16:48:11.949 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.949 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.949 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.950 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4469.0:1", 2021-01-15T16:48:11.950 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'95", 2021-01-15T16:48:11.950 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "95", 2021-01-15T16:48:11.950 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.950 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.951 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.951 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4472.0:1", 2021-01-15T16:48:11.951 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'96", 2021-01-15T16:48:11.951 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "96", 2021-01-15T16:48:11.951 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.952 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.952 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.952 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4475.0:1", 2021-01-15T16:48:11.952 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'97", 2021-01-15T16:48:11.952 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "97", 2021-01-15T16:48:11.953 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.953 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.953 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.953 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4478.0:1", 2021-01-15T16:48:11.953 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'98", 2021-01-15T16:48:11.954 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "98", 2021-01-15T16:48:11.954 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.954 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.954 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.954 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4481.0:1", 2021-01-15T16:48:11.955 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'99", 2021-01-15T16:48:11.955 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "99", 2021-01-15T16:48:11.955 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.955 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.955 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.956 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4484.0:1", 2021-01-15T16:48:11.956 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'100", 2021-01-15T16:48:11.956 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "100", 2021-01-15T16:48:11.956 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.956 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.957 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.957 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4487.0:1", 2021-01-15T16:48:11.957 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'101", 2021-01-15T16:48:11.957 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "101", 2021-01-15T16:48:11.957 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.958 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.958 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.958 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4490.0:1", 2021-01-15T16:48:11.958 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'102", 2021-01-15T16:48:11.958 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "102", 2021-01-15T16:48:11.959 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.959 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.959 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.959 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4493.0:1", 2021-01-15T16:48:11.960 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'103", 2021-01-15T16:48:11.960 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "103", 2021-01-15T16:48:11.960 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.960 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.960 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.961 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4496.0:1", 2021-01-15T16:48:11.961 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'104", 2021-01-15T16:48:11.961 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "104", 2021-01-15T16:48:11.961 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.961 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.962 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.962 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4499.0:1", 2021-01-15T16:48:11.962 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'105", 2021-01-15T16:48:11.962 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "105", 2021-01-15T16:48:11.962 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.963 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.963 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.963 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4502.0:1", 2021-01-15T16:48:11.963 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'106", 2021-01-15T16:48:11.963 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "106", 2021-01-15T16:48:11.964 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.964 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.964 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.964 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4505.0:1", 2021-01-15T16:48:11.964 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'107", 2021-01-15T16:48:11.965 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "107", 2021-01-15T16:48:11.965 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.965 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.965 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.965 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4508.0:1", 2021-01-15T16:48:11.966 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'108", 2021-01-15T16:48:11.966 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "108", 2021-01-15T16:48:11.966 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.966 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.966 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.967 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4511.0:1", 2021-01-15T16:48:11.967 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'109", 2021-01-15T16:48:11.967 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "109", 2021-01-15T16:48:11.967 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.967 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.968 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.968 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4514.0:1", 2021-01-15T16:48:11.968 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'110", 2021-01-15T16:48:11.968 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "110", 2021-01-15T16:48:11.968 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.969 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.969 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.969 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4517.0:1", 2021-01-15T16:48:11.969 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'111", 2021-01-15T16:48:11.969 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "111", 2021-01-15T16:48:11.970 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.970 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.970 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.970 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4520.0:1", 2021-01-15T16:48:11.970 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'112", 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "112", 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4523.0:1", 2021-01-15T16:48:11.971 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'113", 2021-01-15T16:48:11.972 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "113", 2021-01-15T16:48:11.972 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.972 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.972 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.973 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4526.0:1", 2021-01-15T16:48:11.973 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'114", 2021-01-15T16:48:11.973 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "114", 2021-01-15T16:48:11.973 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.973 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.974 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.974 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4529.0:1", 2021-01-15T16:48:11.974 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'115", 2021-01-15T16:48:11.974 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "115", 2021-01-15T16:48:11.974 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.975 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.975 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.975 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4532.0:1", 2021-01-15T16:48:11.975 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'116", 2021-01-15T16:48:11.975 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "116", 2021-01-15T16:48:11.976 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.976 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.976 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.976 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4535.0:1", 2021-01-15T16:48:11.976 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'117", 2021-01-15T16:48:11.977 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "117", 2021-01-15T16:48:11.977 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.977 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.977 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.977 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4538.0:1", 2021-01-15T16:48:11.978 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'118", 2021-01-15T16:48:11.978 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "118", 2021-01-15T16:48:11.978 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.978 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.978 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.979 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4541.0:1", 2021-01-15T16:48:11.979 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'119", 2021-01-15T16:48:11.979 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "119", 2021-01-15T16:48:11.979 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4544.0:1", 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'120", 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "120", 2021-01-15T16:48:11.980 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.981 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.981 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.981 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4547.0:1", 2021-01-15T16:48:11.981 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'121", 2021-01-15T16:48:11.981 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "121", 2021-01-15T16:48:11.982 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.982 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.982 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.983 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4550.0:1", 2021-01-15T16:48:11.983 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'122", 2021-01-15T16:48:11.983 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "122", 2021-01-15T16:48:11.983 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.984 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.984 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.984 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4553.0:1", 2021-01-15T16:48:11.984 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'123", 2021-01-15T16:48:11.984 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "123", 2021-01-15T16:48:11.985 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.985 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.985 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.985 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4556.0:1", 2021-01-15T16:48:11.985 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'124", 2021-01-15T16:48:11.986 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "124", 2021-01-15T16:48:11.986 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.986 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.986 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.987 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4559.0:1", 2021-01-15T16:48:11.987 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'125", 2021-01-15T16:48:11.987 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "125", 2021-01-15T16:48:11.987 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.987 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.988 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.988 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4562.0:1", 2021-01-15T16:48:11.988 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'126", 2021-01-15T16:48:11.988 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "126", 2021-01-15T16:48:11.988 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.989 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.989 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4565.0:1", 2021-01-15T16:48:11.989 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'127", 2021-01-15T16:48:11.989 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "127", 2021-01-15T16:48:11.990 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.990 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.990 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.990 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4568.0:1", 2021-01-15T16:48:11.990 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'128", 2021-01-15T16:48:11.991 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "128", 2021-01-15T16:48:11.991 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.991 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.991 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.991 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4571.0:1", 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'129", 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "129", 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.992 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4574.0:1", 2021-01-15T16:48:11.993 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'130", 2021-01-15T16:48:11.993 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "130", 2021-01-15T16:48:11.993 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.993 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.993 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.994 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4577.0:1", 2021-01-15T16:48:11.994 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'131", 2021-01-15T16:48:11.994 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "131", 2021-01-15T16:48:11.994 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.994 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.995 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.995 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4580.0:1", 2021-01-15T16:48:11.995 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'132", 2021-01-15T16:48:11.995 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "132", 2021-01-15T16:48:11.995 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.996 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.996 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.996 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4583.0:1", 2021-01-15T16:48:11.996 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'133", 2021-01-15T16:48:11.996 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "133", 2021-01-15T16:48:11.997 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.997 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.997 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.997 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4586.0:1", 2021-01-15T16:48:11.997 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'134", 2021-01-15T16:48:11.998 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "134", 2021-01-15T16:48:11.998 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.998 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:11.998 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:11.999 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4589.0:1", 2021-01-15T16:48:11.999 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'135", 2021-01-15T16:48:11.999 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "135", 2021-01-15T16:48:11.999 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:11.999 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.000 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.000 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4592.0:1", 2021-01-15T16:48:12.000 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'136", 2021-01-15T16:48:12.000 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "136", 2021-01-15T16:48:12.000 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4595.0:1", 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'137", 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "137", 2021-01-15T16:48:12.001 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.002 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.002 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.002 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4598.0:1", 2021-01-15T16:48:12.002 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'138", 2021-01-15T16:48:12.002 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "138", 2021-01-15T16:48:12.003 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.003 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.003 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.004 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4601.0:1", 2021-01-15T16:48:12.004 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'139", 2021-01-15T16:48:12.004 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "139", 2021-01-15T16:48:12.004 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.004 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.005 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.005 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4604.0:1", 2021-01-15T16:48:12.005 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'140", 2021-01-15T16:48:12.005 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "140", 2021-01-15T16:48:12.005 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.006 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.006 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.006 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4607.0:1", 2021-01-15T16:48:12.006 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'141", 2021-01-15T16:48:12.006 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "141", 2021-01-15T16:48:12.007 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.007 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.007 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.007 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4610.0:1", 2021-01-15T16:48:12.007 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'142", 2021-01-15T16:48:12.008 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "142", 2021-01-15T16:48:12.008 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.008 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.008 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.008 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4613.0:1", 2021-01-15T16:48:12.009 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'143", 2021-01-15T16:48:12.009 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "143", 2021-01-15T16:48:12.009 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.009 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.009 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.010 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4616.0:1", 2021-01-15T16:48:12.010 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'144", 2021-01-15T16:48:12.010 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "144", 2021-01-15T16:48:12.010 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.010 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4619.0:1", 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'145", 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "145", 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.011 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.012 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.012 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4622.0:1", 2021-01-15T16:48:12.012 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'146", 2021-01-15T16:48:12.012 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "146", 2021-01-15T16:48:12.012 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.013 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.013 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.013 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4625.0:1", 2021-01-15T16:48:12.013 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'147", 2021-01-15T16:48:12.084 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "147", 2021-01-15T16:48:12.084 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.084 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:48:12.085 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4628.0:1", 2021-01-15T16:48:12.085 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "34'148", 2021-01-15T16:48:12.085 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "148", 2021-01-15T16:48:12.086 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:48:12.086 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:12.086 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:48:12.086 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:48:12.086 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_missing_t": { 2021-01-15T16:48:12.087 INFO:tasks.workunit.client.0.smithi145.stdout: "missing": [], 2021-01-15T16:48:12.087 INFO:tasks.workunit.client.0.smithi145.stdout: "may_include_deletes": true 2021-01-15T16:48:12.087 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:48:12.087 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:48:12.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:48:12.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2021-01-15T16:48:12.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2021-01-15T16:48:12.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:48:12.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=148 2021-01-15T16:48:12.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 148 '!=' 148 ']' 2021-01-15T16:48:12.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-01-15T16:48:12.095 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to max 2 2021-01-15T16:48:12.095 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'150 2021-01-15T16:48:12.096 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'149 2021-01-15T16:48:12.096 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:48:12.096 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'1 2021-01-15T16:48:12.096 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'2 2021-01-15T16:48:12.096 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'3 2021-01-15T16:48:12.097 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'4 2021-01-15T16:48:12.097 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'5 2021-01-15T16:48:12.097 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'6 2021-01-15T16:48:12.097 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'7 2021-01-15T16:48:12.098 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'8 2021-01-15T16:48:12.098 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'9 2021-01-15T16:48:12.098 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'10 2021-01-15T16:48:12.098 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'11 2021-01-15T16:48:12.098 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'12 2021-01-15T16:48:12.099 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'13 2021-01-15T16:48:12.099 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'14 2021-01-15T16:48:12.099 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'15 2021-01-15T16:48:12.099 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'16 2021-01-15T16:48:12.099 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'17 2021-01-15T16:48:12.100 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'18 2021-01-15T16:48:12.100 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'19 2021-01-15T16:48:12.100 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'20 2021-01-15T16:48:12.100 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'21 2021-01-15T16:48:12.100 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'22 2021-01-15T16:48:12.101 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'23 2021-01-15T16:48:12.101 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'24 2021-01-15T16:48:12.101 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'25 2021-01-15T16:48:12.101 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'26 2021-01-15T16:48:12.102 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'27 2021-01-15T16:48:12.102 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'28 2021-01-15T16:48:12.102 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'29 2021-01-15T16:48:12.102 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'30 2021-01-15T16:48:12.103 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'31 2021-01-15T16:48:12.103 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'32 2021-01-15T16:48:12.103 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'33 2021-01-15T16:48:12.104 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'34 2021-01-15T16:48:12.104 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'35 2021-01-15T16:48:12.104 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'36 2021-01-15T16:48:12.104 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'37 2021-01-15T16:48:12.104 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'38 2021-01-15T16:48:12.105 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'39 2021-01-15T16:48:12.105 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'40 2021-01-15T16:48:12.105 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'41 2021-01-15T16:48:12.105 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'42 2021-01-15T16:48:12.105 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'43 2021-01-15T16:48:12.106 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'44 2021-01-15T16:48:12.106 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'45 2021-01-15T16:48:12.106 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'46 2021-01-15T16:48:12.106 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'47 2021-01-15T16:48:12.106 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'48 2021-01-15T16:48:12.107 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'49 2021-01-15T16:48:12.107 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'50 2021-01-15T16:48:12.107 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'51 2021-01-15T16:48:12.107 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'52 2021-01-15T16:48:12.107 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'53 2021-01-15T16:48:12.108 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'54 2021-01-15T16:48:12.108 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'55 2021-01-15T16:48:12.108 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'56 2021-01-15T16:48:12.109 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'57 2021-01-15T16:48:12.109 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'58 2021-01-15T16:48:12.109 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'59 2021-01-15T16:48:12.109 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'60 2021-01-15T16:48:12.109 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'61 2021-01-15T16:48:12.110 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'62 2021-01-15T16:48:12.110 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'63 2021-01-15T16:48:12.110 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'64 2021-01-15T16:48:12.110 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'65 2021-01-15T16:48:12.110 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'66 2021-01-15T16:48:12.111 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'67 2021-01-15T16:48:12.111 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'68 2021-01-15T16:48:12.111 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'69 2021-01-15T16:48:12.111 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'70 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'71 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'72 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'73 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'74 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'75 2021-01-15T16:48:12.112 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'76 2021-01-15T16:48:12.113 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'77 2021-01-15T16:48:12.114 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'78 2021-01-15T16:48:12.114 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'79 2021-01-15T16:48:12.114 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'80 2021-01-15T16:48:12.114 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'81 2021-01-15T16:48:12.115 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'82 2021-01-15T16:48:12.115 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'83 2021-01-15T16:48:12.115 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'84 2021-01-15T16:48:12.115 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'85 2021-01-15T16:48:12.115 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'86 2021-01-15T16:48:12.116 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'87 2021-01-15T16:48:12.116 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'88 2021-01-15T16:48:12.116 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'89 2021-01-15T16:48:12.117 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'90 2021-01-15T16:48:12.117 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'91 2021-01-15T16:48:12.117 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'92 2021-01-15T16:48:12.117 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'93 2021-01-15T16:48:12.117 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'94 2021-01-15T16:48:12.118 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'95 2021-01-15T16:48:12.118 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'96 2021-01-15T16:48:12.118 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'97 2021-01-15T16:48:12.118 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'98 2021-01-15T16:48:12.119 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'99 2021-01-15T16:48:12.119 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'100 2021-01-15T16:48:12.119 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-01-15T16:48:12.119 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-01-15T16:48:12.119 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-01-15T16:48:12.120 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-01-15T16:48:12.120 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-01-15T16:48:12.120 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-01-15T16:48:12.120 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-01-15T16:48:12.120 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-01-15T16:48:12.121 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-01-15T16:48:12.121 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-01-15T16:48:12.121 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-01-15T16:48:12.121 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-01-15T16:48:12.121 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-01-15T16:48:12.122 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-01-15T16:48:12.122 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-01-15T16:48:12.122 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-01-15T16:48:12.122 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-01-15T16:48:12.122 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-01-15T16:48:12.123 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-01-15T16:48:12.123 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-01-15T16:48:12.123 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-01-15T16:48:12.123 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-01-15T16:48:12.124 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-01-15T16:48:12.124 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-01-15T16:48:12.124 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-01-15T16:48:12.125 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-01-15T16:48:12.125 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-01-15T16:48:12.125 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-01-15T16:48:12.125 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-01-15T16:48:12.126 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-01-15T16:48:12.126 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-01-15T16:48:12.126 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-01-15T16:48:12.126 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-01-15T16:48:12.127 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-01-15T16:48:12.127 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-01-15T16:48:12.127 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-01-15T16:48:12.127 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-01-15T16:48:12.127 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-01-15T16:48:12.128 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-01-15T16:48:12.128 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-01-15T16:48:12.128 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:48:12.129 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:48:12.129 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:48:12.129 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:48:12.129 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:48:12.129 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:48:12.130 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:48:12.130 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:48:12.130 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to max 2 2021-01-15T16:48:12.130 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'150 2021-01-15T16:48:12.130 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'149 2021-01-15T16:48:12.131 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:48:12.131 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'1 2021-01-15T16:48:12.131 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'2 2021-01-15T16:48:12.131 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'3 2021-01-15T16:48:12.132 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'4 2021-01-15T16:48:12.132 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'5 2021-01-15T16:48:12.132 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'6 2021-01-15T16:48:12.132 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'7 2021-01-15T16:48:12.132 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'8 2021-01-15T16:48:12.133 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'9 2021-01-15T16:48:12.133 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'10 2021-01-15T16:48:12.133 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'11 2021-01-15T16:48:12.133 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'12 2021-01-15T16:48:12.133 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'13 2021-01-15T16:48:12.134 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'14 2021-01-15T16:48:12.134 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'15 2021-01-15T16:48:12.135 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'16 2021-01-15T16:48:12.135 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'17 2021-01-15T16:48:12.135 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'18 2021-01-15T16:48:12.135 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'19 2021-01-15T16:48:12.135 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'20 2021-01-15T16:48:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'21 2021-01-15T16:48:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'22 2021-01-15T16:48:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'23 2021-01-15T16:48:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'24 2021-01-15T16:48:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'25 2021-01-15T16:48:12.137 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'26 2021-01-15T16:48:12.137 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'27 2021-01-15T16:48:12.137 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'28 2021-01-15T16:48:12.137 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'29 2021-01-15T16:48:12.138 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'30 2021-01-15T16:48:12.138 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'31 2021-01-15T16:48:12.138 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'32 2021-01-15T16:48:12.139 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'33 2021-01-15T16:48:12.139 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'34 2021-01-15T16:48:12.139 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'35 2021-01-15T16:48:12.140 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'36 2021-01-15T16:48:12.140 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'37 2021-01-15T16:48:12.140 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'38 2021-01-15T16:48:12.140 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'39 2021-01-15T16:48:12.140 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'40 2021-01-15T16:48:12.141 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'41 2021-01-15T16:48:12.141 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'42 2021-01-15T16:48:12.141 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'43 2021-01-15T16:48:12.141 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'44 2021-01-15T16:48:12.142 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'45 2021-01-15T16:48:12.142 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'46 2021-01-15T16:48:12.142 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'47 2021-01-15T16:48:12.142 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'48 2021-01-15T16:48:12.142 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'49 2021-01-15T16:48:12.143 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'50 2021-01-15T16:48:12.143 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'51 2021-01-15T16:48:12.143 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'52 2021-01-15T16:48:12.143 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'53 2021-01-15T16:48:12.143 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'54 2021-01-15T16:48:12.144 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'55 2021-01-15T16:48:12.144 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'56 2021-01-15T16:48:12.144 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'57 2021-01-15T16:48:12.144 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'58 2021-01-15T16:48:12.145 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'59 2021-01-15T16:48:12.145 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'60 2021-01-15T16:48:12.145 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'61 2021-01-15T16:48:12.145 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'62 2021-01-15T16:48:12.146 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'63 2021-01-15T16:48:12.146 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'64 2021-01-15T16:48:12.146 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'65 2021-01-15T16:48:12.146 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'66 2021-01-15T16:48:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'67 2021-01-15T16:48:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'68 2021-01-15T16:48:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'69 2021-01-15T16:48:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'70 2021-01-15T16:48:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'71 2021-01-15T16:48:12.148 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'72 2021-01-15T16:48:12.148 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'73 2021-01-15T16:48:12.148 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'74 2021-01-15T16:48:12.148 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'75 2021-01-15T16:48:12.149 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'76 2021-01-15T16:48:12.149 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'77 2021-01-15T16:48:12.149 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'78 2021-01-15T16:48:12.149 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'79 2021-01-15T16:48:12.149 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'80 2021-01-15T16:48:12.150 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'81 2021-01-15T16:48:12.150 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'82 2021-01-15T16:48:12.150 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'83 2021-01-15T16:48:12.150 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'84 2021-01-15T16:48:12.151 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'85 2021-01-15T16:48:12.151 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'86 2021-01-15T16:48:12.151 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'87 2021-01-15T16:48:12.151 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'88 2021-01-15T16:48:12.151 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'89 2021-01-15T16:48:12.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'90 2021-01-15T16:48:12.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'91 2021-01-15T16:48:12.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'92 2021-01-15T16:48:12.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'93 2021-01-15T16:48:12.152 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'94 2021-01-15T16:48:12.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'95 2021-01-15T16:48:12.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'96 2021-01-15T16:48:12.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'97 2021-01-15T16:48:12.153 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'98 2021-01-15T16:48:12.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'99 2021-01-15T16:48:12.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'100 2021-01-15T16:48:12.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-01-15T16:48:12.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-01-15T16:48:12.154 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-01-15T16:48:12.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-01-15T16:48:12.155 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-01-15T16:48:12.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-01-15T16:48:12.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-01-15T16:48:12.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-01-15T16:48:12.156 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-01-15T16:48:12.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-01-15T16:48:12.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-01-15T16:48:12.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-01-15T16:48:12.157 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-01-15T16:48:12.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-01-15T16:48:12.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-01-15T16:48:12.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-01-15T16:48:12.158 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-01-15T16:48:12.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-01-15T16:48:12.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-01-15T16:48:12.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-01-15T16:48:12.159 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-01-15T16:48:12.160 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-01-15T16:48:12.160 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-01-15T16:48:12.160 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-01-15T16:48:12.160 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-01-15T16:48:12.160 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-01-15T16:48:12.161 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-01-15T16:48:12.161 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-01-15T16:48:12.161 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-01-15T16:48:12.161 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-01-15T16:48:12.162 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-01-15T16:48:12.162 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-01-15T16:48:12.162 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-01-15T16:48:12.162 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-01-15T16:48:12.163 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-01-15T16:48:12.163 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-01-15T16:48:12.163 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-01-15T16:48:12.163 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-01-15T16:48:12.164 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-01-15T16:48:12.164 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-01-15T16:48:12.164 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:48:12.164 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:48:12.164 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:48:12.165 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:48:12.165 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:48:12.166 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:48:12.166 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:48:12.166 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:48:12.166 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to max 2 2021-01-15T16:48:12.167 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'150 2021-01-15T16:48:12.167 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to copy log version 34'149 2021-01-15T16:48:12.167 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:48:12.167 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'1 2021-01-15T16:48:12.167 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'2 2021-01-15T16:48:12.168 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'3 2021-01-15T16:48:12.168 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'4 2021-01-15T16:48:12.168 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'5 2021-01-15T16:48:12.168 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'6 2021-01-15T16:48:12.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'7 2021-01-15T16:48:12.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'8 2021-01-15T16:48:12.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'9 2021-01-15T16:48:12.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'10 2021-01-15T16:48:12.169 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'11 2021-01-15T16:48:12.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'12 2021-01-15T16:48:12.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'13 2021-01-15T16:48:12.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'14 2021-01-15T16:48:12.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'15 2021-01-15T16:48:12.170 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'16 2021-01-15T16:48:12.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'17 2021-01-15T16:48:12.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'18 2021-01-15T16:48:12.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'19 2021-01-15T16:48:12.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'20 2021-01-15T16:48:12.171 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'21 2021-01-15T16:48:12.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'22 2021-01-15T16:48:12.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'23 2021-01-15T16:48:12.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'24 2021-01-15T16:48:12.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'25 2021-01-15T16:48:12.172 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'26 2021-01-15T16:48:12.173 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'27 2021-01-15T16:48:12.173 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'28 2021-01-15T16:48:12.173 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'29 2021-01-15T16:48:12.173 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'30 2021-01-15T16:48:12.174 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'31 2021-01-15T16:48:12.174 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'32 2021-01-15T16:48:12.174 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'33 2021-01-15T16:48:12.174 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'34 2021-01-15T16:48:12.174 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'35 2021-01-15T16:48:12.175 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'36 2021-01-15T16:48:12.175 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'37 2021-01-15T16:48:12.175 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'38 2021-01-15T16:48:12.175 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'39 2021-01-15T16:48:12.175 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'40 2021-01-15T16:48:12.176 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'41 2021-01-15T16:48:12.176 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'42 2021-01-15T16:48:12.177 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'43 2021-01-15T16:48:12.177 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'44 2021-01-15T16:48:12.177 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'45 2021-01-15T16:48:12.177 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'46 2021-01-15T16:48:12.177 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'47 2021-01-15T16:48:12.178 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'48 2021-01-15T16:48:12.178 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'49 2021-01-15T16:48:12.178 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'50 2021-01-15T16:48:12.178 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'51 2021-01-15T16:48:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'52 2021-01-15T16:48:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'53 2021-01-15T16:48:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'54 2021-01-15T16:48:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'55 2021-01-15T16:48:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'56 2021-01-15T16:48:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'57 2021-01-15T16:48:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'58 2021-01-15T16:48:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'59 2021-01-15T16:48:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'60 2021-01-15T16:48:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'61 2021-01-15T16:48:12.181 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'62 2021-01-15T16:48:12.181 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'63 2021-01-15T16:48:12.181 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'64 2021-01-15T16:48:12.181 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'65 2021-01-15T16:48:12.181 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'66 2021-01-15T16:48:12.182 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'67 2021-01-15T16:48:12.182 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'68 2021-01-15T16:48:12.182 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'69 2021-01-15T16:48:12.182 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'70 2021-01-15T16:48:12.183 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'71 2021-01-15T16:48:12.183 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'72 2021-01-15T16:48:12.183 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'73 2021-01-15T16:48:12.183 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'74 2021-01-15T16:48:12.184 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'75 2021-01-15T16:48:12.184 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'76 2021-01-15T16:48:12.184 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'77 2021-01-15T16:48:12.184 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'78 2021-01-15T16:48:12.185 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'79 2021-01-15T16:48:12.185 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'80 2021-01-15T16:48:12.185 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'81 2021-01-15T16:48:12.185 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'82 2021-01-15T16:48:12.185 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'83 2021-01-15T16:48:12.186 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'84 2021-01-15T16:48:12.186 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'85 2021-01-15T16:48:12.186 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'86 2021-01-15T16:48:12.187 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'87 2021-01-15T16:48:12.187 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'88 2021-01-15T16:48:12.187 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'89 2021-01-15T16:48:12.187 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'90 2021-01-15T16:48:12.187 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'91 2021-01-15T16:48:12.188 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'92 2021-01-15T16:48:12.188 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'93 2021-01-15T16:48:12.188 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'94 2021-01-15T16:48:12.188 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'95 2021-01-15T16:48:12.189 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'96 2021-01-15T16:48:12.189 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'97 2021-01-15T16:48:12.189 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'98 2021-01-15T16:48:12.189 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'99 2021-01-15T16:48:12.189 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'100 2021-01-15T16:48:12.190 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-01-15T16:48:12.190 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-01-15T16:48:12.190 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-01-15T16:48:12.190 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-01-15T16:48:12.190 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-01-15T16:48:12.191 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-01-15T16:48:12.191 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-01-15T16:48:12.191 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-01-15T16:48:12.191 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-01-15T16:48:12.191 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-01-15T16:48:12.192 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-01-15T16:48:12.192 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-01-15T16:48:12.192 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-01-15T16:48:12.192 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-01-15T16:48:12.192 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-01-15T16:48:12.193 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-01-15T16:48:12.193 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-01-15T16:48:12.193 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-01-15T16:48:12.193 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-01-15T16:48:12.193 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-01-15T16:48:12.194 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-01-15T16:48:12.194 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-01-15T16:48:12.194 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-01-15T16:48:12.194 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-01-15T16:48:12.195 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-01-15T16:48:12.195 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-01-15T16:48:12.195 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-01-15T16:48:12.195 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-01-15T16:48:12.203 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-01-15T16:48:12.203 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-01-15T16:48:12.203 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-01-15T16:48:12.203 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-01-15T16:48:12.203 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-01-15T16:48:12.204 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-01-15T16:48:12.204 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-01-15T16:48:12.204 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-01-15T16:48:12.204 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-01-15T16:48:12.204 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-01-15T16:48:12.205 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-01-15T16:48:12.205 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-01-15T16:48:12.205 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:48:12.205 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:48:12.206 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:48:12.206 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:48:12.206 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:48:12.206 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:48:12.206 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:48:12.207 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:18.649+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:48:12.207 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_after v 34'150 2021-01-15T16:48:12.207 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:48:12.208 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'1 2021-01-15T16:48:12.208 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'2 2021-01-15T16:48:12.208 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'3 2021-01-15T16:48:12.208 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'4 2021-01-15T16:48:12.208 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'5 2021-01-15T16:48:12.209 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'6 2021-01-15T16:48:12.209 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'7 2021-01-15T16:48:12.209 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'8 2021-01-15T16:48:12.209 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'9 2021-01-15T16:48:12.210 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'10 2021-01-15T16:48:12.210 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'11 2021-01-15T16:48:12.210 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'12 2021-01-15T16:48:12.210 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'13 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'14 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'15 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'16 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'17 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'18 2021-01-15T16:48:12.211 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'19 2021-01-15T16:48:12.212 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'20 2021-01-15T16:48:12.212 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'21 2021-01-15T16:48:12.212 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'22 2021-01-15T16:48:12.212 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'23 2021-01-15T16:48:12.212 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'24 2021-01-15T16:48:12.213 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'25 2021-01-15T16:48:12.213 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'26 2021-01-15T16:48:12.213 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'27 2021-01-15T16:48:12.213 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'28 2021-01-15T16:48:12.213 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'29 2021-01-15T16:48:12.214 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'30 2021-01-15T16:48:12.214 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'31 2021-01-15T16:48:12.214 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'32 2021-01-15T16:48:12.214 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'33 2021-01-15T16:48:12.214 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'34 2021-01-15T16:48:12.215 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'35 2021-01-15T16:48:12.215 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'36 2021-01-15T16:48:12.215 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'37 2021-01-15T16:48:12.215 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'38 2021-01-15T16:48:12.215 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'39 2021-01-15T16:48:12.216 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'40 2021-01-15T16:48:12.216 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'41 2021-01-15T16:48:12.216 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'42 2021-01-15T16:48:12.216 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'43 2021-01-15T16:48:12.216 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'44 2021-01-15T16:48:12.217 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'45 2021-01-15T16:48:12.217 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'46 2021-01-15T16:48:12.218 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'47 2021-01-15T16:48:12.218 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'48 2021-01-15T16:48:12.218 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'49 2021-01-15T16:48:12.218 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'50 2021-01-15T16:48:12.219 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'51 2021-01-15T16:48:12.219 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'52 2021-01-15T16:48:12.219 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'53 2021-01-15T16:48:12.219 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'54 2021-01-15T16:48:12.219 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'55 2021-01-15T16:48:12.220 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'56 2021-01-15T16:48:12.220 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'57 2021-01-15T16:48:12.220 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'58 2021-01-15T16:48:12.220 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'59 2021-01-15T16:48:12.220 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'60 2021-01-15T16:48:12.221 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'61 2021-01-15T16:48:12.221 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'62 2021-01-15T16:48:12.221 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'63 2021-01-15T16:48:12.221 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'64 2021-01-15T16:48:12.221 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'65 2021-01-15T16:48:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'66 2021-01-15T16:48:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'67 2021-01-15T16:48:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'68 2021-01-15T16:48:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'69 2021-01-15T16:48:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'70 2021-01-15T16:48:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'71 2021-01-15T16:48:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'72 2021-01-15T16:48:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'73 2021-01-15T16:48:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'74 2021-01-15T16:48:12.223 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'75 2021-01-15T16:48:12.224 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'76 2021-01-15T16:48:12.224 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'77 2021-01-15T16:48:12.224 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'78 2021-01-15T16:48:12.224 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'79 2021-01-15T16:48:12.224 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'80 2021-01-15T16:48:12.225 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'81 2021-01-15T16:48:12.225 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'82 2021-01-15T16:48:12.225 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'83 2021-01-15T16:48:12.225 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'84 2021-01-15T16:48:12.225 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'85 2021-01-15T16:48:12.226 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'86 2021-01-15T16:48:12.226 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'87 2021-01-15T16:48:12.226 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'88 2021-01-15T16:48:12.226 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'89 2021-01-15T16:48:12.226 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'90 2021-01-15T16:48:12.227 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'91 2021-01-15T16:48:12.227 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'92 2021-01-15T16:48:12.227 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'93 2021-01-15T16:48:12.227 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'94 2021-01-15T16:48:12.227 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'95 2021-01-15T16:48:12.228 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'96 2021-01-15T16:48:12.228 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'97 2021-01-15T16:48:12.229 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'98 2021-01-15T16:48:12.229 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'99 2021-01-15T16:48:12.229 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup version 34'100 2021-01-15T16:48:12.229 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-01-15T16:48:12.230 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-01-15T16:48:12.230 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-01-15T16:48:12.230 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-01-15T16:48:12.230 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-01-15T16:48:12.230 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-01-15T16:48:12.231 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-01-15T16:48:12.231 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-01-15T16:48:12.231 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-01-15T16:48:12.231 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-01-15T16:48:12.231 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-01-15T16:48:12.232 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-01-15T16:48:12.232 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-01-15T16:48:12.232 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-01-15T16:48:12.232 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-01-15T16:48:12.232 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-01-15T16:48:12.233 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-01-15T16:48:12.233 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-01-15T16:48:12.233 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-01-15T16:48:12.233 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-01-15T16:48:12.233 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-01-15T16:48:12.234 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-01-15T16:48:12.234 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-01-15T16:48:12.234 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-01-15T16:48:12.234 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-01-15T16:48:12.234 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-01-15T16:48:12.235 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-01-15T16:48:12.235 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-01-15T16:48:12.235 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-01-15T16:48:12.235 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-01-15T16:48:12.235 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-01-15T16:48:12.236 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-01-15T16:48:12.236 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-01-15T16:48:12.236 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-01-15T16:48:12.236 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-01-15T16:48:12.236 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-01-15T16:48:12.237 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-01-15T16:48:12.237 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-01-15T16:48:12.237 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-01-15T16:48:12.237 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-01-15T16:48:12.237 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-01-15T16:48:12.238 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-01-15T16:48:12.238 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-01-15T16:48:12.238 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-01-15T16:48:12.238 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-01-15T16:48:12.239 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-01-15T16:48:12.239 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-01-15T16:48:12.239 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-01-15T16:48:12.239 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'149 2021-01-15T16:48:12.239 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:47:59.722+0000 7f3732409700 20 copy_up_to/copy_after copy dup from log version 34'150 2021-01-15T16:48:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-01-15T16:48:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:48:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-01-15T16:48:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:48:12.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:48:12.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:48:12.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:48:12.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:48:12.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:48:12.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:48:12.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:48:12.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:48:12.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:48:12.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:48:12.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:48:12.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:48:12.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:48:12.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:48:12.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:48:12.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:48:12.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:48:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:48:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:48:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-01-15T16:48:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-01-15T16:48:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-01-15T16:48:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:48:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:48:12.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:48:12.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:48:12.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:48:12.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:48:12.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:48:12.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:48:12.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:48:12.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:48:12.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:48:12.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:48:12.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:48:12.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:48:12.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:48:12.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:48:12.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:48:12.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:48:12.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:48:12.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:48:12.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:48:12.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:48:12.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:48:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:48:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:48:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-01-15T16:48:12.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:48:12.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119796 2021-01-15T16:48:12.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:48:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:48:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:48:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-01-15T16:48:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2021-01-15T16:48:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:121: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:123: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2021-01-15T16:48:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2021-01-15T16:48:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2021-01-15T16:48:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2021-01-15T16:48:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-01-15T16:48:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-01-15T16:48:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-01-15T16:48:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:48:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:48:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:48:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-01-15T16:48:12.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:48:12.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:12.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:12.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:12.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:12.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:48:12.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:48:12.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:48:12.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:48:12.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:48:12.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:48:12.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:48:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:48:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:48:12.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:48:12.653 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:48:12.653 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.653 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:48:12.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:48:12.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get fsid 2021-01-15T16:48:12.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:48:12.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:48:12.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:48:12.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:48:12.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:48:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:48:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:48:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:48:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:48:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:48:12.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:48:12.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get mon_host 2021-01-15T16:48:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-01-15T16:48:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:48:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:48:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:48:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-01-15T16:48:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:48:13.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:48:13.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:13.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:13.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:13.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:13.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:48:13.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:48:13.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-01-15T16:48:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:13.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-01-15T16:48:13.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-01-15T16:48:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-01-15T16:48:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:48:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-01-15T16:48:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-01-15T16:48:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-01-15T16:48:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:13.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:13.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:13.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:13.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:13.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:13.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:13.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:13.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:13.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:13.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:13.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:13.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-01-15T16:48:13.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:13.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1d840fef-929c-4d00-b013-2c7884239cb9 2021-01-15T16:48:13.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1d840fef-929c-4d00-b013-2c7884239cb9' 2021-01-15T16:48:13.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:13.791 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 1d840fef-929c-4d00-b013-2c7884239cb9 2021-01-15T16:48:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBNxwFgD8OHLxAApDnM1Z+OZiVzie8pQnaigw== 2021-01-15T16:48:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBNxwFgD8OHLxAApDnM1Z+OZiVzie8pQnaigw=="}' 2021-01-15T16:48:13.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1d840fef-929c-4d00-b013-2c7884239cb9 -i td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:48:14.175 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:14.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:48:14.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBNxwFgD8OHLxAApDnM1Z+OZiVzie8pQnaigw== --osd-uuid 1d840fef-929c-4d00-b013-2c7884239cb9 2021-01-15T16:48:14.212 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.211+0000 7f8b9ee68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:14.213 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.213+0000 7f8b9ee68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.215+0000 7f8b9ee68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.216+0000 7f8b9ee68080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:48:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.216+0000 7f8b9ee68080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:48:14.217 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.216+0000 7f8b9ee68080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:48:14.257 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:14.256+0000 7f8b9ee68080 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-01-15T16:48:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-01-15T16:48:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:48:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:18.916 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:48:19.441 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:48:19.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:48:19.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:19.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:48:19.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:48:19.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:48:19.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:19.466+0000 7fb92c0ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:19.481 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:19.480+0000 7fb92c0ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:19.482+0000 7fb92c0ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:19.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:48:19.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:48:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:48:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:48:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:48:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:19.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:48:19.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:19.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:48:19.870 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:20.000 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:19.999+0000 7fb92c0ae080 -1 Falling back to public interface 2021-01-15T16:48:20.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:48:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:20.790+0000 7fb92c0ae080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:48:21.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:48:21.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:21.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:48:21.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:21.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:48:21.264 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:21.683 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/149680,v1:127.0.0.1:6803/149680] [v2:127.0.0.1:6804/149680,v1:127.0.0.1:6805/149680] exists,up 1d840fef-929c-4d00-b013-2c7884239cb9 2021-01-15T16:48:21.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:48:21.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:48:21.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:48:21.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:21.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-01-15T16:48:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:48:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-01-15T16:48:21.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:21.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:21.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:21.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:21.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-01-15T16:48:21.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-01-15T16:48:21.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:21.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:21.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:21.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:21.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:21.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-01-15T16:48:21.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:21.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=60e1bfaf-777f-4e63-8bde-b611f85f9ff2 2021-01-15T16:48:21.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 60e1bfaf-777f-4e63-8bde-b611f85f9ff2' 2021-01-15T16:48:21.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:21.693 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 60e1bfaf-777f-4e63-8bde-b611f85f9ff2 2021-01-15T16:48:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBVxwFghK2aKRAA/xEK6wtj0AJVZGFTmH1H1w== 2021-01-15T16:48:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBVxwFghK2aKRAA/xEK6wtj0AJVZGFTmH1H1w=="}' 2021-01-15T16:48:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 60e1bfaf-777f-4e63-8bde-b611f85f9ff2 -i td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:48:22.222 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:48:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBVxwFghK2aKRAA/xEK6wtj0AJVZGFTmH1H1w== --osd-uuid 60e1bfaf-777f-4e63-8bde-b611f85f9ff2 2021-01-15T16:48:22.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.260+0000 7fbb907b0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:22.262 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.262+0000 7fbb907b0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:22.263 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.262+0000 7fbb907b0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:22.263 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.263+0000 7fbb907b0080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:48:22.263 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.263+0000 7fbb907b0080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:48:22.264 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.263+0000 7fbb907b0080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:48:22.309 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:22.309+0000 7fbb907b0080 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-01-15T16:48:27.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-01-15T16:48:27.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:27.080 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:48:27.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:48:27.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:27.601 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:48:27.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:48:27.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:27.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:48:27.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:48:27.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:48:27.626 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:27.625+0000 7f2843a42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:27.652 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:27.652+0000 7f2843a42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:27.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:27.654+0000 7f2843a42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:48:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:48:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:48:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:48:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:48:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:48:28.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:28.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:48:28.030 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:28.168+0000 7f2843a42080 -1 Falling back to public interface 2021-01-15T16:48:28.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:48:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:28.987+0000 7f2843a42080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:48:29.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:48:29.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:29.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:48:29.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:29.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:48:29.420 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:29.831 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/150944,v1:127.0.0.1:6811/150944] [v2:127.0.0.1:6812/150944,v1:127.0.0.1:6813/150944] exists,up 60e1bfaf-777f-4e63-8bde-b611f85f9ff2 2021-01-15T16:48:29.831 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 6a018128-0a10-4faf-939c-339927384322 2021-01-15T16:48:29.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:48:29.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:48:29.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:48:29.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:29.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-01-15T16:48:29.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:29.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:29.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:48:29.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:29.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-01-15T16:48:29.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:29.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-01-15T16:48:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-01-15T16:48:29.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:29.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:29.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:29.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:29.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:29.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:29.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:29.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-01-15T16:48:29.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:29.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6a018128-0a10-4faf-939c-339927384322 2021-01-15T16:48:29.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 6a018128-0a10-4faf-939c-339927384322' 2021-01-15T16:48:29.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:29.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBdxwFgbo1YMhAAgwrixVrwTxMzqVg8qZcSKQ== 2021-01-15T16:48:29.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBdxwFgbo1YMhAAgwrixVrwTxMzqVg8qZcSKQ=="}' 2021-01-15T16:48:29.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6a018128-0a10-4faf-939c-339927384322 -i td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:48:30.379 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:48:30.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:48:30.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBdxwFgbo1YMhAAgwrixVrwTxMzqVg8qZcSKQ== --osd-uuid 6a018128-0a10-4faf-939c-339927384322 2021-01-15T16:48:30.417 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.416+0000 7f5f13c29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:30.418 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.418+0000 7f5f13c29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:30.420 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.420+0000 7f5f13c29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:30.421 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.420+0000 7f5f13c29080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:48:30.422 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.420+0000 7f5f13c29080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:48:30.422 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.420+0000 7f5f13c29080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:48:30.460 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:30.459+0000 7f5f13c29080 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-01-15T16:48:35.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-01-15T16:48:35.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:35.207 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:48:35.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:48:35.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:35.730 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:48:35.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:48:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:48:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:48:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:48:35.754 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:35.753+0000 7fa7b8019080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:35.756 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:35.755+0000 7fa7b8019080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:35.757 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:35.757+0000 7fa7b8019080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:36.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:48:36.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:48:36.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:48:36.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:48:36.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:48:36.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:36.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:48:36.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:36.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:48:36.122 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:36.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:36.272+0000 7fa7b8019080 -1 Falling back to public interface 2021-01-15T16:48:36.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:48:37.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:37.109+0000 7fa7b8019080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:48:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:48:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:48:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:48:37.534 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:37.972 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/152216,v1:127.0.0.1:6819/152216] [v2:127.0.0.1:6820/152216,v1:127.0.0.1:6821/152216] exists,up 6a018128-0a10-4faf-939c-339927384322 2021-01-15T16:48:37.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:48:37.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:48:37.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:48:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-01-15T16:48:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:48:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-01-15T16:48:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-01-15T16:48:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-01-15T16:48:37.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:37.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:37.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:37.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:37.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:37.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:37.978 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:37.978 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:37.978 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:37.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:37.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:37.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:37.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:37.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:37.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:37.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:37.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:37.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-01-15T16:48:37.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:37.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=571b0f36-591a-4184-af60-d21eef07dd45 2021-01-15T16:48:37.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 571b0f36-591a-4184-af60-d21eef07dd45' 2021-01-15T16:48:37.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:37.983 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 571b0f36-591a-4184-af60-d21eef07dd45 2021-01-15T16:48:37.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBlxwFgSqXfOhAAh5zKNGJnUQ7ez4+4OPIqsQ== 2021-01-15T16:48:37.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBlxwFgSqXfOhAAh5zKNGJnUQ7ez4+4OPIqsQ=="}' 2021-01-15T16:48:37.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 571b0f36-591a-4184-af60-d21eef07dd45 -i td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:48:38.536 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:48:38.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:48:38.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBlxwFgSqXfOhAAh5zKNGJnUQ7ez4+4OPIqsQ== --osd-uuid 571b0f36-591a-4184-af60-d21eef07dd45 2021-01-15T16:48:38.574 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.574+0000 7f8d28edb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:38.576 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.576+0000 7f8d28edb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.577+0000 7f8d28edb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:38.578 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.578+0000 7f8d28edb080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:48:38.579 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.578+0000 7f8d28edb080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:48:38.579 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.578+0000 7f8d28edb080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:48:38.666 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:38.666+0000 7f8d28edb080 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-01-15T16:48:43.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-01-15T16:48:43.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:43.177 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:48:43.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:48:43.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:43.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:48:43.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:43.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:48:43.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:48:43.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:48:43.707 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:48:43.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:43.729+0000 7fb1f02ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:43.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:43.731+0000 7fb1f02ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:43.733 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:43.732+0000 7fb1f02ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:44.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:48:44.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:48:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:48:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:48:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:48:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:44.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:48:44.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:44.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:48:44.181 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:44.248 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:44.248+0000 7fb1f02ca080 -1 Falling back to public interface 2021-01-15T16:48:44.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:48:45.052 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:45.051+0000 7fb1f02ca080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:48:45.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:48:45.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:45.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:48:45.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:45.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:48:45.572 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:45.979 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/153480,v1:127.0.0.1:6827/153480] [v2:127.0.0.1:6828/153480,v1:127.0.0.1:6829/153480] exists,up 571b0f36-591a-4184-af60-d21eef07dd45 2021-01-15T16:48:45.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:48:45.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:48:45.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:48:45.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:45.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-01-15T16:48:45.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:45.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:45.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T16:48:45.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:45.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:48:45.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:45.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:45.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:45.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:45.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-01-15T16:48:45.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-01-15T16:48:45.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:45.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:45.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:45.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:45.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:45.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:45.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:45.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:45.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:45.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:45.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:45.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:45.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:45.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:45.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:45.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:45.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:45.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:45.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:45.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:45.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:45.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-01-15T16:48:45.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:45.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=691c112a-6080-4b09-a8da-460743d48d3a 2021-01-15T16:48:45.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 691c112a-6080-4b09-a8da-460743d48d3a' 2021-01-15T16:48:45.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:45.989 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 691c112a-6080-4b09-a8da-460743d48d3a 2021-01-15T16:48:45.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBtxwFgpXk/OxAAJXQpG+OpZZiZ755P9gl2zw== 2021-01-15T16:48:45.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBtxwFgpXk/OxAAJXQpG+OpZZiZ755P9gl2zw=="}' 2021-01-15T16:48:45.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 691c112a-6080-4b09-a8da-460743d48d3a -i td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:48:46.661 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T16:48:46.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:48:46.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBtxwFgpXk/OxAAJXQpG+OpZZiZ755P9gl2zw== --osd-uuid 691c112a-6080-4b09-a8da-460743d48d3a 2021-01-15T16:48:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.699+0000 7f814b319080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:46.701 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.701+0000 7f814b319080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.702+0000 7f814b319080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.703+0000 7f814b319080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:48:46.704 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.703+0000 7f814b319080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:48:46.704 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.703+0000 7f814b319080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:48:46.759 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:46.759+0000 7f814b319080 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-01-15T16:48:51.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-01-15T16:48:51.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:51.536 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T16:48:51.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T16:48:51.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T16:48:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:48:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:52.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:48:52.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:48:52.060 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T16:48:52.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:52.082+0000 7fcd6f9ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:52.084 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:52.084+0000 7fcd6f9ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:52.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:52.086+0000 7fcd6f9ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:52.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T16:48:52.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:48:52.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T16:48:52.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:48:52.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:48:52.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:52.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:48:52.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:52.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:48:52.465 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:48:52.601 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:52.601+0000 7fcd6f9ac080 -1 Falling back to public interface 2021-01-15T16:48:52.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:48:53.404 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:53.403+0000 7fcd6f9ac080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T16:48:53.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:48:53.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:48:53.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:48:53.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:48:53.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:48:53.856 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:48:54.343 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/154745,v1:127.0.0.1:6835/154745] [v2:127.0.0.1:6836/154745,v1:127.0.0.1:6837/154745] exists,up 691c112a-6080-4b09-a8da-460743d48d3a 2021-01-15T16:48:54.343 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 86c8a8f2-6b2c-4dbe-ba91-f35a875ee023 2021-01-15T16:48:54.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:48:54.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:48:54.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:48:54.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:48:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-01-15T16:48:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:48:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:48:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T16:48:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:48:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-01-15T16:48:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:48:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:48:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:48:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:48:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-01-15T16:48:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-01-15T16:48:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:48:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-01-15T16:48:54.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:48:54.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:48:54.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:48:54.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:48:54.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:48:54.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:48:54.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:48:54.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:54.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:48:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:48:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:48:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:48:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:48:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:48:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:48:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:48:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:48:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:48:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:48:54.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-01-15T16:48:54.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:48:54.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=86c8a8f2-6b2c-4dbe-ba91-f35a875ee023 2021-01-15T16:48:54.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 86c8a8f2-6b2c-4dbe-ba91-f35a875ee023' 2021-01-15T16:48:54.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:48:54.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB2xwFg2yVNFRAAlkuudOogo9OlytZI/2JBDQ== 2021-01-15T16:48:54.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB2xwFg2yVNFRAAlkuudOogo9OlytZI/2JBDQ=="}' 2021-01-15T16:48:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 86c8a8f2-6b2c-4dbe-ba91-f35a875ee023 -i td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:48:54.845 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T16:48:54.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:48:54.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB2xwFg2yVNFRAAlkuudOogo9OlytZI/2JBDQ== --osd-uuid 86c8a8f2-6b2c-4dbe-ba91-f35a875ee023 2021-01-15T16:48:54.882 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.882+0000 7f74f2e97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:54.901 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.901+0000 7f74f2e97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:54.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.902+0000 7f74f2e97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:48:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.903+0000 7f74f2e97080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:48:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.903+0000 7f74f2e97080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:48:54.904 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.903+0000 7f74f2e97080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:48:54.968 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:48:54.968+0000 7f74f2e97080 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-01-15T16:48:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-01-15T16:48:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:48:59.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T16:48:59.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:48:59.707 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T16:49:00.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T16:49:00.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:00.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:49:00.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:49:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:49:00.237 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T16:49:00.257 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:00.257+0000 7f51d1fb6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:00.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:00.259+0000 7f51d1fb6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:00.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:00.260+0000 7f51d1fb6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T16:49:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:49:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T16:49:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:49:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:49:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:49:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:00.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:49:00.628 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:49:00.776 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:00.776+0000 7f51d1fb6080 -1 Falling back to public interface 2021-01-15T16:49:01.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:49:01.628 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:01.628+0000 7f51d1fb6080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T16:49:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:49:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:49:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:49:02.028 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:49:02.426 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/156009,v1:127.0.0.1:6843/156009] [v2:127.0.0.1:6844/156009,v1:127.0.0.1:6845/156009] exists,up 86c8a8f2-6b2c-4dbe-ba91-f35a875ee023 2021-01-15T16:49:02.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:49:02.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:49:02.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:49:02.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-01-15T16:49:02.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:49:02.917 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:49:02.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:49:03.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 40 2021-01-15T16:49:03.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:03.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-01-15T16:49:04.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:04.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-01-15T16:49:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-01-15T16:49:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-01-15T16:49:04.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:04.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-01-15T16:49:05.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-01-15T16:49:05.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-01-15T16:49:05.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-01-15T16:49:05.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-01-15T16:49:05.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-01-15T16:49:05.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:05.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-01-15T16:49:06.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-01-15T16:49:06.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-01-15T16:49:06.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-01-15T16:49:06.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-01-15T16:49:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-01-15T16:49:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-01-15T16:49:06.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:06.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-01-15T16:49:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-01-15T16:49:07.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-01-15T16:49:07.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-01-15T16:49:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-01-15T16:49:07.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-01-15T16:49:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:07.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-01-15T16:49:08.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-01-15T16:49:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-01-15T16:49:08.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-01-15T16:49:08.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-01-15T16:49:08.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-01-15T16:49:08.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:08.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-01-15T16:49:09.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-01-15T16:49:09.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-01-15T16:49:09.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-01-15T16:49:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-01-15T16:49:09.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-01-15T16:49:09.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-01-15T16:49:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-01-15T16:49:10.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:10.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-01-15T16:49:10.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:10.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-01-15T16:49:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:49:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-01-15T16:49:10.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:49:10.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-01-15T16:49:10.950 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:49:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-01-15T16:49:11.503 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-01-15T16:49:11.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:49:11.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-01-15T16:49:12.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-01-15T16:49:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:49:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:49:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:49:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:49:12.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:49:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:49:12.516 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:49:12.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:49:12.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:49:12.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:49:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:49:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:49:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:49:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:49:12.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:49:13.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:49:13.009 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:49:13.009 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:49:13.009 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:49:13.010 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:49:13.010 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:49:13.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:49:13.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:49:13.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836494 2021-01-15T16:49:13.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836494 2021-01-15T16:49:13.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494' 2021-01-15T16:49:13.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:49:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T16:49:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T16:49:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672971' 2021-01-15T16:49:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:49:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2021-01-15T16:49:13.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2021-01-15T16:49:13.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672971 2-64424509449' 2021-01-15T16:49:13.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:49:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345928 2021-01-15T16:49:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345928 2021-01-15T16:49:13.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672971 2-64424509449 3-85899345928' 2021-01-15T16:49:13.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:49:13.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T16:49:13.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T16:49:13.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672971 2-64424509449 3-85899345928 4-107374182406' 2021-01-15T16:49:13.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:13.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:49:13.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018885 2021-01-15T16:49:13.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018885 2021-01-15T16:49:13.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836494 1-42949672971 2-64424509449 3-85899345928 4-107374182406 5-128849018885' 2021-01-15T16:49:13.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:13.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836494 2021-01-15T16:49:13.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:13.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:49:13.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836494 2021-01-15T16:49:13.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:13.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836494 2021-01-15T16:49:13.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2021-01-15T16:49:13.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:49:13.832 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836494 2021-01-15T16:49:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836494 2021-01-15T16:49:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:14.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T16:49:14.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:14.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:49:14.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T16:49:14.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:14.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T16:49:14.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T16:49:14.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:49:14.227 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T16:49:14.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T16:49:14.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:14.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2021-01-15T16:49:14.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:14.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:49:14.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-01-15T16:49:14.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:14.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2021-01-15T16:49:14.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-01-15T16:49:14.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:49:14.620 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2021-01-15T16:49:15.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509449 2021-01-15T16:49:15.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:15.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345928 2021-01-15T16:49:15.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:15.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:49:15.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345928 2021-01-15T16:49:15.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:15.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345928 2021-01-15T16:49:15.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-01-15T16:49:15.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:49:15.019 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345928 2021-01-15T16:49:15.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345928 2021-01-15T16:49:15.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:49:16.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:49:16.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:49:16.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345928 2021-01-15T16:49:16.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:16.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182406 2021-01-15T16:49:16.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:16.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:49:16.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182406 2021-01-15T16:49:16.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:16.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T16:49:16.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-01-15T16:49:16.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:49:16.827 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182406 2021-01-15T16:49:17.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182407 -lt 107374182406 2021-01-15T16:49:17.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:17.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018885 2021-01-15T16:49:17.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:17.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:49:17.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018885 2021-01-15T16:49:17.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:17.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018885 2021-01-15T16:49:17.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2021-01-15T16:49:17.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:49:17.320 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018885 2021-01-15T16:49:17.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018885 2021-01-15T16:49:17.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:49:17.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:17.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:18.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:49:18.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:18.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:18.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:18.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:49:18.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:18.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:18.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:19.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:49:19.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:49:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:49:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:49:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:49:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:49:19.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:49:19.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:19.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:19.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:19.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:19.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:19.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:19.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:19.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:49:19.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:19.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:19.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:20.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:49:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:49:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:49:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:49:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:49:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:49:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:49:20.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:49:20.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:49:20.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8150 2021-01-15T16:49:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8150 '!=' null 2021-01-15T16:49:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:49:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:49:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:49:20.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:49:20.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:20.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:20.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:20.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:20.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:20.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:20.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:21.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:49:21.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:21.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:21.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:21.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:49:21.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:49:21.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:49:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:49:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:49:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:49:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:49:21.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:49:21.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:49:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20803 2021-01-15T16:49:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20803 '!=' null 2021-01-15T16:49:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:49:22.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:49:22.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:49:22.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:49:22.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:22.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:22.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:22.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:22.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:22.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:22.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:22.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:49:22.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:22.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:22.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:23.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:49:23.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:49:23.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:49:23.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:49:23.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:49:23.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:49:23.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:49:23.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:49:23.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:49:23.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17852 2021-01-15T16:49:23.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17852 '!=' null 2021-01-15T16:49:23.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:49:23.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:49:23.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:49:23.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:49:23.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:23.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:23.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:23.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:23.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:23.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:23.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:24.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:49:24.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:24.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:24.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:24.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:49:24.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:49:24.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:49:24.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:49:24.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:49:24.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:49:24.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:49:24.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:49:24.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:49:25.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17852 2021-01-15T16:49:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17852 '!=' null 2021-01-15T16:49:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:49:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:49:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:49:25.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:49:25.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:49:25.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:49:25.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:49:25.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:49:25.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:49:25.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:49:25.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:49:25.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:49:25.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:49:25.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:49:25.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:49:26.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:49:26.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:49:26.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:49:26.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-01-15T16:49:26.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:49:26.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:49:26.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:49:26.676 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:49:26.677 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:49:26.677 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:49:26.677 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:49:26.677 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:49:26.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:49:26.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:26.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:49:26.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2021-01-15T16:49:26.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2021-01-15T16:49:26.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2021-01-15T16:49:26.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:26.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:49:26.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672976 2021-01-15T16:49:26.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672976 2021-01-15T16:49:26.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2021-01-15T16:49:26.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:26.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:49:27.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509454 2021-01-15T16:49:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509454 2021-01-15T16:49:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2021-01-15T16:49:27.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:27.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:49:27.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345933 2021-01-15T16:49:27.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345933 2021-01-15T16:49:27.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933' 2021-01-15T16:49:27.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:27.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:49:27.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182411 2021-01-15T16:49:27.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182411 2021-01-15T16:49:27.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933 4-107374182411' 2021-01-15T16:49:27.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:49:27.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:49:27.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018889 2021-01-15T16:49:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018889 2021-01-15T16:49:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933 4-107374182411 5-128849018889' 2021-01-15T16:49:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:27.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2021-01-15T16:49:27.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:27.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:49:27.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2021-01-15T16:49:27.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:27.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2021-01-15T16:49:27.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-01-15T16:49:27.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:49:27.492 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836498 2021-01-15T16:49:27.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836498 2021-01-15T16:49:27.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:27.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672976 2021-01-15T16:49:27.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:27.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:49:27.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2021-01-15T16:49:27.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:27.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672976 2021-01-15T16:49:27.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-01-15T16:49:27.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:49:27.888 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672976 2021-01-15T16:49:28.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672976 -lt 42949672976 2021-01-15T16:49:28.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:28.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509454 2021-01-15T16:49:28.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:28.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:49:28.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509454 2021-01-15T16:49:28.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:28.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509454 2021-01-15T16:49:28.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-01-15T16:49:28.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:49:28.281 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509454 2021-01-15T16:49:28.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509454 -lt 64424509454 2021-01-15T16:49:28.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345933 2021-01-15T16:49:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:28.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:49:28.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345933 2021-01-15T16:49:28.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:28.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345933 2021-01-15T16:49:28.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2021-01-15T16:49:28.680 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345933 2021-01-15T16:49:28.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:49:29.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345933 -lt 85899345933 2021-01-15T16:49:29.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:29.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182411 2021-01-15T16:49:29.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:29.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:49:29.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182411 2021-01-15T16:49:29.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:29.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182411 2021-01-15T16:49:29.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182411' 2021-01-15T16:49:29.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:49:29.079 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182411 2021-01-15T16:49:29.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182411 -lt 107374182411 2021-01-15T16:49:29.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:49:29.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018889 2021-01-15T16:49:29.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:49:29.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:49:29.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018889 2021-01-15T16:49:29.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:49:29.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018889 2021-01-15T16:49:29.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2021-01-15T16:49:29.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:49:29.506 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018889 2021-01-15T16:49:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018890 -lt 128849018889 2021-01-15T16:49:29.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:49:29.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-01-15T16:49:30.283 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:49:30.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-01-15T16:49:30.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-01-15T16:49:30.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:49:30.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:49:30.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:49:30.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:49:30.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:49:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:49:30.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-01-15T16:49:30.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:49:30.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-01-15T16:49:30.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:49:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2021-01-15T16:49:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:49:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:49:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:49:31.323 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:49:31.323 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_log_t": { 2021-01-15T16:49:31.324 INFO:tasks.workunit.client.0.smithi145.stdout: "head": "33'40", 2021-01-15T16:49:31.324 INFO:tasks.workunit.client.0.smithi145.stdout: "tail": "0'0", 2021-01-15T16:49:31.324 INFO:tasks.workunit.client.0.smithi145.stdout: "log": [ 2021-01-15T16:49:31.324 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.325 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.325 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c00128d9:::obj-1:head", 2021-01-15T16:49:31.325 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "32'1", 2021-01-15T16:49:31.325 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.325 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4187.0:1", 2021-01-15T16:49:31.326 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.326 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:03.954894+0000", 2021-01-15T16:49:31.326 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.326 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.327 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.327 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.327 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.327 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.327 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.328 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.328 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.328 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.328 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.328 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.329 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.329 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.329 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.329 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.329 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.330 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.330 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d7942fc0:::obj-2:head", 2021-01-15T16:49:31.330 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'2", 2021-01-15T16:49:31.330 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.330 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4190.0:1", 2021-01-15T16:49:31.331 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.331 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:04.322595+0000", 2021-01-15T16:49:31.331 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.331 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.331 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.332 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.332 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.332 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.332 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.332 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.333 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.333 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.333 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.333 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.334 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.334 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.334 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.335 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.335 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1dda57ab:::obj-3:head", 2021-01-15T16:49:31.335 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'3", 2021-01-15T16:49:31.335 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.335 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4193.0:1", 2021-01-15T16:49:31.336 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.336 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:04.527924+0000", 2021-01-15T16:49:31.336 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.336 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.336 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.337 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.337 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.337 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.337 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.337 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.338 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.338 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.338 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.338 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.338 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.339 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.339 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.339 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.339 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.339 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.340 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ba1802d0:::obj-4:head", 2021-01-15T16:49:31.340 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'4", 2021-01-15T16:49:31.340 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.340 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4196.0:1", 2021-01-15T16:49:31.340 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.341 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:04.783331+0000", 2021-01-15T16:49:31.341 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.341 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.341 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.341 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.342 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.342 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.342 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.342 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.342 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.343 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.343 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.343 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.343 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.343 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.344 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.344 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.344 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.344 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.344 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d5d4a582:::obj-5:head", 2021-01-15T16:49:31.345 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'5", 2021-01-15T16:49:31.345 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.345 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4199.0:1", 2021-01-15T16:49:31.345 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.345 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:04.926309+0000", 2021-01-15T16:49:31.346 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.346 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.346 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.346 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.346 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.347 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.347 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.347 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.348 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.348 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.348 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.348 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.348 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.349 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.349 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.349 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.349 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.349 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:25cac09a:::obj-6:head", 2021-01-15T16:49:31.350 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'6", 2021-01-15T16:49:31.350 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.350 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4202.0:1", 2021-01-15T16:49:31.350 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.350 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.093213+0000", 2021-01-15T16:49:31.351 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.351 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.351 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.351 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.352 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.352 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.352 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.354 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.354 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.354 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.354 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:b11b7838:::obj-7:head", 2021-01-15T16:49:31.355 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'7", 2021-01-15T16:49:31.355 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.355 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4205.0:1", 2021-01-15T16:49:31.355 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.355 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.270185+0000", 2021-01-15T16:49:31.356 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.356 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.356 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.356 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.356 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.357 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.357 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.357 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.357 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.357 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.358 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.358 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.358 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.358 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.358 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.359 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.359 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.359 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.359 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a0f702de:::obj-8:head", 2021-01-15T16:49:31.359 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'8", 2021-01-15T16:49:31.360 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.360 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4208.0:1", 2021-01-15T16:49:31.360 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.360 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.411464+0000", 2021-01-15T16:49:31.360 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.361 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.361 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.361 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.361 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.361 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.362 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.362 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.362 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.362 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.362 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.363 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.363 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.363 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.363 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.363 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.364 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.364 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1e5c3677:::obj-9:head", 2021-01-15T16:49:31.364 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'9", 2021-01-15T16:49:31.364 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.365 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4211.0:1", 2021-01-15T16:49:31.365 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.365 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.569804+0000", 2021-01-15T16:49:31.365 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.365 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.366 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.366 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.366 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.366 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.366 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.367 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.367 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.367 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.367 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.367 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.368 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.368 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.368 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.368 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.368 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.369 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.369 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:86dc045c:::obj-10:head", 2021-01-15T16:49:31.369 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'10", 2021-01-15T16:49:31.369 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.369 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4214.0:1", 2021-01-15T16:49:31.370 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.370 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.722388+0000", 2021-01-15T16:49:31.370 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.370 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.370 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.371 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.371 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.371 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.371 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.371 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.372 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.372 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.372 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.372 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.372 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.373 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.373 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.373 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.373 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.374 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.374 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1e6eca61:::obj-11:head", 2021-01-15T16:49:31.374 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'11", 2021-01-15T16:49:31.374 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.374 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4217.0:1", 2021-01-15T16:49:31.375 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.375 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:05.862854+0000", 2021-01-15T16:49:31.375 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.375 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.375 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.376 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.376 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.376 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.376 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.376 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.377 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.377 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.377 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.377 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.377 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.378 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.378 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.378 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.378 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.378 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.379 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:18dee445:::obj-12:head", 2021-01-15T16:49:31.379 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'12", 2021-01-15T16:49:31.379 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.379 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4220.0:1", 2021-01-15T16:49:31.379 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.380 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.038942+0000", 2021-01-15T16:49:31.380 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.380 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.380 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.380 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.381 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.381 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.381 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.381 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.381 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.382 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.382 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.382 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.382 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.383 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.383 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.383 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.384 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:283c5c90:::obj-13:head", 2021-01-15T16:49:31.384 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'13", 2021-01-15T16:49:31.384 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.384 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4223.0:1", 2021-01-15T16:49:31.384 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.385 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.181100+0000", 2021-01-15T16:49:31.385 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.385 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.385 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.385 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.386 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.386 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.386 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.386 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.386 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.387 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.387 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.387 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.387 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.387 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.388 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.388 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.388 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.388 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.388 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cc896bf0:::obj-14:head", 2021-01-15T16:49:31.389 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'14", 2021-01-15T16:49:31.389 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.389 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4226.0:1", 2021-01-15T16:49:31.389 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.389 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.332944+0000", 2021-01-15T16:49:31.390 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.390 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.390 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.390 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.390 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.391 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.391 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.391 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.391 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.391 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.392 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.392 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.392 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.392 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.392 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.393 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.393 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.393 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.393 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a581f807:::obj-15:head", 2021-01-15T16:49:31.393 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'15", 2021-01-15T16:49:31.394 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.394 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4229.0:1", 2021-01-15T16:49:31.394 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.394 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.483283+0000", 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.395 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.396 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.396 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.396 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.396 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.396 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.397 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.397 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.397 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.397 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.397 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.398 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.398 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.398 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c902ca7c:::obj-16:head", 2021-01-15T16:49:31.398 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'16", 2021-01-15T16:49:31.399 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.399 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4232.0:1", 2021-01-15T16:49:31.399 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.399 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.660039+0000", 2021-01-15T16:49:31.399 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.400 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.400 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.400 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.400 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.400 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.401 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.401 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.401 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.401 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.401 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.402 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.402 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.402 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.402 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.402 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.403 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.403 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.403 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:655283c8:::obj-17:head", 2021-01-15T16:49:31.403 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'17", 2021-01-15T16:49:31.403 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.404 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4235.0:1", 2021-01-15T16:49:31.404 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.404 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.811183+0000", 2021-01-15T16:49:31.404 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.404 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.405 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.405 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.405 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.405 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.405 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.406 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.406 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.406 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.406 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.406 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.407 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.407 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.407 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.407 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.407 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.408 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.408 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d759d762:::obj-18:head", 2021-01-15T16:49:31.408 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'18", 2021-01-15T16:49:31.408 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.408 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4238.0:1", 2021-01-15T16:49:31.409 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.409 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:06.978873+0000", 2021-01-15T16:49:31.409 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.409 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.409 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.410 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.410 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.410 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.410 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.410 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.411 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.411 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.411 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.411 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.412 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.413 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1b934a2d:::obj-19:head", 2021-01-15T16:49:31.413 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'19", 2021-01-15T16:49:31.413 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.413 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4241.0:1", 2021-01-15T16:49:31.413 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.414 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.113797+0000", 2021-01-15T16:49:31.414 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.414 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.414 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.415 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.416 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.416 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.416 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.416 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.416 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.417 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.417 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.417 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.417 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.417 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2021-01-15T16:49:31.418 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'20", 2021-01-15T16:49:31.418 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.418 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4244.0:1", 2021-01-15T16:49:31.418 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.418 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.322001+0000", 2021-01-15T16:49:31.419 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.419 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.419 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.419 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.419 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.420 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.420 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.420 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.420 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.420 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.421 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.421 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.421 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.421 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.421 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.422 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.422 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.422 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.422 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ab396690:::obj-21:head", 2021-01-15T16:49:31.422 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'21", 2021-01-15T16:49:31.423 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.423 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4247.0:1", 2021-01-15T16:49:31.423 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.423 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.472250+0000", 2021-01-15T16:49:31.423 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.424 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.424 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.424 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.424 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.424 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.425 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.425 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.425 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.425 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.426 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.426 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.426 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.426 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.426 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.427 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.427 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.427 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.427 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:254983db:::obj-22:head", 2021-01-15T16:49:31.427 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'22", 2021-01-15T16:49:31.428 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.428 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4250.0:1", 2021-01-15T16:49:31.428 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.428 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.615595+0000", 2021-01-15T16:49:31.428 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.429 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.429 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.429 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.429 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.429 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.430 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.430 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.430 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.430 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.430 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.431 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.431 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.431 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.431 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.431 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.432 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.432 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.432 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e656299f:::obj-23:head", 2021-01-15T16:49:31.432 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'23", 2021-01-15T16:49:31.432 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.433 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4253.0:1", 2021-01-15T16:49:31.433 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.433 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.756502+0000", 2021-01-15T16:49:31.433 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.433 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.434 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.434 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.434 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.434 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.434 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.435 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.435 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.435 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.435 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.435 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.436 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.436 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.436 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.436 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.436 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.437 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.437 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5cdf412b:::obj-24:head", 2021-01-15T16:49:31.437 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'24", 2021-01-15T16:49:31.437 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.437 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4256.0:1", 2021-01-15T16:49:31.438 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.438 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:07.967035+0000", 2021-01-15T16:49:31.438 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.438 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.438 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.439 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.439 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.439 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.439 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.439 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.440 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.440 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.440 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.440 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.440 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.441 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.441 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.441 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.441 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.441 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.442 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cacae143:::obj-25:head", 2021-01-15T16:49:31.442 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'25", 2021-01-15T16:49:31.442 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.442 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4259.0:1", 2021-01-15T16:49:31.442 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.443 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.120234+0000", 2021-01-15T16:49:31.443 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.443 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.443 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.443 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.444 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.444 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.444 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.444 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.444 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.445 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.445 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.445 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.445 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.445 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.446 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.446 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.446 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.446 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.446 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:34b5bff6:::obj-26:head", 2021-01-15T16:49:31.447 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'26", 2021-01-15T16:49:31.447 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.447 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4262.0:1", 2021-01-15T16:49:31.447 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.447 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.290957+0000", 2021-01-15T16:49:31.448 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.448 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.448 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.448 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.448 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.449 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.449 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.449 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.449 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.449 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.450 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.450 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.450 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.450 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.450 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.451 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.451 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.451 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.451 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a354f8f6:::obj-27:head", 2021-01-15T16:49:31.451 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'27", 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4265.0:1", 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.471151+0000", 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.452 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.453 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.453 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.453 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.453 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.453 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.454 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.454 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.454 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.454 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.454 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.455 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.455 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.455 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.455 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.455 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.456 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.456 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7421db49:::obj-28:head", 2021-01-15T16:49:31.456 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'28", 2021-01-15T16:49:31.456 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.456 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4268.0:1", 2021-01-15T16:49:31.457 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.457 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.621099+0000", 2021-01-15T16:49:31.457 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.457 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.457 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.458 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.458 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.458 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.458 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.458 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.459 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.459 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.459 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.459 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.459 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.460 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.460 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.460 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.460 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.460 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.461 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:54bc79ba:::obj-29:head", 2021-01-15T16:49:31.461 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'29", 2021-01-15T16:49:31.461 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.461 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4271.0:1", 2021-01-15T16:49:31.461 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.462 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.763122+0000", 2021-01-15T16:49:31.462 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.462 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.462 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.462 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.463 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.463 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.463 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.463 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.463 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.464 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.464 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.464 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.464 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.464 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.465 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.465 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.465 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.465 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.465 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:01b15595:::obj-30:head", 2021-01-15T16:49:31.466 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'30", 2021-01-15T16:49:31.466 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.466 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4274.0:1", 2021-01-15T16:49:31.466 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.466 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:08.906629+0000", 2021-01-15T16:49:31.467 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.467 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.467 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.467 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.467 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.468 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.468 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.468 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.468 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.468 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.469 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.469 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.469 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.470 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.470 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.470 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2021-01-15T16:49:31.470 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'31", 2021-01-15T16:49:31.471 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.471 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4277.0:1", 2021-01-15T16:49:31.471 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.471 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.047926+0000", 2021-01-15T16:49:31.471 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.472 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.472 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.472 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.472 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.472 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.473 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.473 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.473 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.473 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.473 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.474 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.474 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.474 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.474 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.474 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.475 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.475 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.475 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1bf2562f:::obj-32:head", 2021-01-15T16:49:31.475 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'32", 2021-01-15T16:49:31.475 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.476 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4280.0:1", 2021-01-15T16:49:31.476 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.476 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.199150+0000", 2021-01-15T16:49:31.476 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.476 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.477 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.477 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.477 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.477 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.477 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.478 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.478 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.478 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.478 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.478 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.479 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.479 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.479 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.479 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.479 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.480 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.480 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:b982ae22:::obj-33:head", 2021-01-15T16:49:31.480 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'33", 2021-01-15T16:49:31.480 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.480 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4283.0:1", 2021-01-15T16:49:31.481 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.481 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.349844+0000", 2021-01-15T16:49:31.481 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.481 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.481 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.482 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.482 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.482 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.482 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.482 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.483 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.483 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.483 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.483 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.483 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.484 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.484 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.484 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.485 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1ad9f98e:::obj-34:head", 2021-01-15T16:49:31.485 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'34", 2021-01-15T16:49:31.485 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.485 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4286.0:1", 2021-01-15T16:49:31.485 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.486 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.485146+0000", 2021-01-15T16:49:31.486 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.486 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.486 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.486 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.487 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.488 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.488 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.488 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.488 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.488 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.489 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.489 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.489 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.489 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.489 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:941f43ff:::obj-35:head", 2021-01-15T16:49:31.490 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'35", 2021-01-15T16:49:31.490 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.490 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4289.0:1", 2021-01-15T16:49:31.490 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.490 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.644553+0000", 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.491 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.492 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.492 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.492 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.492 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.492 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.493 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.493 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.493 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.493 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.493 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.494 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.494 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.494 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:24ebabe9:::obj-36:head", 2021-01-15T16:49:31.494 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'36", 2021-01-15T16:49:31.494 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.495 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4292.0:1", 2021-01-15T16:49:31.495 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.495 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.820065+0000", 2021-01-15T16:49:31.495 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.495 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.496 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.496 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.496 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.496 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.496 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.497 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.497 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.497 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.497 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.497 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.498 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.498 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.498 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.498 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.498 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.499 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.499 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5f3a5398:::obj-37:head", 2021-01-15T16:49:31.499 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'37", 2021-01-15T16:49:31.499 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.592 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4295.0:1", 2021-01-15T16:49:31.592 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.593 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:09.970802+0000", 2021-01-15T16:49:31.593 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.593 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.593 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.594 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.594 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.594 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.594 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.595 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.595 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.595 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.595 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.596 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.596 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.596 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.596 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.596 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.597 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.597 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c96cb37e:::obj-38:head", 2021-01-15T16:49:31.597 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'38", 2021-01-15T16:49:31.597 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.597 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4298.0:1", 2021-01-15T16:49:31.598 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.598 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:10.105451+0000", 2021-01-15T16:49:31.598 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.598 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.598 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.599 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.599 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.599 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.600 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.600 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.600 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.600 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.601 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.601 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.601 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.601 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.602 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.602 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.602 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.602 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.602 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:69e36996:::obj-39:head", 2021-01-15T16:49:31.603 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'39", 2021-01-15T16:49:31.603 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.603 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4301.0:1", 2021-01-15T16:49:31.603 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.603 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:10.272161+0000", 2021-01-15T16:49:31.604 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.604 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.604 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.604 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.604 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.605 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.605 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.605 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.605 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.606 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.606 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.606 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.607 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:49:31.607 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:49:31.607 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:9f75773e:::obj-40:head", 2021-01-15T16:49:31.608 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'40", 2021-01-15T16:49:31.608 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:49:31.608 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4304.0:1", 2021-01-15T16:49:31.608 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:49:31.608 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:49:10.464983+0000", 2021-01-15T16:49:31.609 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:49:31.609 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:49:31.609 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:49:31.609 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:49:31.609 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:49:31.610 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:49:31.610 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.610 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.610 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:49:31.610 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:49:31.611 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:49:31.611 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:49:31.611 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:49:31.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.612 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.612 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:49:31.612 INFO:tasks.workunit.client.0.smithi145.stdout: "dups": [] 2021-01-15T16:49:31.612 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:49:31.612 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_missing_t": { 2021-01-15T16:49:31.613 INFO:tasks.workunit.client.0.smithi145.stdout: "missing": [], 2021-01-15T16:49:31.613 INFO:tasks.workunit.client.0.smithi145.stdout: "may_include_deletes": true 2021-01-15T16:49:31.613 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:49:31.613 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:49:31.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:49:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=40 2021-01-15T16:49:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 40 '!=' 40 ']' 2021-01-15T16:49:31.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:49:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=0 2021-01-15T16:49:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:49:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-01-15T16:49:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-01-15T16:49:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:49:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-01-15T16:49:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:49:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:49:31.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:49:31.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:49:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:49:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:49:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:49:31.619 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after v 0'0 2021-01-15T16:49:31.620 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'40 2021-01-15T16:49:31.620 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'39 2021-01-15T16:49:31.620 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'38 2021-01-15T16:49:31.620 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'37 2021-01-15T16:49:31.620 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'36 2021-01-15T16:49:31.621 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'35 2021-01-15T16:49:31.621 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'34 2021-01-15T16:49:31.621 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'33 2021-01-15T16:49:31.621 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'32 2021-01-15T16:49:31.621 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'31 2021-01-15T16:49:31.622 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'30 2021-01-15T16:49:31.622 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'29 2021-01-15T16:49:31.622 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'28 2021-01-15T16:49:31.622 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'27 2021-01-15T16:49:31.623 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'26 2021-01-15T16:49:31.623 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'25 2021-01-15T16:49:31.623 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'24 2021-01-15T16:49:31.623 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'23 2021-01-15T16:49:31.623 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'22 2021-01-15T16:49:31.624 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'21 2021-01-15T16:49:31.624 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'20 2021-01-15T16:49:31.624 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'19 2021-01-15T16:49:31.624 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'18 2021-01-15T16:49:31.624 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'17 2021-01-15T16:49:31.625 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'16 2021-01-15T16:49:31.625 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'15 2021-01-15T16:49:31.625 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'14 2021-01-15T16:49:31.625 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'13 2021-01-15T16:49:31.625 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'12 2021-01-15T16:49:31.626 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'11 2021-01-15T16:49:31.626 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'10 2021-01-15T16:49:31.626 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'9 2021-01-15T16:49:31.626 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'8 2021-01-15T16:49:31.626 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'7 2021-01-15T16:49:31.627 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'6 2021-01-15T16:49:31.627 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'5 2021-01-15T16:49:31.627 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'4 2021-01-15T16:49:31.627 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'3 2021-01-15T16:49:31.627 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 33'2 2021-01-15T16:49:31.628 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_after copy log version 32'1 2021-01-15T16:49:31.628 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:49:11.789+0000 7fb909582700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:49:31.628 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after v 0'0 2021-01-15T16:49:31.628 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'40 2021-01-15T16:49:31.628 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'39 2021-01-15T16:49:31.629 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'38 2021-01-15T16:49:31.629 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'37 2021-01-15T16:49:31.629 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'36 2021-01-15T16:49:31.629 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'35 2021-01-15T16:49:31.629 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'34 2021-01-15T16:49:31.630 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'33 2021-01-15T16:49:31.630 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'32 2021-01-15T16:49:31.630 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'31 2021-01-15T16:49:31.630 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'30 2021-01-15T16:49:31.631 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'29 2021-01-15T16:49:31.631 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'28 2021-01-15T16:49:31.631 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'27 2021-01-15T16:49:31.631 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'26 2021-01-15T16:49:31.631 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'25 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'24 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'23 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'22 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'21 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'20 2021-01-15T16:49:31.632 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'19 2021-01-15T16:49:31.633 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'18 2021-01-15T16:49:31.633 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'17 2021-01-15T16:49:31.633 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'16 2021-01-15T16:49:31.633 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'15 2021-01-15T16:49:31.634 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'14 2021-01-15T16:49:31.634 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'13 2021-01-15T16:49:31.634 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'12 2021-01-15T16:49:31.634 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'11 2021-01-15T16:49:31.634 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'10 2021-01-15T16:49:31.635 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'9 2021-01-15T16:49:31.635 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'8 2021-01-15T16:49:31.635 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'7 2021-01-15T16:49:31.635 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'6 2021-01-15T16:49:31.636 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'5 2021-01-15T16:49:31.636 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'4 2021-01-15T16:49:31.636 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'3 2021-01-15T16:49:31.636 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'2 2021-01-15T16:49:31.636 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 32'1 2021-01-15T16:49:31.637 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:49:31.637 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after v 0'0 2021-01-15T16:49:31.637 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'40 2021-01-15T16:49:31.637 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'39 2021-01-15T16:49:31.637 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'38 2021-01-15T16:49:31.638 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'37 2021-01-15T16:49:31.638 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'36 2021-01-15T16:49:31.638 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'35 2021-01-15T16:49:31.638 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'34 2021-01-15T16:49:31.638 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'33 2021-01-15T16:49:31.639 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'32 2021-01-15T16:49:31.639 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'31 2021-01-15T16:49:31.639 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'30 2021-01-15T16:49:31.639 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'29 2021-01-15T16:49:31.639 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'28 2021-01-15T16:49:31.640 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'27 2021-01-15T16:49:31.640 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'26 2021-01-15T16:49:31.640 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'25 2021-01-15T16:49:31.640 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'24 2021-01-15T16:49:31.640 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'23 2021-01-15T16:49:31.641 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'22 2021-01-15T16:49:31.641 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'21 2021-01-15T16:49:31.641 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'20 2021-01-15T16:49:31.641 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'19 2021-01-15T16:49:31.641 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'18 2021-01-15T16:49:31.642 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'17 2021-01-15T16:49:31.642 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'16 2021-01-15T16:49:31.642 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'15 2021-01-15T16:49:31.642 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'14 2021-01-15T16:49:31.642 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'13 2021-01-15T16:49:31.643 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'12 2021-01-15T16:49:31.643 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'11 2021-01-15T16:49:31.643 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'10 2021-01-15T16:49:31.643 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'9 2021-01-15T16:49:31.643 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'8 2021-01-15T16:49:31.644 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'7 2021-01-15T16:49:31.644 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'6 2021-01-15T16:49:31.644 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'5 2021-01-15T16:49:31.644 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'4 2021-01-15T16:49:31.644 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'3 2021-01-15T16:49:31.645 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 33'2 2021-01-15T16:49:31.645 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_after copy log version 32'1 2021-01-15T16:49:31.645 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.4.log:2021-01-15T16:49:12.090+0000 7fcd4ce80700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:49:31.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:49:31.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:49:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:49:31.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:49:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:49:31.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:49:31.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:49:31.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:49:31.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:49:31.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:49:31.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:49:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:49:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:49:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:49:31.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:49:31.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:31.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:31.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:49:31.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:49:31.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:49:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-01-15T16:49:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-01-15T16:49:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-01-15T16:49:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:49:31.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:49:31.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:49:31.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:49:31.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:49:31.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:49:31.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:49:31.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:49:31.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:49:31.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:49:31.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:49:31.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:49:31.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:49:31.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:49:31.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:49:31.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:49:31.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:49:31.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:49:31.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:49:31.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:49:31.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:49:31.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:49:31.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:31.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:31.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:49:31.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:49:31.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:49:31.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-01-15T16:49:31.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:49:31.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:31.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:31.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.119796 2021-01-15T16:49:31.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:49:31.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:49:31.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:49:31.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-01-15T16:49:31.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_2 td/osd-backfill-recovery-log 2021-01-15T16:49:31.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:129: TEST_recovery_2: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:131: TEST_recovery_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 151 10 141 20 2021-01-15T16:49:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=151 2021-01-15T16:49:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=10 2021-01-15T16:49:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=141 2021-01-15T16:49:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=20 2021-01-15T16:49:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-01-15T16:49:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-01-15T16:49:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:49:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:49:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:49:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-01-15T16:49:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-01-15T16:49:31.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:49:31.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:31.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:31.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:31.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:31.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:31.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:31.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:49:32.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:49:32.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:49:32.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:49:32.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:49:32.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:49:32.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:49:32.166 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:49:32.166 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:49:32.166 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:49:32.167 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:49:32.167 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:32.167 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:32.167 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:49:32.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:49:32.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get fsid 2021-01-15T16:49:32.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:49:32.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:49:32.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:49:32.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:49:32.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:49:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:49:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:49:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:49:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:49:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.119796/ceph-mon.a.asok 2021-01-15T16:49:32.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:49:32.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.119796/ceph-mon.a.asok config get mon_host 2021-01-15T16:49:32.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-01-15T16:49:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:49:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:49:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:49:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-01-15T16:49:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:49:32.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:49:32.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:32.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:32.907 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:32.907 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:32.908 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:32.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:32.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:49:32.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:49:32.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-01-15T16:49:32.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:32.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:32.933 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-01-15T16:49:32.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-01-15T16:49:32.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:49:32.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-01-15T16:49:32.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:32.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:49:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:49:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:49:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-01-15T16:49:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:49:32.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:49:32.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:49:32.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:49:32.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-01-15T16:49:32.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-01-15T16:49:32.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:49:32.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:32.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:49:32.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:49:32.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:32.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:32.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:32.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:32.942 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:32.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:32.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:32.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:49:32.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:49:32.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:49:32.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:49:32.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:49:32.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:49:32.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:49:32.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:32.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:49:32.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:49:32.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-01-15T16:49:32.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:49:32.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=382f10b1-b473-46e9-bd75-130a15a79638 2021-01-15T16:49:32.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 382f10b1-b473-46e9-bd75-130a15a79638' 2021-01-15T16:49:32.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:49:32.946 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 382f10b1-b473-46e9-bd75-130a15a79638 2021-01-15T16:49:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCcxwFg9dPBOBAAArK+QT+idL6EwaruMpz7tg== 2021-01-15T16:49:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCcxwFg9dPBOBAAArK+QT+idL6EwaruMpz7tg=="}' 2021-01-15T16:49:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 382f10b1-b473-46e9-bd75-130a15a79638 -i td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:49:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:49:33.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-01-15T16:49:33.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCcxwFg9dPBOBAAArK+QT+idL6EwaruMpz7tg== --osd-uuid 382f10b1-b473-46e9-bd75-130a15a79638 2021-01-15T16:49:33.334 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.333+0000 7f108552d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:33.335 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.335+0000 7f108552d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:33.336 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.336+0000 7f108552d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:33.337 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.336+0000 7f108552d080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:49:33.337 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.336+0000 7f108552d080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:49:33.337 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.336+0000 7f108552d080 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-01-15T16:49:33.407 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:33.407+0000 7f108552d080 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-01-15T16:49:38.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-01-15T16:49:38.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:49:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:49:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:49:38.005 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:49:38.527 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:49:38.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:49:38.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:38.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:49:38.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:49:38.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:49:38.551 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:38.551+0000 7f7d49f2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:38.577+0000 7f7d49f2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:38.579 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:38.578+0000 7f7d49f2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:38.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:49:38.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:49:38.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:49:38.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:49:38.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:49:38.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:38.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:49:38.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:38.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:49:38.945 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:49:39.095 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:39.095+0000 7f7d49f2b080 -1 Falling back to public interface 2021-01-15T16:49:39.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:49:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:39.903+0000 7f7d49f2b080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:49:40.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:49:40.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:40.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:49:40.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:40.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:49:40.335 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:49:40.780 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/160142,v1:127.0.0.1:6803/160142] [v2:127.0.0.1:6804/160142,v1:127.0.0.1:6805/160142] exists,up 382f10b1-b473-46e9-bd75-130a15a79638 2021-01-15T16:49:40.781 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 c3bee42e-f3f4-484b-85cc-8f334ea05d37 2021-01-15T16:49:40.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:49:40.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:49:40.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:49:40.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:49:40.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-01-15T16:49:40.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:40.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:49:40.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:49:40.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:49:40.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-01-15T16:49:40.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:49:40.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:49:40.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:49:40.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:49:40.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-01-15T16:49:40.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-01-15T16:49:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:49:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:49:40.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:49:40.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:40.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:40.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:40.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:40.786 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:40.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:40.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:40.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:49:40.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:49:40.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:49:40.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:49:40.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:49:40.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:49:40.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:49:40.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:40.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:49:40.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:49:40.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-01-15T16:49:40.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:49:40.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c3bee42e-f3f4-484b-85cc-8f334ea05d37 2021-01-15T16:49:40.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 c3bee42e-f3f4-484b-85cc-8f334ea05d37' 2021-01-15T16:49:40.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:49:40.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCkxwFgIrBdLxAAKJBV8Au4HcrQMk+5LDEOoQ== 2021-01-15T16:49:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCkxwFgIrBdLxAAKJBV8Au4HcrQMk+5LDEOoQ=="}' 2021-01-15T16:49:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c3bee42e-f3f4-484b-85cc-8f334ea05d37 -i td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:49:41.314 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:49:41.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-01-15T16:49:41.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCkxwFgIrBdLxAAKJBV8Au4HcrQMk+5LDEOoQ== --osd-uuid c3bee42e-f3f4-484b-85cc-8f334ea05d37 2021-01-15T16:49:41.352 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.352+0000 7f7effa18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.353+0000 7f7effa18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.354+0000 7f7effa18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.355+0000 7f7effa18080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:49:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.355+0000 7f7effa18080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:49:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.355+0000 7f7effa18080 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-01-15T16:49:41.401 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:41.401+0000 7f7effa18080 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-01-15T16:49:46.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-01-15T16:49:46.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:49:46.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:49:46.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:49:46.184 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:49:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:49:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:49:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:49:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:49:46.708 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:49:46.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:46.730+0000 7f9a671f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:46.758 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:46.757+0000 7f9a671f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:46.760 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:46.759+0000 7f9a671f4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:47.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:49:47.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:49:47.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:49:47.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:49:47.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:49:47.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:47.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:49:47.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:47.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:49:47.118 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:49:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:47.291+0000 7f9a671f4080 -1 Falling back to public interface 2021-01-15T16:49:47.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:49:48.076 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:48.075+0000 7f9a671f4080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:49:48.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:49:48.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:48.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:49:48.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:48.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:49:48.508 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:49:48.931 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/161406,v1:127.0.0.1:6811/161406] [v2:127.0.0.1:6812/161406,v1:127.0.0.1:6813/161406] exists,up c3bee42e-f3f4-484b-85cc-8f334ea05d37 2021-01-15T16:49:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:49:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:49:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:49:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:49:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-01-15T16:49:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:49:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:49:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:49:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-01-15T16:49:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:49:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:49:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:49:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:49:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-01-15T16:49:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-01-15T16:49:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:49:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:49:48.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:49:48.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:48.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:48.936 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:48.936 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:48.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:48.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:48.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:48.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:49:48.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:49:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:49:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:49:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:49:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:49:48.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:49:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:49:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:49:48.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-01-15T16:49:48.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:49:48.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=86c608ab-2d07-4916-a3a0-bb4fc05baacd 2021-01-15T16:49:48.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 86c608ab-2d07-4916-a3a0-bb4fc05baacd' 2021-01-15T16:49:48.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:49:48.941 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 86c608ab-2d07-4916-a3a0-bb4fc05baacd 2021-01-15T16:49:48.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCsxwFgb05jOBAAiYpcISVyUL1n8Osd82hKNw== 2021-01-15T16:49:48.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCsxwFgb05jOBAAiYpcISVyUL1n8Osd82hKNw=="}' 2021-01-15T16:49:48.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 86c608ab-2d07-4916-a3a0-bb4fc05baacd -i td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:49:49.460 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:49:49.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-01-15T16:49:49.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCsxwFgb05jOBAAiYpcISVyUL1n8Osd82hKNw== --osd-uuid 86c608ab-2d07-4916-a3a0-bb4fc05baacd 2021-01-15T16:49:49.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.498+0000 7f04a5214080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:49.500 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.500+0000 7f04a5214080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:49.502 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.502+0000 7f04a5214080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:49.503 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.503+0000 7f04a5214080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:49:49.503 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.503+0000 7f04a5214080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:49:49.503 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.503+0000 7f04a5214080 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-01-15T16:49:49.540 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:49.540+0000 7f04a5214080 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-01-15T16:49:54.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-01-15T16:49:54.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:49:54.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:49:54.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:49:54.294 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:49:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:49:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:49:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:49:54.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:49:54.820 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:49:54.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:54.842+0000 7fb904f64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:54.844 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:54.844+0000 7fb904f64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:54.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:54.845+0000 7fb904f64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:55.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:49:55.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:49:55.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:49:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:49:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:49:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:49:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:55.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:49:55.254 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:49:55.368 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:55.368+0000 7fb904f64080 -1 Falling back to public interface 2021-01-15T16:49:55.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:49:56.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:56.148+0000 7fb904f64080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:49:56.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:49:56.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:49:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:49:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:49:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:49:56.643 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:49:57.044 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/162670,v1:127.0.0.1:6819/162670] [v2:127.0.0.1:6820/162670,v1:127.0.0.1:6821/162670] exists,up 86c608ab-2d07-4916-a3a0-bb4fc05baacd 2021-01-15T16:49:57.045 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 4c017fba-1344-4ab9-a0b0-a77c2a570ede 2021-01-15T16:49:57.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:49:57.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:49:57.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:49:57.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:49:57.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-01-15T16:49:57.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:49:57.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:49:57.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T16:49:57.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:49:57.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-01-15T16:49:57.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:49:57.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:49:57.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:49:57.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:49:57.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-01-15T16:49:57.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-01-15T16:49:57.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:49:57.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:49:57.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:49:57.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:49:57.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:49:57.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:49:57.050 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:49:57.051 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:49:57.051 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:49:57.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:57.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:49:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:49:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:49:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:49:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:49:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:49:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:49:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:49:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:49:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:49:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:49:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-01-15T16:49:57.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:49:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4c017fba-1344-4ab9-a0b0-a77c2a570ede 2021-01-15T16:49:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 4c017fba-1344-4ab9-a0b0-a77c2a570ede' 2021-01-15T16:49:57.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:49:57.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC1xwFg51eDAxAAuDp4A9caJ4/o2SWKBPsErw== 2021-01-15T16:49:57.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC1xwFg51eDAxAAuDp4A9caJ4/o2SWKBPsErw=="}' 2021-01-15T16:49:57.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4c017fba-1344-4ab9-a0b0-a77c2a570ede -i td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:49:57.625 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T16:49:57.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-01-15T16:49:57.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC1xwFg51eDAxAAuDp4A9caJ4/o2SWKBPsErw== --osd-uuid 4c017fba-1344-4ab9-a0b0-a77c2a570ede 2021-01-15T16:49:57.663 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.663+0000 7f22e981d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:57.665 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.665+0000 7f22e981d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:57.667 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.666+0000 7f22e981d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:49:57.667 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.667+0000 7f22e981d080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:49:57.668 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.667+0000 7f22e981d080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:49:57.668 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.667+0000 7f22e981d080 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-01-15T16:49:57.747 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:49:57.747+0000 7f22e981d080 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-01-15T16:50:02.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-01-15T16:50:02.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:50:02.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T16:50:02.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:50:02.274 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T16:50:02.807 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T16:50:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T16:50:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:50:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:50:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:50:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:50:02.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:02.830+0000 7f47866cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:02.865 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:02.863+0000 7f47866cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:02.867 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:02.865+0000 7f47866cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T16:50:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:50:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T16:50:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:50:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:50:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:50:03.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:03.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:50:03.206 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:50:03.382 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:03.381+0000 7f47866cd080 -1 Falling back to public interface 2021-01-15T16:50:03.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:50:04.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:04.232+0000 7f47866cd080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T16:50:04.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:50:04.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:04.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:50:04.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:04.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T16:50:04.618 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:50:05.030 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/163934,v1:127.0.0.1:6827/163934] [v2:127.0.0.1:6828/163934,v1:127.0.0.1:6829/163934] exists,up 4c017fba-1344-4ab9-a0b0-a77c2a570ede 2021-01-15T16:50:05.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:50:05.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:50:05.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:50:05.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:50:05.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-01-15T16:50:05.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:50:05.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:50:05.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T16:50:05.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:50:05.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-01-15T16:50:05.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-01-15T16:50:05.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:50:05.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:50:05.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:50:05.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:50:05.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:50:05.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:50:05.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:50:05.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:50:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:50:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:50:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:50:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:50:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:50:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:50:05.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:50:05.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:50:05.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-01-15T16:50:05.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:50:05.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=35311f9e-0a3c-4495-8607-77d0d64cebc8 2021-01-15T16:50:05.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 35311f9e-0a3c-4495-8607-77d0d64cebc8' 2021-01-15T16:50:05.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:50:05.040 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 35311f9e-0a3c-4495-8607-77d0d64cebc8 2021-01-15T16:50:05.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC9xwFg+fq8AhAA4apTL/+jH6nzy5taR8wPBQ== 2021-01-15T16:50:05.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC9xwFg+fq8AhAA4apTL/+jH6nzy5taR8wPBQ=="}' 2021-01-15T16:50:05.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 35311f9e-0a3c-4495-8607-77d0d64cebc8 -i td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:50:05.635 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T16:50:05.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-01-15T16:50:05.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC9xwFg+fq8AhAA4apTL/+jH6nzy5taR8wPBQ== --osd-uuid 35311f9e-0a3c-4495-8607-77d0d64cebc8 2021-01-15T16:50:05.674 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.673+0000 7f8f2ba57080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:05.675 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.675+0000 7f8f2ba57080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:05.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.676+0000 7f8f2ba57080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:05.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.677+0000 7f8f2ba57080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:50:05.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.677+0000 7f8f2ba57080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:50:05.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.677+0000 7f8f2ba57080 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-01-15T16:50:05.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:05.772+0000 7f8f2ba57080 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-01-15T16:50:10.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-01-15T16:50:10.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:50:10.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T16:50:10.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:50:10.558 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T16:50:11.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T16:50:11.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:50:11.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:50:11.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:50:11.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:50:11.103 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T16:50:11.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:11.125+0000 7fd181085080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:11.127 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:11.127+0000 7fd181085080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:11.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:11.128+0000 7fd181085080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:11.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T16:50:11.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:50:11.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T16:50:11.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:50:11.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:50:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:50:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:50:11.496 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:50:11.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:11.642+0000 7fd181085080 -1 Falling back to public interface 2021-01-15T16:50:11.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:50:12.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:12.525+0000 7fd181085080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T16:50:12.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:50:12.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:12.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:50:12.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:12.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:50:12.889 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:50:13.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:50:14.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:50:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T16:50:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T16:50:14.302 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:50:14.696 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/165198,v1:127.0.0.1:6835/165198] [v2:127.0.0.1:6836/165198,v1:127.0.0.1:6837/165198] exists,up 35311f9e-0a3c-4495-8607-77d0d64cebc8 2021-01-15T16:50:14.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:50:14.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:50:14.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:50:14.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:50:14.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-01-15T16:50:14.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-recovery-log 2021-01-15T16:50:14.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:50:14.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T16:50:14.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:50:14.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-01-15T16:50:14.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-01-15T16:50:14.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:50:14.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:50:14.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:50:14.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-01-15T16:50:14.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-01-15T16:50:14.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:50:14.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2021-01-15T16:50:14.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-01-15T16:50:14.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:50:14.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:50:14.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:50:14.701 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:50:14.701 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:50:14.701 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:50:14.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:50:14.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' 2021-01-15T16:50:14.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:50:14.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:50:14.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:50:14.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-01-15T16:50:14.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-01-15T16:50:14.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:50:14.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:50:14.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:50:14.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:50:14.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:50:14.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-01-15T16:50:14.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:50:14.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4fa077fb-ceec-4f27-98d1-a026d7203cef 2021-01-15T16:50:14.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 4fa077fb-ceec-4f27-98d1-a026d7203cef' 2021-01-15T16:50:14.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:50:14.706 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 4fa077fb-ceec-4f27-98d1-a026d7203cef 2021-01-15T16:50:14.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDGxwFgopVfKhAAixqHO3BKDdMKVBC6Q4JX8g== 2021-01-15T16:50:14.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDGxwFgopVfKhAAixqHO3BKDdMKVBC6Q4JX8g=="}' 2021-01-15T16:50:14.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4fa077fb-ceec-4f27-98d1-a026d7203cef -i td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:50:15.281 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T16:50:15.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-01-15T16:50:15.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGxwFgopVfKhAAixqHO3BKDdMKVBC6Q4JX8g== --osd-uuid 4fa077fb-ceec-4f27-98d1-a026d7203cef 2021-01-15T16:50:15.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.318+0000 7f5cb33d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:15.320 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.320+0000 7f5cb33d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:15.322 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.322+0000 7f5cb33d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.323+0000 7f5cb33d3080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:50:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.323+0000 7f5cb33d3080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:50:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.323+0000 7f5cb33d3080 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-01-15T16:50:15.409 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:15.408+0000 7f5cb33d3080 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-01-15T16:50:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-01-15T16:50:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:50:20.175 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T16:50:20.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T16:50:20.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:50:20.702 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T16:50:20.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T16:50:20.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=a61f6efd-b39a-43f9-a81a-b7f210831141 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.119796/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:50:20.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:50:20.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:50:20.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:50:20.726 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:20.726+0000 7f64a04ab080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:20.771+0000 7f64a04ab080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:20.772+0000 7f64a04ab080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:50:21.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T16:50:21.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:50:21.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T16:50:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:50:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:50:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:50:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:21.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:50:21.103 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:50:21.293 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:21.292+0000 7f64a04ab080 -1 Falling back to public interface 2021-01-15T16:50:21.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:50:22.129 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:50:22.128+0000 7f64a04ab080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T16:50:22.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:50:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:50:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:50:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:50:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T16:50:22.496 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:50:22.970 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/166488,v1:127.0.0.1:6843/166488] [v2:127.0.0.1:6844/166488,v1:127.0.0.1:6845/166488] exists,up 4fa077fb-ceec-4f27-98d1-a026d7203cef 2021-01-15T16:50:22.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:50:22.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:50:22.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:50:22.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-01-15T16:50:22.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:50:23.493 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:50:23.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:50:24.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 141 2021-01-15T16:50:24.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:24.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-01-15T16:50:25.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:25.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-01-15T16:50:25.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:25.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-01-15T16:50:25.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:25.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-01-15T16:50:25.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:25.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-01-15T16:50:25.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:25.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-01-15T16:50:26.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-01-15T16:50:26.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-01-15T16:50:26.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-01-15T16:50:26.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-01-15T16:50:26.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-01-15T16:50:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-01-15T16:50:26.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:26.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-01-15T16:50:27.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-01-15T16:50:27.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-01-15T16:50:27.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-01-15T16:50:27.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-01-15T16:50:27.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-01-15T16:50:27.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-01-15T16:50:27.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:27.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-01-15T16:50:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-01-15T16:50:28.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-01-15T16:50:28.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-01-15T16:50:28.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-01-15T16:50:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-01-15T16:50:28.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:28.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-01-15T16:50:29.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-01-15T16:50:29.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-01-15T16:50:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-01-15T16:50:29.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-01-15T16:50:29.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-01-15T16:50:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-01-15T16:50:29.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:29.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-01-15T16:50:30.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-01-15T16:50:30.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-01-15T16:50:30.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-01-15T16:50:30.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-01-15T16:50:30.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-01-15T16:50:30.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:30.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-01-15T16:50:31.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-01-15T16:50:31.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2021-01-15T16:50:31.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2021-01-15T16:50:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2021-01-15T16:50:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2021-01-15T16:50:31.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:31.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2021-01-15T16:50:32.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2021-01-15T16:50:32.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2021-01-15T16:50:32.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2021-01-15T16:50:32.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2021-01-15T16:50:32.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2021-01-15T16:50:32.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2021-01-15T16:50:32.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:32.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2021-01-15T16:50:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2021-01-15T16:50:33.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2021-01-15T16:50:33.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2021-01-15T16:50:33.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2021-01-15T16:50:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2021-01-15T16:50:33.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:33.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2021-01-15T16:50:34.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2021-01-15T16:50:34.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2021-01-15T16:50:34.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2021-01-15T16:50:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2021-01-15T16:50:34.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2021-01-15T16:50:34.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:34.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2021-01-15T16:50:35.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2021-01-15T16:50:35.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2021-01-15T16:50:35.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2021-01-15T16:50:35.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2021-01-15T16:50:35.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2021-01-15T16:50:35.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2021-01-15T16:50:35.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:35.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2021-01-15T16:50:36.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2021-01-15T16:50:36.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2021-01-15T16:50:36.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2021-01-15T16:50:36.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2021-01-15T16:50:36.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2021-01-15T16:50:36.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:36.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2021-01-15T16:50:37.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2021-01-15T16:50:37.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2021-01-15T16:50:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2021-01-15T16:50:37.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2021-01-15T16:50:37.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2021-01-15T16:50:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2021-01-15T16:50:38.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2021-01-15T16:50:38.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2021-01-15T16:50:38.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2021-01-15T16:50:38.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2021-01-15T16:50:38.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2021-01-15T16:50:38.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2021-01-15T16:50:38.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:38.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2021-01-15T16:50:39.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2021-01-15T16:50:39.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2021-01-15T16:50:39.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2021-01-15T16:50:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2021-01-15T16:50:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2021-01-15T16:50:39.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:39.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2021-01-15T16:50:40.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2021-01-15T16:50:40.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2021-01-15T16:50:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2021-01-15T16:50:40.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2021-01-15T16:50:40.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2021-01-15T16:50:40.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2021-01-15T16:50:40.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:40.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2021-01-15T16:50:41.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2021-01-15T16:50:41.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2021-01-15T16:50:41.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2021-01-15T16:50:41.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2021-01-15T16:50:41.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2021-01-15T16:50:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2021-01-15T16:50:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2021-01-15T16:50:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2021-01-15T16:50:42.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2021-01-15T16:50:42.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2021-01-15T16:50:42.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2021-01-15T16:50:42.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2021-01-15T16:50:42.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:42.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2021-01-15T16:50:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2021-01-15T16:50:43.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2021-01-15T16:50:43.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2021-01-15T16:50:43.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2021-01-15T16:50:43.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2021-01-15T16:50:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2021-01-15T16:50:44.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2021-01-15T16:50:44.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2021-01-15T16:50:44.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2021-01-15T16:50:44.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2021-01-15T16:50:44.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2021-01-15T16:50:44.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2021-01-15T16:50:44.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:44.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2021-01-15T16:50:45.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2021-01-15T16:50:45.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2021-01-15T16:50:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2021-01-15T16:50:45.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2021-01-15T16:50:45.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2021-01-15T16:50:45.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:45.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2021-01-15T16:50:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2021-01-15T16:50:46.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2021-01-15T16:50:46.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2021-01-15T16:50:46.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2021-01-15T16:50:46.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2021-01-15T16:50:46.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-01-15T16:50:46.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2021-01-15T16:50:46.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:50:46.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-01-15T16:50:47.339 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:50:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-01-15T16:50:47.928 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-01-15T16:50:47.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 20 '!=' 0 ']' 2021-01-15T16:50:47.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: seq 1 20 2021-01-15T16:50:47.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:47.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-1 /etc/passwd 2021-01-15T16:50:49.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:49.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-2 /etc/passwd 2021-01-15T16:50:49.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:49.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-3 /etc/passwd 2021-01-15T16:50:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-4 /etc/passwd 2021-01-15T16:50:50.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:50.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-5 /etc/passwd 2021-01-15T16:50:50.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:50.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-6 /etc/passwd 2021-01-15T16:50:50.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:50.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-7 /etc/passwd 2021-01-15T16:50:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-8 /etc/passwd 2021-01-15T16:50:51.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:51.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-9 /etc/passwd 2021-01-15T16:50:51.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:51.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-10 /etc/passwd 2021-01-15T16:50:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-11 /etc/passwd 2021-01-15T16:50:51.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:51.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-12 /etc/passwd 2021-01-15T16:50:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-13 /etc/passwd 2021-01-15T16:50:52.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:52.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-14 /etc/passwd 2021-01-15T16:50:52.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:52.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-15 /etc/passwd 2021-01-15T16:50:53.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:53.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-16 /etc/passwd 2021-01-15T16:50:53.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:53.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-17 /etc/passwd 2021-01-15T16:50:53.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:53.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-18 /etc/passwd 2021-01-15T16:50:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-19 /etc/passwd 2021-01-15T16:50:53.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2021-01-15T16:50:53.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-20 /etc/passwd 2021-01-15T16:50:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-01-15T16:50:55.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-01-15T16:50:55.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:50:55.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:50:55.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:50:55.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:50:55.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:50:55.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:50:55.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:50:55.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:50:55.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:50:55.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:50:55.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:50:55.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:50:55.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:50:55.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:50:55.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:50:55.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:50:55.685 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:50:55.686 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:50:55.686 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:50:55.686 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:50:55.686 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:50:55.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:50:55.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:55.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:50:55.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836497 2021-01-15T16:50:55.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836497 2021-01-15T16:50:55.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497' 2021-01-15T16:50:55.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:55.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:50:55.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672977 2021-01-15T16:50:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672977 2021-01-15T16:50:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672977' 2021-01-15T16:50:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:55.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:50:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509454 2021-01-15T16:50:56.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509454 2021-01-15T16:50:56.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672977 2-64424509454' 2021-01-15T16:50:56.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:56.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:50:56.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345933 2021-01-15T16:50:56.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345933 2021-01-15T16:50:56.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672977 2-64424509454 3-85899345933' 2021-01-15T16:50:56.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:56.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:50:56.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182412 2021-01-15T16:50:56.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182412 2021-01-15T16:50:56.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672977 2-64424509454 3-85899345933 4-107374182412' 2021-01-15T16:50:56.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:50:56.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:50:56.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018889 2021-01-15T16:50:56.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018889 2021-01-15T16:50:56.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672977 2-64424509454 3-85899345933 4-107374182412 5-128849018889' 2021-01-15T16:50:56.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:56.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836497 2021-01-15T16:50:56.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:56.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:50:56.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2021-01-15T16:50:56.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:56.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836497 2021-01-15T16:50:56.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-01-15T16:50:56.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:50:56.532 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836497 2021-01-15T16:50:56.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836497 2021-01-15T16:50:56.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:56.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672977 2021-01-15T16:50:56.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:56.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:50:56.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672977 2021-01-15T16:50:56.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:56.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672977 2021-01-15T16:50:56.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2021-01-15T16:50:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:50:56.927 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672977 2021-01-15T16:50:57.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672977 -lt 42949672977 2021-01-15T16:50:57.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:57.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509454 2021-01-15T16:50:57.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:57.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:50:57.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509454 2021-01-15T16:50:57.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:57.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509454 2021-01-15T16:50:57.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-01-15T16:50:57.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:50:57.317 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509454 2021-01-15T16:50:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509454 -lt 64424509454 2021-01-15T16:50:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:57.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345933 2021-01-15T16:50:57.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:57.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:50:57.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345933 2021-01-15T16:50:57.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:57.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345933 2021-01-15T16:50:57.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2021-01-15T16:50:57.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:50:57.711 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345933 2021-01-15T16:50:58.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345933 -lt 85899345933 2021-01-15T16:50:58.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:58.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182412 2021-01-15T16:50:58.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:50:58.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182412 2021-01-15T16:50:58.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:58.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182412 2021-01-15T16:50:58.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182412' 2021-01-15T16:50:58.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:50:58.116 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182412 2021-01-15T16:50:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182412 -lt 107374182412 2021-01-15T16:50:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:50:58.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018889 2021-01-15T16:50:58.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:50:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:50:58.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018889 2021-01-15T16:50:58.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:50:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018889 2021-01-15T16:50:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2021-01-15T16:50:58.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:50:58.508 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018889 2021-01-15T16:50:58.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018890 -lt 128849018889 2021-01-15T16:50:58.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:50:58.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:50:58.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:50:59.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:50:59.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:50:59.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:50:59.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:50:59.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:50:59.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:50:59.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:50:59.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:50:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:50:59.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:50:59.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:50:59.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:00.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:00.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:51:00.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:51:00.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:51:00.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:00.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:00.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:00.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:00.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:00.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:00.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:00.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:01.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:01.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:01.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:01.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:01.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:01.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:01.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:01.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:01.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:01.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19528 2021-01-15T16:51:01.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19528 '!=' null 2021-01-15T16:51:01.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:01.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:01.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:01.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:01.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:02.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:02.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:02.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:02.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:02.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:02.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:02.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:02.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:02.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:02.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:02.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:02.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:02.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:03.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=36058 2021-01-15T16:51:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 36058 '!=' null 2021-01-15T16:51:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:03.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:03.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:03.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:03.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:03.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:03.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:03.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:03.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:03.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:03.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:03.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:03.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:04.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:04.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:04.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:04.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:04.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:04.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:04.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:04.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:04.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:04.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31204 2021-01-15T16:51:04.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31204 '!=' null 2021-01-15T16:51:04.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:04.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:04.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:05.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:05.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:05.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:05.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:05.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:05.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:05.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:05.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:05.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:05.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:05.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:05.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:05.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:05.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:06.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31204 2021-01-15T16:51:06.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31204 '!=' null 2021-01-15T16:51:06.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:06.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:06.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:06.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:06.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:06.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:06.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:06.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:06.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:06.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:06.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:06.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:06.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:06.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:06.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:07.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:07.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:07.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:07.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:07.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:07.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:07.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:07.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:07.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:07.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=46461 2021-01-15T16:51:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 46461 '!=' null 2021-01-15T16:51:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:08.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:08.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:08.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:08.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:08.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:08.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:08.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:08.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:08.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:08.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:08.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:08.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:09.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:09.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:09.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:09.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:09.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:09.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:09.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:09.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31898 2021-01-15T16:51:09.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31898 '!=' null 2021-01-15T16:51:09.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:09.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:09.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:09.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:09.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:09.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:09.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:09.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:09.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:09.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:09.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:10.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:10.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:10.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:10.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:10.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:10.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:11.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31898 2021-01-15T16:51:11.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31898 '!=' null 2021-01-15T16:51:11.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:11.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:11.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:11.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:11.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:11.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:11.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:11.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:11.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:11.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:11.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:11.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:11.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:11.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:11.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:12.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:12.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:12.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:12.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:12.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:12.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:12.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:12.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:12.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:12.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31898 2021-01-15T16:51:12.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31898 '!=' null 2021-01-15T16:51:12.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:12.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:12.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:12.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:12.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:12.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:12.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:12.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:12.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:12.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:12.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:13.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:13.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:13.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:13.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:13.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:13.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:13.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:13.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:14.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=47848 2021-01-15T16:51:14.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 47848 '!=' null 2021-01-15T16:51:14.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:14.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:14.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:14.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:14.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:14.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:14.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:14.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:14.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:14.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:14.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:14.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:15.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:15.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:15.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:15.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:15.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:15.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31205 2021-01-15T16:51:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31205 '!=' null 2021-01-15T16:51:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:15.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:15.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:15.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:15.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:15.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:15.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:15.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:15.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:16.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:16.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:16.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:16.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:16.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:16.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:16.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:16.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:16.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:16.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=47154 2021-01-15T16:51:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 47154 '!=' null 2021-01-15T16:51:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:17.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:17.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:17.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:17.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:17.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:17.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:17.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:17.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:17.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:17.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:17.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:17.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:17.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:18.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:18.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:18.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:18.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:18.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:18.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:18.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=47154 2021-01-15T16:51:18.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 47154 '!=' null 2021-01-15T16:51:18.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:18.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:18.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:18.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:18.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:18.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:18.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:19.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:19.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:19.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:19.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:19.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:19.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:19.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:19.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:19.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:19.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:19.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:19.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:19.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:20.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31898 2021-01-15T16:51:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31898 '!=' null 2021-01-15T16:51:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:20.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:20.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:20.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:20.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:20.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:20.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:20.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:20.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:20.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:20.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:20.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:21.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:21.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:21.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:21.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:21.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=31898 2021-01-15T16:51:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 31898 '!=' null 2021-01-15T16:51:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:21.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:21.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:21.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:21.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:21.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:21.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:21.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:21.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:22.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:22.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:22.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:22.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:22.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:22.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:22.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:22.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:22.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:23.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23923 2021-01-15T16:51:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23923 '!=' null 2021-01-15T16:51:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:23.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:23.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:23.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:23.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:23.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:23.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:23.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:23.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:51:23.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:23.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:23.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:51:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:51:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:51:24.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:51:24.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:51:24.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:51:24.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:51:24.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:51:24.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:51:24.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23923 2021-01-15T16:51:24.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23923 '!=' null 2021-01-15T16:51:24.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:51:24.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:51:24.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:51:24.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:51:24.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:51:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:51:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:51:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:51:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:51:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:51:24.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:51:25.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:51:25.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:51:25.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:51:25.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:51:25.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:51:25.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:51:25.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:51:25.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-01-15T16:51:25.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:51:25.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:51:26.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:51:26.193 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:51:26.193 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T16:51:26.193 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T16:51:26.193 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T16:51:26.193 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T16:51:26.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:51:26.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:51:26.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836506 2021-01-15T16:51:26.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836506 2021-01-15T16:51:26.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506' 2021-01-15T16:51:26.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:51:26.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672985 2021-01-15T16:51:26.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672985 2021-01-15T16:51:26.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985' 2021-01-15T16:51:26.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:51:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509462 2021-01-15T16:51:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509462 2021-01-15T16:51:26.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509462' 2021-01-15T16:51:26.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T16:51:26.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345941 2021-01-15T16:51:26.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345941 2021-01-15T16:51:26.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509462 3-85899345941' 2021-01-15T16:51:26.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T16:51:26.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182420 2021-01-15T16:51:26.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182420 2021-01-15T16:51:26.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509462 3-85899345941 4-107374182420' 2021-01-15T16:51:26.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:51:26.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T16:51:27.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018897 2021-01-15T16:51:27.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018897 2021-01-15T16:51:27.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-64424509462 3-85899345941 4-107374182420 5-128849018897' 2021-01-15T16:51:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:27.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836506 2021-01-15T16:51:27.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:51:27.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836506 2021-01-15T16:51:27.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:27.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836506 2021-01-15T16:51:27.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2021-01-15T16:51:27.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:51:27.014 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836506 2021-01-15T16:51:27.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836506 -lt 21474836506 2021-01-15T16:51:27.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:27.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672985 2021-01-15T16:51:27.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:27.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:51:27.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672985 2021-01-15T16:51:27.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:27.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672985 2021-01-15T16:51:27.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2021-01-15T16:51:27.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:51:27.406 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672985 2021-01-15T16:51:27.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672985 -lt 42949672985 2021-01-15T16:51:27.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:27.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509462 2021-01-15T16:51:27.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:27.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:51:27.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509462 2021-01-15T16:51:27.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:27.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509462 2021-01-15T16:51:27.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2021-01-15T16:51:27.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:51:27.800 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509462 2021-01-15T16:51:28.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509462 -lt 64424509462 2021-01-15T16:51:28.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:28.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345941 2021-01-15T16:51:28.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:28.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T16:51:28.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345941 2021-01-15T16:51:28.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345941 2021-01-15T16:51:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2021-01-15T16:51:28.193 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345941 2021-01-15T16:51:28.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:51:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345939 -lt 85899345941 2021-01-15T16:51:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:51:29.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:51:29.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T16:51:29.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345941 -lt 85899345941 2021-01-15T16:51:29.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:29.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182420 2021-01-15T16:51:29.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:29.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T16:51:29.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182420 2021-01-15T16:51:29.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:29.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182420 2021-01-15T16:51:29.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182420' 2021-01-15T16:51:29.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T16:51:29.993 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182420 2021-01-15T16:51:30.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182420 -lt 107374182420 2021-01-15T16:51:30.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:51:30.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018897 2021-01-15T16:51:30.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:51:30.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T16:51:30.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018897 2021-01-15T16:51:30.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:51:30.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018897 2021-01-15T16:51:30.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2021-01-15T16:51:30.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T16:51:30.393 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018897 2021-01-15T16:51:30.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018898 -lt 128849018897 2021-01-15T16:51:30.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-01-15T16:51:30.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-01-15T16:51:31.217 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:51:31.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-01-15T16:51:31.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-01-15T16:51:31.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:51:31.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:51:31.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:51:31.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:51:31.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:51:31.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:51:31.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-01-15T16:51:31.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:51:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:51:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:51:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-01-15T16:51:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=4 2021-01-15T16:51:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:51:31.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-01-15T16:51:31.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-01-15T16:51:32.521 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T16:51:32.521 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_log_t": { 2021-01-15T16:51:32.522 INFO:tasks.workunit.client.0.smithi145.stdout: "head": "37'161", 2021-01-15T16:51:32.522 INFO:tasks.workunit.client.0.smithi145.stdout: "tail": "33'10", 2021-01-15T16:51:32.522 INFO:tasks.workunit.client.0.smithi145.stdout: "log": [ 2021-01-15T16:51:32.522 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.522 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.523 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1e6eca61:::obj-11:head", 2021-01-15T16:51:32.523 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'11", 2021-01-15T16:51:32.523 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.523 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4219.0:1", 2021-01-15T16:51:32.523 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.524 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:26.653912+0000", 2021-01-15T16:51:32.524 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.524 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.524 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.524 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.525 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.525 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.525 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.525 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.526 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.526 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.526 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.526 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.526 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.527 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.527 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.527 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.527 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.527 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.528 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:18dee445:::obj-12:head", 2021-01-15T16:51:32.528 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'12", 2021-01-15T16:51:32.528 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.529 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4222.0:1", 2021-01-15T16:51:32.529 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.529 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:26.796177+0000", 2021-01-15T16:51:32.530 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.530 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.531 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.531 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.531 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.531 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.531 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.532 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.532 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.533 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.533 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.533 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.533 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.533 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.534 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.534 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.534 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.534 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.534 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:283c5c90:::obj-13:head", 2021-01-15T16:51:32.535 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'13", 2021-01-15T16:51:32.535 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.535 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4225.0:1", 2021-01-15T16:51:32.535 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.535 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:26.921011+0000", 2021-01-15T16:51:32.536 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.536 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.536 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.536 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.536 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.537 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.537 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.537 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.537 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.537 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.538 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.538 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.538 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.538 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.538 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.539 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.539 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.539 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.540 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cc896bf0:::obj-14:head", 2021-01-15T16:51:32.540 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'14", 2021-01-15T16:51:32.540 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.540 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4228.0:1", 2021-01-15T16:51:32.541 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.541 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.072909+0000", 2021-01-15T16:51:32.541 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.541 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.541 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.542 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.542 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.542 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.542 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.542 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.543 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.543 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.543 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.543 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.543 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.544 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.544 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.544 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.544 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.544 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.545 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a581f807:::obj-15:head", 2021-01-15T16:51:32.545 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'15", 2021-01-15T16:51:32.545 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.545 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4231.0:1", 2021-01-15T16:51:32.545 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.546 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.265116+0000", 2021-01-15T16:51:32.546 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.546 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.546 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.546 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.547 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.547 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.547 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.547 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.547 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.548 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.548 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.548 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.549 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.549 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.549 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.550 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.550 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.550 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.550 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c902ca7c:::obj-16:head", 2021-01-15T16:51:32.551 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'16", 2021-01-15T16:51:32.551 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.551 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4234.0:1", 2021-01-15T16:51:32.551 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.551 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.399843+0000", 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.552 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.553 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.553 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.553 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.553 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.553 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.554 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.554 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.554 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.554 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.554 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.555 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.555 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.555 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:655283c8:::obj-17:head", 2021-01-15T16:51:32.555 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'17", 2021-01-15T16:51:32.556 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.556 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4237.0:1", 2021-01-15T16:51:32.556 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.556 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.543225+0000", 2021-01-15T16:51:32.556 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.557 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.557 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.557 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.557 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.557 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.558 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.558 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.558 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.558 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.560 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.560 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.560 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d759d762:::obj-18:head", 2021-01-15T16:51:32.560 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'18", 2021-01-15T16:51:32.560 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.561 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4240.0:1", 2021-01-15T16:51:32.561 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.561 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.692322+0000", 2021-01-15T16:51:32.561 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.562 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.562 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.563 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.563 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.563 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.564 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.564 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.564 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.564 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.565 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.566 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.566 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1b934a2d:::obj-19:head", 2021-01-15T16:51:32.566 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'19", 2021-01-15T16:51:32.566 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.567 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4243.0:1", 2021-01-15T16:51:32.567 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.567 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:27.868888+0000", 2021-01-15T16:51:32.567 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.567 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.568 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.568 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.568 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.568 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.568 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.569 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.569 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.569 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.569 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.569 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.570 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.570 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.570 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.570 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.571 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.571 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2021-01-15T16:51:32.571 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'20", 2021-01-15T16:51:32.571 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.571 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4246.0:1", 2021-01-15T16:51:32.572 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.572 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.012059+0000", 2021-01-15T16:51:32.572 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.572 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.572 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.573 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.573 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.573 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.573 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.574 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.574 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.574 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.574 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.574 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.575 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.575 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.575 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.575 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.576 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ab396690:::obj-21:head", 2021-01-15T16:51:32.576 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'21", 2021-01-15T16:51:32.576 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.576 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4249.0:1", 2021-01-15T16:51:32.576 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.577 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.162415+0000", 2021-01-15T16:51:32.577 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.577 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.577 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.577 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.578 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.578 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.578 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.578 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.579 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.579 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.579 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.579 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.579 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.580 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.580 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.580 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.581 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:254983db:::obj-22:head", 2021-01-15T16:51:32.581 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'22", 2021-01-15T16:51:32.581 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.581 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4252.0:1", 2021-01-15T16:51:32.582 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.582 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.304968+0000", 2021-01-15T16:51:32.582 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.582 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.582 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.583 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.584 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.584 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.584 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.584 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.584 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.585 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.585 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.585 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.585 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e656299f:::obj-23:head", 2021-01-15T16:51:32.586 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'23", 2021-01-15T16:51:32.586 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.586 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4255.0:1", 2021-01-15T16:51:32.586 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.586 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.488680+0000", 2021-01-15T16:51:32.587 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.587 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.587 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.587 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.588 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.588 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.588 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.588 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.588 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.589 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.589 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.589 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.589 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.589 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.590 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.590 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.590 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.590 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5cdf412b:::obj-24:head", 2021-01-15T16:51:32.591 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'24", 2021-01-15T16:51:32.591 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.591 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4258.0:1", 2021-01-15T16:51:32.591 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.591 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.640022+0000", 2021-01-15T16:51:32.592 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.592 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.592 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.592 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.592 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.593 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.594 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.594 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.594 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.594 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.595 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.595 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.595 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cacae143:::obj-25:head", 2021-01-15T16:51:32.595 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'25", 2021-01-15T16:51:32.596 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.596 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4261.0:1", 2021-01-15T16:51:32.596 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.596 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.782766+0000", 2021-01-15T16:51:32.596 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.597 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.597 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.597 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.597 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.597 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.598 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.598 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.598 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.598 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.599 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.599 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.599 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.599 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.599 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.600 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.600 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.600 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:34b5bff6:::obj-26:head", 2021-01-15T16:51:32.600 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'26", 2021-01-15T16:51:32.600 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.601 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4264.0:1", 2021-01-15T16:51:32.601 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.601 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:28.923968+0000", 2021-01-15T16:51:32.601 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.602 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.602 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.602 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.603 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.603 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.603 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.604 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.604 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.604 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.604 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.605 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.605 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.605 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.606 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.606 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.606 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.607 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a354f8f6:::obj-27:head", 2021-01-15T16:51:32.607 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'27", 2021-01-15T16:51:32.607 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.607 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4267.0:1", 2021-01-15T16:51:32.607 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.608 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.057961+0000", 2021-01-15T16:51:32.608 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.608 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.609 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.609 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.609 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.609 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.609 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.610 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.610 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.610 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.610 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.610 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.611 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.611 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.611 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.612 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.612 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7421db49:::obj-28:head", 2021-01-15T16:51:32.612 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'28", 2021-01-15T16:51:32.612 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.612 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4270.0:1", 2021-01-15T16:51:32.613 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.613 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.242840+0000", 2021-01-15T16:51:32.613 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.613 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.613 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.614 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.614 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.614 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.614 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.615 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.615 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.615 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.615 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.615 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.616 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.616 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.616 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.617 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.617 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:54bc79ba:::obj-29:head", 2021-01-15T16:51:32.617 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'29", 2021-01-15T16:51:32.618 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.618 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4273.0:1", 2021-01-15T16:51:32.618 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.618 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.409151+0000", 2021-01-15T16:51:32.618 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.619 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.619 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.619 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.619 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.619 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.620 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.620 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.620 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.620 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.621 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.621 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.621 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.621 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.621 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.622 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.622 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.622 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:01b15595:::obj-30:head", 2021-01-15T16:51:32.622 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'30", 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4276.0:1", 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.551578+0000", 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.623 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.624 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.625 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.626 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2021-01-15T16:51:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'31", 2021-01-15T16:51:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.627 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4279.0:1", 2021-01-15T16:51:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.694756+0000", 2021-01-15T16:51:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.628 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.630 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.631 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1bf2562f:::obj-32:head", 2021-01-15T16:51:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'32", 2021-01-15T16:51:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4282.0:1", 2021-01-15T16:51:32.632 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:29.862739+0000", 2021-01-15T16:51:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.633 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.634 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.634 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.636 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.637 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:b982ae22:::obj-33:head", 2021-01-15T16:51:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'33", 2021-01-15T16:51:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4285.0:1", 2021-01-15T16:51:32.638 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.005226+0000", 2021-01-15T16:51:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.639 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.641 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.642 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1ad9f98e:::obj-34:head", 2021-01-15T16:51:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'34", 2021-01-15T16:51:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4288.0:1", 2021-01-15T16:51:32.643 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.147751+0000", 2021-01-15T16:51:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.644 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.645 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.646 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.647 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:941f43ff:::obj-35:head", 2021-01-15T16:51:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'35", 2021-01-15T16:51:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4291.0:1", 2021-01-15T16:51:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.648 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.307321+0000", 2021-01-15T16:51:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.649 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.650 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:24ebabe9:::obj-36:head", 2021-01-15T16:51:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'36", 2021-01-15T16:51:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.653 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4294.0:1", 2021-01-15T16:51:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.491177+0000", 2021-01-15T16:51:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.654 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.655 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.656 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.657 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5f3a5398:::obj-37:head", 2021-01-15T16:51:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'37", 2021-01-15T16:51:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4297.0:1", 2021-01-15T16:51:32.658 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.632938+0000", 2021-01-15T16:51:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.659 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.660 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.661 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.662 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c96cb37e:::obj-38:head", 2021-01-15T16:51:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'38", 2021-01-15T16:51:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4300.0:1", 2021-01-15T16:51:32.663 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.776272+0000", 2021-01-15T16:51:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.664 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.666 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.667 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:69e36996:::obj-39:head", 2021-01-15T16:51:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'39", 2021-01-15T16:51:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4303.0:1", 2021-01-15T16:51:32.668 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:30.943306+0000", 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.669 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.670 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.671 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.672 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:9f75773e:::obj-40:head", 2021-01-15T16:51:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'40", 2021-01-15T16:51:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4306.0:1", 2021-01-15T16:51:32.673 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.151994+0000", 2021-01-15T16:51:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.674 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.675 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.676 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.677 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.678 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2c01cd48:::obj-41:head", 2021-01-15T16:51:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'41", 2021-01-15T16:51:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.679 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4309.0:1", 2021-01-15T16:51:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.286663+0000", 2021-01-15T16:51:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.680 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5e676e3d:::obj-42:head", 2021-01-15T16:51:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'42", 2021-01-15T16:51:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4312.0:1", 2021-01-15T16:51:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.428694+0000", 2021-01-15T16:51:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1215f697:::obj-43:head", 2021-01-15T16:51:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'43", 2021-01-15T16:51:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4315.0:1", 2021-01-15T16:51:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.570432+0000", 2021-01-15T16:51:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3a0aa357:::obj-44:head", 2021-01-15T16:51:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'44", 2021-01-15T16:51:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4318.0:1", 2021-01-15T16:51:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.695 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.747753+0000", 2021-01-15T16:51:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.696 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.697 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.698 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6366950a:::obj-45:head", 2021-01-15T16:51:32.699 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'45", 2021-01-15T16:51:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4321.0:1", 2021-01-15T16:51:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.700 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:31.900372+0000", 2021-01-15T16:51:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.701 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.702 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.703 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.704 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:621cb2b3:::obj-46:head", 2021-01-15T16:51:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'46", 2021-01-15T16:51:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.705 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4324.0:1", 2021-01-15T16:51:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.034130+0000", 2021-01-15T16:51:32.706 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.707 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.709 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.710 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d83ab3cd:::obj-47:head", 2021-01-15T16:51:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'47", 2021-01-15T16:51:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.711 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4327.0:1", 2021-01-15T16:51:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.176319+0000", 2021-01-15T16:51:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.712 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.713 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.714 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.715 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3d8e6d4a:::obj-48:head", 2021-01-15T16:51:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'48", 2021-01-15T16:51:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.716 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4330.0:1", 2021-01-15T16:51:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.368959+0000", 2021-01-15T16:51:32.717 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.718 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.719 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.720 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.721 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:50b1e2f2:::obj-49:head", 2021-01-15T16:51:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'49", 2021-01-15T16:51:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4333.0:1", 2021-01-15T16:51:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.722 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.519697+0000", 2021-01-15T16:51:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.723 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.724 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.725 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.726 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.727 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c75430e6:::obj-50:head", 2021-01-15T16:51:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'50", 2021-01-15T16:51:32.728 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4336.0:1", 2021-01-15T16:51:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.729 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.671031+0000", 2021-01-15T16:51:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.730 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.731 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.733 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.734 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.735 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:eaf3ca0a:::obj-51:head", 2021-01-15T16:51:32.736 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'51", 2021-01-15T16:51:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4339.0:1", 2021-01-15T16:51:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.737 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.813459+0000", 2021-01-15T16:51:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.738 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.739 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.740 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.741 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:86a15bea:::obj-52:head", 2021-01-15T16:51:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'52", 2021-01-15T16:51:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4342.0:1", 2021-01-15T16:51:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.742 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:32.988510+0000", 2021-01-15T16:51:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.743 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.744 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.745 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.746 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1df15dae:::obj-53:head", 2021-01-15T16:51:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'53", 2021-01-15T16:51:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.747 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4345.0:1", 2021-01-15T16:51:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.140087+0000", 2021-01-15T16:51:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.748 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.749 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.750 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.752 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:29304dc8:::obj-54:head", 2021-01-15T16:51:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'54", 2021-01-15T16:51:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4348.0:1", 2021-01-15T16:51:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.753 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.289254+0000", 2021-01-15T16:51:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.754 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.755 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.756 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:de02c26d:::obj-55:head", 2021-01-15T16:51:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'55", 2021-01-15T16:51:32.757 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4351.0:1", 2021-01-15T16:51:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.440111+0000", 2021-01-15T16:51:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.758 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.759 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.760 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.762 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0d36fc08:::obj-56:head", 2021-01-15T16:51:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'56", 2021-01-15T16:51:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.763 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4354.0:1", 2021-01-15T16:51:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.617105+0000", 2021-01-15T16:51:32.764 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.765 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.766 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.767 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.768 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.769 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2b2cbcba:::obj-57:head", 2021-01-15T16:51:32.770 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'57", 2021-01-15T16:51:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4357.0:1", 2021-01-15T16:51:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.758956+0000", 2021-01-15T16:51:32.771 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.772 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.773 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.774 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:838be76e:::obj-58:head", 2021-01-15T16:51:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'58", 2021-01-15T16:51:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.775 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4360.0:1", 2021-01-15T16:51:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:33.892696+0000", 2021-01-15T16:51:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.776 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.777 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.778 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.779 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2c66fd6f:::obj-59:head", 2021-01-15T16:51:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'59", 2021-01-15T16:51:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4363.0:1", 2021-01-15T16:51:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.780 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.035143+0000", 2021-01-15T16:51:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.781 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.782 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2b18273b:::obj-60:head", 2021-01-15T16:51:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'60", 2021-01-15T16:51:32.784 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4366.0:1", 2021-01-15T16:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.186169+0000", 2021-01-15T16:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.786 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.787 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.788 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:fcfa3cd1:::obj-61:head", 2021-01-15T16:51:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'61", 2021-01-15T16:51:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.789 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4369.0:1", 2021-01-15T16:51:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.345497+0000", 2021-01-15T16:51:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.790 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.791 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.792 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.793 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:16e51a68:::obj-62:head", 2021-01-15T16:51:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'62", 2021-01-15T16:51:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.794 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4372.0:1", 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.646929+0000", 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.795 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.796 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.797 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.798 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c15678fc:::obj-63:head", 2021-01-15T16:51:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'63", 2021-01-15T16:51:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4375.0:1", 2021-01-15T16:51:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.799 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.781574+0000", 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.800 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.801 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.802 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e066252d:::obj-64:head", 2021-01-15T16:51:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'64", 2021-01-15T16:51:32.803 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4378.0:1", 2021-01-15T16:51:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:34.964950+0000", 2021-01-15T16:51:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.804 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.805 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.806 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.807 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:95a4ceaf:::obj-65:head", 2021-01-15T16:51:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'65", 2021-01-15T16:51:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4381.0:1", 2021-01-15T16:51:32.808 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.116085+0000", 2021-01-15T16:51:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.809 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.810 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.811 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.812 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6e97b31a:::obj-66:head", 2021-01-15T16:51:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'66", 2021-01-15T16:51:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4384.0:1", 2021-01-15T16:51:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.813 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.249868+0000", 2021-01-15T16:51:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.814 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.815 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.816 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ceed8b4e:::obj-67:head", 2021-01-15T16:51:32.817 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'67", 2021-01-15T16:51:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4387.0:1", 2021-01-15T16:51:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.400513+0000", 2021-01-15T16:51:32.818 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.819 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.820 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.821 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:05e2e722:::obj-68:head", 2021-01-15T16:51:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'68", 2021-01-15T16:51:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.822 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4390.0:1", 2021-01-15T16:51:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.568590+0000", 2021-01-15T16:51:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.823 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.824 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.825 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.826 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c40f7de9:::obj-69:head", 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'69", 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4393.0:1", 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.827 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.702673+0000", 2021-01-15T16:51:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.828 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.829 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.830 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7bab3581:::obj-70:head", 2021-01-15T16:51:32.831 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'70", 2021-01-15T16:51:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4396.0:1", 2021-01-15T16:51:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.836610+0000", 2021-01-15T16:51:32.832 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.833 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.834 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.835 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cb9edc4f:::obj-71:head", 2021-01-15T16:51:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'71", 2021-01-15T16:51:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.836 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4399.0:1", 2021-01-15T16:51:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:35.987463+0000", 2021-01-15T16:51:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.837 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.838 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.839 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.840 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:88ae729f:::obj-72:head", 2021-01-15T16:51:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'72", 2021-01-15T16:51:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4402.0:1", 2021-01-15T16:51:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.841 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.180033+0000", 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.842 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.843 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.844 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:887b5de4:::obj-73:head", 2021-01-15T16:51:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'73", 2021-01-15T16:51:32.845 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4405.0:1", 2021-01-15T16:51:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.313285+0000", 2021-01-15T16:51:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.846 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.847 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.848 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.849 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:996efeaa:::obj-74:head", 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'74", 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4408.0:1", 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.850 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.446247+0000", 2021-01-15T16:51:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.851 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.852 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.853 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:942222a0:::obj-75:head", 2021-01-15T16:51:32.854 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'75", 2021-01-15T16:51:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4411.0:1", 2021-01-15T16:51:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.580444+0000", 2021-01-15T16:51:32.855 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.856 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.857 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.858 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:35f0499f:::obj-76:head", 2021-01-15T16:51:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'76", 2021-01-15T16:51:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.859 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4414.0:1", 2021-01-15T16:51:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.714331+0000", 2021-01-15T16:51:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.860 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.861 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.862 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.863 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:15cd38e3:::obj-77:head", 2021-01-15T16:51:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'77", 2021-01-15T16:51:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4417.0:1", 2021-01-15T16:51:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.864 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:36.917324+0000", 2021-01-15T16:51:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.865 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.866 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.867 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7adeb77b:::obj-78:head", 2021-01-15T16:51:32.868 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'78", 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4420.0:1", 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.165323+0000", 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.869 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.870 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.871 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.872 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c92176ea:::obj-79:head", 2021-01-15T16:51:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'79", 2021-01-15T16:51:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.873 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4423.0:1", 2021-01-15T16:51:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.309151+0000", 2021-01-15T16:51:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.874 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.875 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.876 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.877 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:14abac33:::obj-80:head", 2021-01-15T16:51:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'80", 2021-01-15T16:51:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4426.0:1", 2021-01-15T16:51:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.878 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.493696+0000", 2021-01-15T16:51:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.879 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.880 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.881 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ea1005df:::obj-81:head", 2021-01-15T16:51:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'81", 2021-01-15T16:51:32.882 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4429.0:1", 2021-01-15T16:51:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.645121+0000", 2021-01-15T16:51:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.883 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.884 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.885 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.886 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:df18954d:::obj-82:head", 2021-01-15T16:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'82", 2021-01-15T16:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4432.0:1", 2021-01-15T16:51:32.887 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.778332+0000", 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.888 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.889 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.890 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:cdd3834b:::obj-83:head", 2021-01-15T16:51:32.891 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'83", 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4435.0:1", 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:37.912613+0000", 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.892 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.893 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.894 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.894 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.894 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.894 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.894 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.895 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.895 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.895 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.895 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.895 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.896 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.896 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:dfbf520a:::obj-84:head", 2021-01-15T16:51:32.896 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'84", 2021-01-15T16:51:32.896 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.896 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4438.0:1", 2021-01-15T16:51:32.897 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.897 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.097404+0000", 2021-01-15T16:51:32.897 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.897 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.897 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.898 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.898 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.898 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.898 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.898 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.899 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.899 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.899 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.899 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.899 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.900 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.900 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.900 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.900 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.900 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.901 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1dd3df2f:::obj-85:head", 2021-01-15T16:51:32.901 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'85", 2021-01-15T16:51:32.901 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.901 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4441.0:1", 2021-01-15T16:51:32.901 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.902 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.239998+0000", 2021-01-15T16:51:32.902 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.902 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.902 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.902 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.903 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.903 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.903 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.903 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.903 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.904 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.904 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.904 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.904 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.904 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.905 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.905 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.905 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.905 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.905 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:54554b77:::obj-86:head", 2021-01-15T16:51:32.906 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'86", 2021-01-15T16:51:32.906 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.906 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4444.0:1", 2021-01-15T16:51:32.906 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.906 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.390187+0000", 2021-01-15T16:51:32.907 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.907 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.907 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.907 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.907 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.908 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.909 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.909 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.909 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.909 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.909 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.910 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.910 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.910 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:43a48f98:::obj-87:head", 2021-01-15T16:51:32.910 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'87", 2021-01-15T16:51:32.910 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.911 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4447.0:1", 2021-01-15T16:51:32.911 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.911 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.533295+0000", 2021-01-15T16:51:32.911 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.911 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.912 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.912 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.912 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.912 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.912 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.913 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.913 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.913 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.913 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.913 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.914 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.914 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.914 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.914 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.914 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3f7b3ecc:::obj-88:head", 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'88", 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4450.0:1", 2021-01-15T16:51:32.915 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.916 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.676238+0000", 2021-01-15T16:51:32.916 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.916 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.916 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.916 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.917 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.917 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.917 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.917 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.917 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.918 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.918 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.918 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.918 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.918 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.919 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.919 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.919 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.919 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.919 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d25ef7d2:::obj-89:head", 2021-01-15T16:51:32.920 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'89", 2021-01-15T16:51:32.920 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.920 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4453.0:1", 2021-01-15T16:51:32.920 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.920 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.835799+0000", 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.921 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.922 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.922 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.922 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.922 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.922 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.923 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.923 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.923 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.923 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.923 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.924 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.924 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.924 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ea27d688:::obj-90:head", 2021-01-15T16:51:32.924 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'90", 2021-01-15T16:51:32.924 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4456.0:1", 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:38.977821+0000", 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.925 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.926 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.926 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.926 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.926 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.926 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.927 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.927 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.927 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.927 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.927 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.928 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.928 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.928 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.928 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.928 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.929 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:088f9d12:::obj-91:head", 2021-01-15T16:51:32.929 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'91", 2021-01-15T16:51:32.929 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.929 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4459.0:1", 2021-01-15T16:51:32.929 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.930 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.129049+0000", 2021-01-15T16:51:32.930 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.930 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.930 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.930 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.931 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.932 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.932 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.932 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.932 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.932 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.933 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.933 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.933 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:206f25a5:::obj-92:head", 2021-01-15T16:51:32.933 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'92", 2021-01-15T16:51:32.934 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.934 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4462.0:1", 2021-01-15T16:51:32.934 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.934 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.296119+0000", 2021-01-15T16:51:32.934 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.935 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.935 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.935 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.935 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.935 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.936 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.936 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.936 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.936 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.936 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.937 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.937 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.937 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.937 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.937 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.938 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.938 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.938 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:501cca61:::obj-93:head", 2021-01-15T16:51:32.938 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'93", 2021-01-15T16:51:32.938 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4465.0:1", 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.463633+0000", 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.939 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.940 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.940 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.940 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.940 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.940 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.941 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.941 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.941 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.941 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.941 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.942 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.942 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.942 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.942 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.942 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.943 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1ed468c0:::obj-94:head", 2021-01-15T16:51:32.943 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'94", 2021-01-15T16:51:32.943 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.943 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4468.0:1", 2021-01-15T16:51:32.943 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.944 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.605627+0000", 2021-01-15T16:51:32.944 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.944 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.944 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.944 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.945 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.945 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.945 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.945 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.945 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.946 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.946 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.946 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.946 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.946 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.947 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.947 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.947 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.947 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.947 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:e8cba918:::obj-95:head", 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'95", 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4471.0:1", 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.748244+0000", 2021-01-15T16:51:32.948 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.949 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.949 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.949 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.949 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.949 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.950 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.950 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.950 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.950 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.950 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.951 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.951 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.951 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.951 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.951 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.952 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.952 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.952 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:403c3dfc:::obj-96:head", 2021-01-15T16:51:32.952 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'96", 2021-01-15T16:51:32.952 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.953 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4474.0:1", 2021-01-15T16:51:32.953 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.953 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:39.890451+0000", 2021-01-15T16:51:32.953 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.953 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.954 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.954 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.954 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.954 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.954 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.955 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.956 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.956 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.956 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.956 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.956 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.957 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7c2d0cb1:::obj-97:head", 2021-01-15T16:51:32.957 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'97", 2021-01-15T16:51:32.957 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.957 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4477.0:1", 2021-01-15T16:51:32.957 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.958 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.108633+0000", 2021-01-15T16:51:32.958 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.958 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.958 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.958 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.959 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.960 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.960 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.960 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.960 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.960 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.961 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.961 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.961 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.961 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6f3e26a7:::obj-98:head", 2021-01-15T16:51:32.961 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'98", 2021-01-15T16:51:32.962 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.962 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4480.0:1", 2021-01-15T16:51:32.962 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.962 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.257977+0000", 2021-01-15T16:51:32.962 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.963 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.964 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.964 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.964 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.964 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.964 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.965 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.965 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.965 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.965 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.965 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.966 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.966 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a4b22c6a:::obj-99:head", 2021-01-15T16:51:32.966 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'99", 2021-01-15T16:51:32.966 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.966 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4483.0:1", 2021-01-15T16:51:32.967 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.967 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.400639+0000", 2021-01-15T16:51:32.967 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.967 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.967 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.968 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.969 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.969 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.969 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.969 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.969 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.970 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.970 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.970 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.970 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.970 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:25c09ea8:::obj-100:head", 2021-01-15T16:51:32.971 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'100", 2021-01-15T16:51:32.971 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.971 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4486.0:1", 2021-01-15T16:51:32.971 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.971 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.543392+0000", 2021-01-15T16:51:32.972 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.972 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.972 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.972 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.972 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.973 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.974 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.974 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.974 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.974 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.974 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.975 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.975 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.975 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:be8b0193:::obj-101:head", 2021-01-15T16:51:32.975 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'101", 2021-01-15T16:51:32.975 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.976 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4489.0:1", 2021-01-15T16:51:32.976 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.976 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.727778+0000", 2021-01-15T16:51:32.976 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.976 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.977 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.977 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.977 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.977 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.977 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.978 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.978 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.978 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.978 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.978 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.979 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.980 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6aa5e219:::obj-102:head", 2021-01-15T16:51:32.980 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'102", 2021-01-15T16:51:32.980 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.980 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4492.0:1", 2021-01-15T16:51:32.980 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.981 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:40.878475+0000", 2021-01-15T16:51:32.981 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.981 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.981 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.981 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.982 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.982 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.982 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.982 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.982 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.983 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.983 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.983 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.983 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.983 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.984 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.984 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.984 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.984 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.984 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ff8cd32f:::obj-103:head", 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'103", 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4495.0:1", 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.021726+0000", 2021-01-15T16:51:32.985 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.986 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.986 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.986 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.986 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.986 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.987 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.987 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.988 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.988 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.988 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.988 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.988 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.989 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.989 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.989 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.989 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.990 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:fcaf5c1c:::obj-104:head", 2021-01-15T16:51:32.990 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'104", 2021-01-15T16:51:32.990 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.990 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4498.0:1", 2021-01-15T16:51:32.990 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.991 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.162657+0000", 2021-01-15T16:51:32.991 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.991 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.991 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.991 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.992 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.992 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.992 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.992 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.992 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.993 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.993 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.993 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.993 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.993 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.994 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.994 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.994 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.994 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.994 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2111aa60:::obj-105:head", 2021-01-15T16:51:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'105", 2021-01-15T16:51:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4501.0:1", 2021-01-15T16:51:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:32.995 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.355449+0000", 2021-01-15T16:51:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:32.996 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:32.997 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.998 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:9fed4eb7:::obj-106:head", 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'106", 2021-01-15T16:51:32.999 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4504.0:1", 2021-01-15T16:51:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.498801+0000", 2021-01-15T16:51:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.000 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.001 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.002 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.003 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0b0c0bce:::obj-107:head", 2021-01-15T16:51:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'107", 2021-01-15T16:51:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.004 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4507.0:1", 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.649548+0000", 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.005 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.006 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.006 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.006 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.006 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.006 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.007 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.007 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.007 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.007 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.007 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.008 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.008 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.008 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.008 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.008 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:b7ce613a:::obj-108:head", 2021-01-15T16:51:33.009 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'108", 2021-01-15T16:51:33.009 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.009 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4510.0:1", 2021-01-15T16:51:33.009 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.009 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.792215+0000", 2021-01-15T16:51:33.010 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.010 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.010 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.010 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.010 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.011 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.011 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.011 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.011 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.011 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.012 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.013 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.013 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.013 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:4f32cd61:::obj-109:head", 2021-01-15T16:51:33.013 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'109", 2021-01-15T16:51:33.013 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.014 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4513.0:1", 2021-01-15T16:51:33.014 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.014 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:41.976340+0000", 2021-01-15T16:51:33.014 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.014 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.015 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.015 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.015 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.015 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.015 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.016 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.017 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.017 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.017 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.017 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.017 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:40f97f88:::obj-110:head", 2021-01-15T16:51:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'110", 2021-01-15T16:51:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4516.0:1", 2021-01-15T16:51:33.018 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.110525+0000", 2021-01-15T16:51:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.019 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.020 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.020 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.020 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.020 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.020 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.021 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.022 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.022 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.022 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.022 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:1d8e7a62:::obj-111:head", 2021-01-15T16:51:33.022 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'111", 2021-01-15T16:51:33.023 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.023 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4519.0:1", 2021-01-15T16:51:33.023 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.023 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.269300+0000", 2021-01-15T16:51:33.023 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.024 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.024 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.024 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.024 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.024 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.025 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.025 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.025 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.025 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.025 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.026 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.026 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.026 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.026 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.026 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.027 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.027 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.027 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3a283967:::obj-112:head", 2021-01-15T16:51:33.027 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'112", 2021-01-15T16:51:33.027 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4522.0:1", 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.403177+0000", 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.028 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.029 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.029 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.029 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.029 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.029 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.030 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.030 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.030 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.030 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.030 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.031 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.031 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.031 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.031 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.031 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.032 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:2acdc9e7:::obj-113:head", 2021-01-15T16:51:33.032 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'113", 2021-01-15T16:51:33.032 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.032 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4525.0:1", 2021-01-15T16:51:33.032 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.571324+0000", 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.033 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.034 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.034 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.034 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.034 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.034 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.035 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.035 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.035 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.035 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.035 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.036 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.036 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.036 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7f18a9ad:::obj-114:head", 2021-01-15T16:51:33.036 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'114", 2021-01-15T16:51:33.037 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.037 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4528.0:1", 2021-01-15T16:51:33.037 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.037 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.721322+0000", 2021-01-15T16:51:33.037 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.038 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.039 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.039 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.039 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.039 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.039 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.040 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.040 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.040 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.040 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.040 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.041 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.041 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:852b8a34:::obj-115:head", 2021-01-15T16:51:33.041 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'115", 2021-01-15T16:51:33.041 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.041 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4531.0:1", 2021-01-15T16:51:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:42.872712+0000", 2021-01-15T16:51:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.043 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.043 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.043 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.043 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.043 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.044 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.044 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.044 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.044 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.044 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.045 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.045 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.045 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.045 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.045 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.046 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:589de4f7:::obj-116:head", 2021-01-15T16:51:33.046 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'116", 2021-01-15T16:51:33.046 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.046 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4534.0:1", 2021-01-15T16:51:33.046 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.024465+0000", 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.047 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.048 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.048 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.048 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.048 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.048 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.049 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.049 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.049 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.049 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.049 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.050 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.050 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.050 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.050 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:14f2571c:::obj-117:head", 2021-01-15T16:51:33.050 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'117", 2021-01-15T16:51:33.051 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.051 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4537.0:1", 2021-01-15T16:51:33.051 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.051 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.232864+0000", 2021-01-15T16:51:33.051 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.052 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.052 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.052 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.052 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.052 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.053 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.054 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.054 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.054 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.054 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.054 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.055 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.055 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:dd36f0e5:::obj-118:head", 2021-01-15T16:51:33.055 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'118", 2021-01-15T16:51:33.055 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.055 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4540.0:1", 2021-01-15T16:51:33.056 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.056 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.366604+0000", 2021-01-15T16:51:33.056 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.056 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.056 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.057 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.057 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.057 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.057 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.057 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.058 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.058 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.058 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.058 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.058 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.059 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.059 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.059 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.059 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.059 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0d7fa432:::obj-119:head", 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'119", 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4543.0:1", 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.060 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.500053+0000", 2021-01-15T16:51:33.061 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.061 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.061 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.061 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.061 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.062 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.062 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.062 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.062 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.062 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.063 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.063 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.063 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.063 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.063 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.064 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.064 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.064 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.064 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:295ec31a:::obj-120:head", 2021-01-15T16:51:33.064 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'120", 2021-01-15T16:51:33.065 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.065 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4546.0:1", 2021-01-15T16:51:33.065 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.065 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.634880+0000", 2021-01-15T16:51:33.065 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.066 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.066 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.066 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.066 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.066 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.067 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.067 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.067 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.067 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.067 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.068 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.069 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.069 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c2863561:::obj-121:head", 2021-01-15T16:51:33.069 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'121", 2021-01-15T16:51:33.069 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.069 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4549.0:1", 2021-01-15T16:51:33.070 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.070 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.769090+0000", 2021-01-15T16:51:33.070 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.070 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.070 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.071 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.071 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.071 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.071 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.071 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.072 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.072 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.072 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.072 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.072 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.073 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0b943493:::obj-122:head", 2021-01-15T16:51:33.074 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'122", 2021-01-15T16:51:33.074 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.074 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4552.0:1", 2021-01-15T16:51:33.074 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:43.944919+0000", 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.075 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.076 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.076 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.076 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.076 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.076 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.077 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.077 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.077 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.078 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.078 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.078 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:4cfecbf8:::obj-123:head", 2021-01-15T16:51:33.078 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'123", 2021-01-15T16:51:33.079 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.079 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4555.0:1", 2021-01-15T16:51:33.079 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.079 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.087446+0000", 2021-01-15T16:51:33.079 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.081 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.081 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.081 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.081 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.081 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.082 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.083 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.083 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:fd18e511:::obj-124:head", 2021-01-15T16:51:33.083 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'124", 2021-01-15T16:51:33.083 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.083 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4558.0:1", 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.213273+0000", 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.084 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.085 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.085 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.085 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.085 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.086 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.086 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.086 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.086 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.086 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.087 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.087 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.087 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.087 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.087 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:4f299322:::obj-125:head", 2021-01-15T16:51:33.088 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'125", 2021-01-15T16:51:33.088 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.088 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4561.0:1", 2021-01-15T16:51:33.088 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.088 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.362855+0000", 2021-01-15T16:51:33.089 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.089 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.089 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.089 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.089 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.090 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.090 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.090 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.090 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.090 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.091 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.091 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.091 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.091 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.091 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.092 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.092 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.092 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.092 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:302ecc7e:::obj-126:head", 2021-01-15T16:51:33.092 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'126", 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4564.0:1", 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.531471+0000", 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.093 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.094 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.094 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.094 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.094 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.094 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.095 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.095 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.095 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.095 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.095 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.096 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.096 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.096 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.096 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.096 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.097 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.097 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:42feb657:::obj-127:head", 2021-01-15T16:51:33.097 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'127", 2021-01-15T16:51:33.097 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.097 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4567.0:1", 2021-01-15T16:51:33.098 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.098 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.666075+0000", 2021-01-15T16:51:33.098 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.098 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.098 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.099 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.100 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.100 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.100 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.100 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.100 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.101 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.101 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.101 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.101 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.101 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:5e6b9a81:::obj-128:head", 2021-01-15T16:51:33.102 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'128", 2021-01-15T16:51:33.102 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.102 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4570.0:1", 2021-01-15T16:51:33.102 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.102 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.808422+0000", 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.103 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.104 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.104 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.104 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.104 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.104 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0d242242:::obj-129:head", 2021-01-15T16:51:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'129", 2021-01-15T16:51:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4573.0:1", 2021-01-15T16:51:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:44.934198+0000", 2021-01-15T16:51:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ed409294:::obj-130:head", 2021-01-15T16:51:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'130", 2021-01-15T16:51:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4576.0:1", 2021-01-15T16:51:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.067764+0000", 2021-01-15T16:51:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.113 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.114 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.114 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.114 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.114 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.114 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.115 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.115 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.115 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.115 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:720761ff:::obj-131:head", 2021-01-15T16:51:33.115 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'131", 2021-01-15T16:51:33.116 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.116 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4579.0:1", 2021-01-15T16:51:33.116 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.116 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.370470+0000", 2021-01-15T16:51:33.116 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.117 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.118 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.118 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.118 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.118 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.118 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.119 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.119 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.119 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.119 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.120 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.120 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7715ee72:::obj-132:head", 2021-01-15T16:51:33.120 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'132", 2021-01-15T16:51:33.120 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.120 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4582.0:1", 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.511863+0000", 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.121 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.122 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.122 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.122 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.122 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.122 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.123 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.123 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.123 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.123 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.123 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.124 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.124 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.124 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.124 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.124 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:580938e3:::obj-133:head", 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'133", 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4585.0:1", 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.653238+0000", 2021-01-15T16:51:33.125 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.126 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.126 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.126 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.126 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.126 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.127 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.127 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.127 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.127 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.127 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.128 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.128 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.128 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.128 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.128 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:05dc98b5:::obj-134:head", 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'134", 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.129 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4588.0:1", 2021-01-15T16:51:33.130 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.130 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.829323+0000", 2021-01-15T16:51:33.130 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.130 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.130 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.131 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.131 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.131 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.131 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.131 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.132 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.132 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.132 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.132 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.132 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:73221e51:::obj-135:head", 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'135", 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4591.0:1", 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.134 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:45.963839+0000", 2021-01-15T16:51:33.135 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.135 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.135 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.135 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.135 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.136 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.136 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.136 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.136 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.136 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.137 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.137 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.137 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.137 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.137 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3a97ede2:::obj-136:head", 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'136", 2021-01-15T16:51:33.138 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.139 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4594.0:1", 2021-01-15T16:51:33.139 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.139 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.106653+0000", 2021-01-15T16:51:33.139 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.139 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.140 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.140 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.140 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.140 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.140 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.141 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.141 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.141 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.141 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.141 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.142 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.142 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.142 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.142 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.142 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:9e95a205:::obj-137:head", 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'137", 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4597.0:1", 2021-01-15T16:51:33.143 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.144 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.240843+0000", 2021-01-15T16:51:33.144 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.144 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.144 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.144 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.145 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.145 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.145 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.145 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.146 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.146 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.146 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.146 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.147 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.148 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:27f846bd:::obj-138:head", 2021-01-15T16:51:33.148 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'138", 2021-01-15T16:51:33.148 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.148 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4600.0:1", 2021-01-15T16:51:33.148 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.441535+0000", 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.149 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.150 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.150 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.150 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.150 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.150 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.151 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.151 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.151 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.151 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.151 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:bb2f7605:::obj-139:head", 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'139", 2021-01-15T16:51:33.152 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.153 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4603.0:1", 2021-01-15T16:51:33.153 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.153 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.573999+0000", 2021-01-15T16:51:33.153 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.153 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.154 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.154 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.154 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.154 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.154 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.155 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.155 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.155 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.155 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.155 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.156 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:27668d5f:::obj-140:head", 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'140", 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4606.0:1", 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.157 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.709163+0000", 2021-01-15T16:51:33.158 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.158 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.158 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.158 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.158 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.159 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.159 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.159 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.159 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.159 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.160 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.160 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.160 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.160 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.160 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.161 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.161 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.161 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.161 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0d70cb86:::obj-141:head", 2021-01-15T16:51:33.161 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'141", 2021-01-15T16:51:33.162 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.162 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4609.0:1", 2021-01-15T16:51:33.162 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.162 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:46.851252+0000", 2021-01-15T16:51:33.162 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.163 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.163 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.163 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.163 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.163 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.164 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.164 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.164 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.164 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.164 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.165 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.165 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.165 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.165 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.165 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.166 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.166 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.166 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:347b39b5:::obj-more-1:head", 2021-01-15T16:51:33.166 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'142", 2021-01-15T16:51:33.166 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.167 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4616.0:1", 2021-01-15T16:51:33.167 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.167 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:47.961849+0000", 2021-01-15T16:51:33.167 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.167 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.168 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.168 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.168 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.168 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.169 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.169 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.169 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.169 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.169 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.170 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.170 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.170 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.170 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.170 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.171 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.171 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.171 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:0c63cc63:::obj-more-2:head", 2021-01-15T16:51:33.171 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'143", 2021-01-15T16:51:33.171 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.172 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4619.0:1", 2021-01-15T16:51:33.172 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.172 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:49.623605+0000", 2021-01-15T16:51:33.172 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.172 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.173 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.173 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.173 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.173 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.173 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.174 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.174 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.174 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.174 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.174 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.175 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.175 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.175 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.175 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.175 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:3ce69b57:::obj-more-3:head", 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'144", 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4622.0:1", 2021-01-15T16:51:33.176 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.177 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:49.809590+0000", 2021-01-15T16:51:33.177 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.177 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.177 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.177 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.178 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.178 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.178 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.178 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.179 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.179 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.179 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.179 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.179 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.180 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:50476200:::obj-more-4:head", 2021-01-15T16:51:33.181 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'145", 2021-01-15T16:51:33.181 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.181 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4625.0:1", 2021-01-15T16:51:33.181 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.181 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:50.020649+0000", 2021-01-15T16:51:33.182 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.182 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.182 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.182 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.182 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.183 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.183 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.183 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.183 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.183 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.184 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.184 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.184 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.185 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.185 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.185 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:7a39cfd8:::obj-more-5:head", 2021-01-15T16:51:33.185 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'146", 2021-01-15T16:51:33.186 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.186 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4628.0:1", 2021-01-15T16:51:33.186 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.186 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:50.320748+0000", 2021-01-15T16:51:33.187 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.187 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.187 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.187 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.187 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.188 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.189 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.189 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.189 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.189 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.189 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.190 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.190 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.190 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:a7159f95:::obj-more-6:head", 2021-01-15T16:51:33.190 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'147", 2021-01-15T16:51:33.190 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.191 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4631.0:1", 2021-01-15T16:51:33.191 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.191 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:50.531247+0000", 2021-01-15T16:51:33.191 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.191 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.192 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.192 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.192 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.192 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.192 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.193 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.193 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.193 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.193 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.193 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.194 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.194 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.194 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.194 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.194 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.195 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.195 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:37f29f5f:::obj-more-7:head", 2021-01-15T16:51:33.195 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'148", 2021-01-15T16:51:33.195 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.195 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4634.0:1", 2021-01-15T16:51:33.196 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.196 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:50.799701+0000", 2021-01-15T16:51:33.196 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.196 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.196 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.197 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.198 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.198 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.198 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.198 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.198 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.199 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.199 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.199 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.199 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:4e4836c1:::obj-more-8:head", 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'149", 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4637.0:1", 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:51.047546+0000", 2021-01-15T16:51:33.200 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.201 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.201 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.201 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.201 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.201 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.202 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.202 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.202 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.202 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.203 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.203 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.203 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.204 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.204 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.204 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.204 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:d56cad57:::obj-more-9:head", 2021-01-15T16:51:33.204 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'150", 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4640.0:1", 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:51.263181+0000", 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.205 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.206 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.206 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.206 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.206 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.206 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.207 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.207 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.207 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.207 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.207 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.208 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.209 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:96ff4b77:::obj-more-10:head", 2021-01-15T16:51:33.209 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'151", 2021-01-15T16:51:33.209 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.209 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4643.0:1", 2021-01-15T16:51:33.209 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.210 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:51.547201+0000", 2021-01-15T16:51:33.210 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.210 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.210 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.210 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.211 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.212 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.212 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.212 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.212 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.212 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:b8937ca5:::obj-more-11:head", 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'152", 2021-01-15T16:51:33.213 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.214 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4646.0:1", 2021-01-15T16:51:33.214 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.214 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:51.782475+0000", 2021-01-15T16:51:33.214 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.214 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.215 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.215 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.215 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.215 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.215 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.216 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.217 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:c949902e:::obj-more-12:head", 2021-01-15T16:51:33.218 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'153", 2021-01-15T16:51:33.218 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.218 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4649.0:1", 2021-01-15T16:51:33.218 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.218 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:52.018041+0000", 2021-01-15T16:51:33.219 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.219 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.219 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.219 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.219 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.220 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.221 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.221 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.221 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.222 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.222 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.222 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:4909985f:::obj-more-13:head", 2021-01-15T16:51:33.222 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'154", 2021-01-15T16:51:33.222 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4652.0:1", 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:52.295218+0000", 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.223 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.224 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.224 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.224 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.224 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.224 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.225 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.225 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.225 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.225 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.225 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.226 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:acf906ce:::obj-more-14:head", 2021-01-15T16:51:33.227 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'155", 2021-01-15T16:51:33.227 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.227 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4655.0:1", 2021-01-15T16:51:33.227 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.227 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:52.513452+0000", 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.228 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.229 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.229 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.229 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.229 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.230 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.230 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.230 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.230 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.230 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.231 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.231 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.231 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:44d31ae5:::obj-more-15:head", 2021-01-15T16:51:33.231 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'156", 2021-01-15T16:51:33.231 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4658.0:1", 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:52.827212+0000", 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.232 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.233 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.233 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.233 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.233 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.234 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.234 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.234 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.234 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.234 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.235 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.235 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.235 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.236 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:54dfd3e0:::obj-more-16:head", 2021-01-15T16:51:33.236 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'157", 2021-01-15T16:51:33.236 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.236 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4661.0:1", 2021-01-15T16:51:33.236 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.237 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:53.059040+0000", 2021-01-15T16:51:33.237 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.237 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.237 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.237 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.238 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.238 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.238 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.238 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.238 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.239 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.239 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.239 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.239 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.239 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6a7dcd50:::obj-more-17:head", 2021-01-15T16:51:33.240 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'158", 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4664.0:1", 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:53.277106+0000", 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.241 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.242 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.242 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.242 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.242 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.243 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.243 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.243 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.243 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.243 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.244 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:fb20876f:::obj-more-18:head", 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'159", 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.245 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4667.0:1", 2021-01-15T16:51:33.246 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.246 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:53.530528+0000", 2021-01-15T16:51:33.246 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.246 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.246 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.247 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.247 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.247 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.247 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.248 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.248 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.248 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.248 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.248 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.249 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:ed63ae36:::obj-more-19:head", 2021-01-15T16:51:33.250 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'160", 2021-01-15T16:51:33.250 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.250 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4670.0:1", 2021-01-15T16:51:33.250 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.251 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:53.754910+0000", 2021-01-15T16:51:33.257 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.257 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.257 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.257 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.257 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.258 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.258 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.258 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.258 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.258 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.259 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.260 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.260 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "modify", 2021-01-15T16:51:33.260 INFO:tasks.workunit.client.0.smithi145.stdout: "object": "1:6a8a392a:::obj-more-20:head", 2021-01-15T16:51:33.260 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "37'161", 2021-01-15T16:51:33.260 INFO:tasks.workunit.client.0.smithi145.stdout: "prior_version": "0'0", 2021-01-15T16:51:33.261 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4673.0:1", 2021-01-15T16:51:33.261 INFO:tasks.workunit.client.0.smithi145.stdout: "extra_reqids": [], 2021-01-15T16:51:33.261 INFO:tasks.workunit.client.0.smithi145.stdout: "mtime": "2021-01-15T16:50:53.974867+0000", 2021-01-15T16:51:33.261 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": 0, 2021-01-15T16:51:33.261 INFO:tasks.workunit.client.0.smithi145.stdout: "mod_desc": { 2021-01-15T16:51:33.262 INFO:tasks.workunit.client.0.smithi145.stdout: "object_mod_desc": { 2021-01-15T16:51:33.262 INFO:tasks.workunit.client.0.smithi145.stdout: "can_local_rollback": false, 2021-01-15T16:51:33.262 INFO:tasks.workunit.client.0.smithi145.stdout: "rollback_info_completed": false, 2021-01-15T16:51:33.262 INFO:tasks.workunit.client.0.smithi145.stdout: "ops": [] 2021-01-15T16:51:33.262 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.263 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.263 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_regions": { 2021-01-15T16:51:33.263 INFO:tasks.workunit.client.0.smithi145.stdout: "object_clean_regions": { 2021-01-15T16:51:33.263 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_offsets": "[8322~18446744073709543293]", 2021-01-15T16:51:33.263 INFO:tasks.workunit.client.0.smithi145.stdout: "clean_omap": true, 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: "new_object": false 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T16:51:33.264 INFO:tasks.workunit.client.0.smithi145.stdout: "dups": [ 2021-01-15T16:51:33.265 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.265 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4189.0:1", 2021-01-15T16:51:33.265 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "32'1", 2021-01-15T16:51:33.265 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "1", 2021-01-15T16:51:33.265 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.266 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.266 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.266 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4192.0:1", 2021-01-15T16:51:33.266 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "32'2", 2021-01-15T16:51:33.266 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "2", 2021-01-15T16:51:33.267 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.267 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.267 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.267 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4195.0:1", 2021-01-15T16:51:33.267 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'3", 2021-01-15T16:51:33.268 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "3", 2021-01-15T16:51:33.268 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.268 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.268 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.268 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4198.0:1", 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'4", 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "4", 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.269 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4201.0:1", 2021-01-15T16:51:33.270 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'5", 2021-01-15T16:51:33.270 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "5", 2021-01-15T16:51:33.270 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.270 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.271 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.271 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4204.0:1", 2021-01-15T16:51:33.271 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'6", 2021-01-15T16:51:33.271 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "6", 2021-01-15T16:51:33.271 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4207.0:1", 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'7", 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "7", 2021-01-15T16:51:33.272 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.273 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.273 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.273 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4210.0:1", 2021-01-15T16:51:33.273 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'8", 2021-01-15T16:51:33.273 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "8", 2021-01-15T16:51:33.274 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.274 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.274 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.274 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4213.0:1", 2021-01-15T16:51:33.274 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'9", 2021-01-15T16:51:33.275 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "9", 2021-01-15T16:51:33.275 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.275 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.275 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T16:51:33.275 INFO:tasks.workunit.client.0.smithi145.stdout: "reqid": "client.4216.0:1", 2021-01-15T16:51:33.276 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "33'10", 2021-01-15T16:51:33.276 INFO:tasks.workunit.client.0.smithi145.stdout: "user_version": "10", 2021-01-15T16:51:33.276 INFO:tasks.workunit.client.0.smithi145.stdout: "return_code": "0" 2021-01-15T16:51:33.276 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.276 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T16:51:33.277 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T16:51:33.277 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_missing_t": { 2021-01-15T16:51:33.277 INFO:tasks.workunit.client.0.smithi145.stdout: "missing": [], 2021-01-15T16:51:33.277 INFO:tasks.workunit.client.0.smithi145.stdout: "may_include_deletes": true 2021-01-15T16:51:33.277 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T16:51:33.278 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T16:51:33.278 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after v 0'0 2021-01-15T16:51:33.278 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'141 2021-01-15T16:51:33.278 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'140 2021-01-15T16:51:33.278 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'139 2021-01-15T16:51:33.279 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'138 2021-01-15T16:51:33.279 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'137 2021-01-15T16:51:33.279 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'136 2021-01-15T16:51:33.279 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'135 2021-01-15T16:51:33.279 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'134 2021-01-15T16:51:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'133 2021-01-15T16:51:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'132 2021-01-15T16:51:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'131 2021-01-15T16:51:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'130 2021-01-15T16:51:33.280 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'129 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'128 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'127 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'126 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'125 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'124 2021-01-15T16:51:33.281 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'123 2021-01-15T16:51:33.282 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'122 2021-01-15T16:51:33.282 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'121 2021-01-15T16:51:33.282 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'120 2021-01-15T16:51:33.282 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'119 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'118 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'117 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'116 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'115 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'114 2021-01-15T16:51:33.283 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'113 2021-01-15T16:51:33.284 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'112 2021-01-15T16:51:33.284 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'111 2021-01-15T16:51:33.284 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'110 2021-01-15T16:51:33.284 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'109 2021-01-15T16:51:33.284 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'108 2021-01-15T16:51:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'107 2021-01-15T16:51:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'106 2021-01-15T16:51:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'105 2021-01-15T16:51:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'104 2021-01-15T16:51:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'103 2021-01-15T16:51:33.286 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'102 2021-01-15T16:51:33.286 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'101 2021-01-15T16:51:33.286 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'100 2021-01-15T16:51:33.286 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'99 2021-01-15T16:51:33.286 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'98 2021-01-15T16:51:33.287 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'97 2021-01-15T16:51:33.287 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'96 2021-01-15T16:51:33.287 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'95 2021-01-15T16:51:33.287 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'94 2021-01-15T16:51:33.287 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'93 2021-01-15T16:51:33.288 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'92 2021-01-15T16:51:33.288 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'91 2021-01-15T16:51:33.288 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'90 2021-01-15T16:51:33.288 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'89 2021-01-15T16:51:33.288 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'88 2021-01-15T16:51:33.289 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'87 2021-01-15T16:51:33.289 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'86 2021-01-15T16:51:33.289 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'85 2021-01-15T16:51:33.289 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'84 2021-01-15T16:51:33.289 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'83 2021-01-15T16:51:33.290 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'82 2021-01-15T16:51:33.290 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'81 2021-01-15T16:51:33.290 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'80 2021-01-15T16:51:33.290 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'79 2021-01-15T16:51:33.290 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'78 2021-01-15T16:51:33.291 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'77 2021-01-15T16:51:33.291 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'76 2021-01-15T16:51:33.291 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'75 2021-01-15T16:51:33.291 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'74 2021-01-15T16:51:33.291 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'73 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'72 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'71 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'70 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'69 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'68 2021-01-15T16:51:33.292 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'67 2021-01-15T16:51:33.293 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'66 2021-01-15T16:51:33.293 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'65 2021-01-15T16:51:33.293 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'64 2021-01-15T16:51:33.293 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'63 2021-01-15T16:51:33.293 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'62 2021-01-15T16:51:33.294 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'61 2021-01-15T16:51:33.294 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'60 2021-01-15T16:51:33.294 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'59 2021-01-15T16:51:33.294 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'58 2021-01-15T16:51:33.294 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'57 2021-01-15T16:51:33.295 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'56 2021-01-15T16:51:33.295 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'55 2021-01-15T16:51:33.295 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'54 2021-01-15T16:51:33.295 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'53 2021-01-15T16:51:33.295 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'52 2021-01-15T16:51:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'51 2021-01-15T16:51:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'50 2021-01-15T16:51:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'49 2021-01-15T16:51:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'48 2021-01-15T16:51:33.296 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'47 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'46 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'45 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'44 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'43 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'42 2021-01-15T16:51:33.297 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'41 2021-01-15T16:51:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'40 2021-01-15T16:51:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'39 2021-01-15T16:51:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'38 2021-01-15T16:51:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'37 2021-01-15T16:51:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'36 2021-01-15T16:51:33.299 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'35 2021-01-15T16:51:33.299 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'34 2021-01-15T16:51:33.299 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'33 2021-01-15T16:51:33.299 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'32 2021-01-15T16:51:33.299 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'31 2021-01-15T16:51:33.300 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'30 2021-01-15T16:51:33.300 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'29 2021-01-15T16:51:33.300 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'28 2021-01-15T16:51:33.300 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'27 2021-01-15T16:51:33.300 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'26 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'25 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'24 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'23 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'22 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'21 2021-01-15T16:51:33.301 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'20 2021-01-15T16:51:33.302 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'19 2021-01-15T16:51:33.302 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'18 2021-01-15T16:51:33.302 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'17 2021-01-15T16:51:33.302 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'16 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'15 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'14 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'13 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'12 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'11 2021-01-15T16:51:33.303 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'10 2021-01-15T16:51:33.304 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'9 2021-01-15T16:51:33.304 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'8 2021-01-15T16:51:33.304 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'7 2021-01-15T16:51:33.304 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'6 2021-01-15T16:51:33.304 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'5 2021-01-15T16:51:33.305 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'4 2021-01-15T16:51:33.305 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'3 2021-01-15T16:51:33.305 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'2 2021-01-15T16:51:33.305 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'1 2021-01-15T16:51:33.305 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:51:33.306 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after v 0'0 2021-01-15T16:51:33.306 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'141 2021-01-15T16:51:33.306 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'140 2021-01-15T16:51:33.306 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'139 2021-01-15T16:51:33.306 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'138 2021-01-15T16:51:33.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'137 2021-01-15T16:51:33.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'136 2021-01-15T16:51:33.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'135 2021-01-15T16:51:33.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'134 2021-01-15T16:51:33.307 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'133 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'132 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'131 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'130 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'129 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'128 2021-01-15T16:51:33.308 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'127 2021-01-15T16:51:33.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'126 2021-01-15T16:51:33.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'125 2021-01-15T16:51:33.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'124 2021-01-15T16:51:33.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'123 2021-01-15T16:51:33.309 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'122 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'121 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'120 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'119 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'118 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'117 2021-01-15T16:51:33.310 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'116 2021-01-15T16:51:33.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'115 2021-01-15T16:51:33.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'114 2021-01-15T16:51:33.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'113 2021-01-15T16:51:33.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'112 2021-01-15T16:51:33.311 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'111 2021-01-15T16:51:33.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'110 2021-01-15T16:51:33.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'109 2021-01-15T16:51:33.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'108 2021-01-15T16:51:33.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'107 2021-01-15T16:51:33.312 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'106 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'105 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'104 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'103 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'102 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'101 2021-01-15T16:51:33.313 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'100 2021-01-15T16:51:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'99 2021-01-15T16:51:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'98 2021-01-15T16:51:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'97 2021-01-15T16:51:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'96 2021-01-15T16:51:33.314 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'95 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'94 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'93 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'92 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'91 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'90 2021-01-15T16:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'89 2021-01-15T16:51:33.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'88 2021-01-15T16:51:33.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'87 2021-01-15T16:51:33.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'86 2021-01-15T16:51:33.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'85 2021-01-15T16:51:33.316 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'84 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'83 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'82 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'81 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'80 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'79 2021-01-15T16:51:33.317 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'78 2021-01-15T16:51:33.318 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'77 2021-01-15T16:51:33.318 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'76 2021-01-15T16:51:33.318 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'75 2021-01-15T16:51:33.318 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'74 2021-01-15T16:51:33.319 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'73 2021-01-15T16:51:33.319 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'72 2021-01-15T16:51:33.319 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'71 2021-01-15T16:51:33.319 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'70 2021-01-15T16:51:33.319 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'69 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'68 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'67 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'66 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'65 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'64 2021-01-15T16:51:33.320 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'63 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'62 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'61 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'60 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'59 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'58 2021-01-15T16:51:33.321 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'57 2021-01-15T16:51:33.322 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'56 2021-01-15T16:51:33.322 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'55 2021-01-15T16:51:33.322 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'54 2021-01-15T16:51:33.322 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'53 2021-01-15T16:51:33.323 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'52 2021-01-15T16:51:33.323 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'51 2021-01-15T16:51:33.323 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'50 2021-01-15T16:51:33.323 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'49 2021-01-15T16:51:33.323 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'48 2021-01-15T16:51:33.324 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'47 2021-01-15T16:51:33.324 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'46 2021-01-15T16:51:33.324 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'45 2021-01-15T16:51:33.324 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'44 2021-01-15T16:51:33.324 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'43 2021-01-15T16:51:33.325 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'42 2021-01-15T16:51:33.325 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'41 2021-01-15T16:51:33.325 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'40 2021-01-15T16:51:33.325 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'39 2021-01-15T16:51:33.325 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'38 2021-01-15T16:51:33.326 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'37 2021-01-15T16:51:33.326 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'36 2021-01-15T16:51:33.326 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'35 2021-01-15T16:51:33.326 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'34 2021-01-15T16:51:33.327 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'33 2021-01-15T16:51:33.327 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'32 2021-01-15T16:51:33.327 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'31 2021-01-15T16:51:33.327 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'30 2021-01-15T16:51:33.327 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'29 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'28 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'27 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'26 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'25 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'24 2021-01-15T16:51:33.328 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'23 2021-01-15T16:51:33.329 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'22 2021-01-15T16:51:33.329 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'21 2021-01-15T16:51:33.329 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'20 2021-01-15T16:51:33.329 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'19 2021-01-15T16:51:33.329 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'18 2021-01-15T16:51:33.330 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'17 2021-01-15T16:51:33.330 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'16 2021-01-15T16:51:33.330 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'15 2021-01-15T16:51:33.330 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'14 2021-01-15T16:51:33.330 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'13 2021-01-15T16:51:33.331 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'12 2021-01-15T16:51:33.331 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'11 2021-01-15T16:51:33.331 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'10 2021-01-15T16:51:33.331 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'9 2021-01-15T16:51:33.331 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'8 2021-01-15T16:51:33.332 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'7 2021-01-15T16:51:33.332 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'6 2021-01-15T16:51:33.332 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'5 2021-01-15T16:51:33.332 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'4 2021-01-15T16:51:33.332 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'3 2021-01-15T16:51:33.333 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'2 2021-01-15T16:51:33.333 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'1 2021-01-15T16:51:33.333 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:51:33.333 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after v 0'0 2021-01-15T16:51:33.333 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'141 2021-01-15T16:51:33.334 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'140 2021-01-15T16:51:33.334 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'139 2021-01-15T16:51:33.334 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'138 2021-01-15T16:51:33.334 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'137 2021-01-15T16:51:33.334 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'136 2021-01-15T16:51:33.335 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'135 2021-01-15T16:51:33.335 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'134 2021-01-15T16:51:33.335 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'133 2021-01-15T16:51:33.335 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'132 2021-01-15T16:51:33.335 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'131 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'130 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'129 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'128 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'127 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'126 2021-01-15T16:51:33.336 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'125 2021-01-15T16:51:33.337 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'124 2021-01-15T16:51:33.337 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'123 2021-01-15T16:51:33.337 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'122 2021-01-15T16:51:33.337 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'121 2021-01-15T16:51:33.337 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'120 2021-01-15T16:51:33.338 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'119 2021-01-15T16:51:33.338 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'118 2021-01-15T16:51:33.338 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'117 2021-01-15T16:51:33.338 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'116 2021-01-15T16:51:33.338 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'115 2021-01-15T16:51:33.339 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'114 2021-01-15T16:51:33.339 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'113 2021-01-15T16:51:33.339 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'112 2021-01-15T16:51:33.339 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'111 2021-01-15T16:51:33.339 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'110 2021-01-15T16:51:33.340 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'109 2021-01-15T16:51:33.340 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'108 2021-01-15T16:51:33.340 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'107 2021-01-15T16:51:33.340 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'106 2021-01-15T16:51:33.340 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'105 2021-01-15T16:51:33.341 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'104 2021-01-15T16:51:33.341 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'103 2021-01-15T16:51:33.341 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'102 2021-01-15T16:51:33.341 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'101 2021-01-15T16:51:33.341 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'100 2021-01-15T16:51:33.342 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'99 2021-01-15T16:51:33.342 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'98 2021-01-15T16:51:33.342 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'97 2021-01-15T16:51:33.342 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'96 2021-01-15T16:51:33.342 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'95 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'94 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'93 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'92 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'91 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'90 2021-01-15T16:51:33.343 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'89 2021-01-15T16:51:33.344 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'88 2021-01-15T16:51:33.344 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'87 2021-01-15T16:51:33.344 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'86 2021-01-15T16:51:33.344 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'85 2021-01-15T16:51:33.344 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'84 2021-01-15T16:51:33.345 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'83 2021-01-15T16:51:33.345 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'82 2021-01-15T16:51:33.345 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'81 2021-01-15T16:51:33.345 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'80 2021-01-15T16:51:33.345 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'79 2021-01-15T16:51:33.346 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'78 2021-01-15T16:51:33.346 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'77 2021-01-15T16:51:33.346 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'76 2021-01-15T16:51:33.346 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'75 2021-01-15T16:51:33.346 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'74 2021-01-15T16:51:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'73 2021-01-15T16:51:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'72 2021-01-15T16:51:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'71 2021-01-15T16:51:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'70 2021-01-15T16:51:33.347 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'69 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'68 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'67 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'66 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'65 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'64 2021-01-15T16:51:33.348 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'63 2021-01-15T16:51:33.349 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'62 2021-01-15T16:51:33.349 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'61 2021-01-15T16:51:33.349 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'60 2021-01-15T16:51:33.349 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'59 2021-01-15T16:51:33.349 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'58 2021-01-15T16:51:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'57 2021-01-15T16:51:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'56 2021-01-15T16:51:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'55 2021-01-15T16:51:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'54 2021-01-15T16:51:33.350 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'53 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'52 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'51 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'50 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'49 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'48 2021-01-15T16:51:33.351 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'47 2021-01-15T16:51:33.352 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'46 2021-01-15T16:51:33.352 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'45 2021-01-15T16:51:33.352 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'44 2021-01-15T16:51:33.352 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'43 2021-01-15T16:51:33.352 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'42 2021-01-15T16:51:33.353 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'41 2021-01-15T16:51:33.353 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'40 2021-01-15T16:51:33.353 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'39 2021-01-15T16:51:33.353 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'38 2021-01-15T16:51:33.353 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'37 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'36 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'35 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'34 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'33 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'32 2021-01-15T16:51:33.354 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'31 2021-01-15T16:51:33.355 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'30 2021-01-15T16:51:33.355 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'29 2021-01-15T16:51:33.355 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'28 2021-01-15T16:51:33.355 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'27 2021-01-15T16:51:33.355 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'26 2021-01-15T16:51:33.356 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'25 2021-01-15T16:51:33.356 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'24 2021-01-15T16:51:33.356 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'23 2021-01-15T16:51:33.356 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'22 2021-01-15T16:51:33.356 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'21 2021-01-15T16:51:33.357 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'20 2021-01-15T16:51:33.357 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'19 2021-01-15T16:51:33.357 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'18 2021-01-15T16:51:33.357 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'17 2021-01-15T16:51:33.357 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'16 2021-01-15T16:51:33.358 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'15 2021-01-15T16:51:33.358 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'14 2021-01-15T16:51:33.358 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'13 2021-01-15T16:51:33.358 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'12 2021-01-15T16:51:33.358 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'11 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'10 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'9 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'8 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'7 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'6 2021-01-15T16:51:33.359 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'5 2021-01-15T16:51:33.360 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'4 2021-01-15T16:51:33.360 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 33'3 2021-01-15T16:51:33.360 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'2 2021-01-15T16:51:33.360 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_after copy log version 32'1 2021-01-15T16:51:33.360 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:50:48.890+0000 7f7d273ff700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:51:33.361 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_after v 37'161 2021-01-15T16:51:33.361 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after earliest_dup_version 0 2021-01-15T16:51:33.361 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 32'1 2021-01-15T16:51:33.361 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 32'2 2021-01-15T16:51:33.361 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'3 2021-01-15T16:51:33.362 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'4 2021-01-15T16:51:33.362 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'5 2021-01-15T16:51:33.362 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'6 2021-01-15T16:51:33.362 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'7 2021-01-15T16:51:33.362 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'8 2021-01-15T16:51:33.363 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'9 2021-01-15T16:51:33.363 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup version 33'10 2021-01-15T16:51:33.363 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'11 2021-01-15T16:51:33.363 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'12 2021-01-15T16:51:33.363 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'13 2021-01-15T16:51:33.364 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'14 2021-01-15T16:51:33.364 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'15 2021-01-15T16:51:33.364 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'16 2021-01-15T16:51:33.364 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'17 2021-01-15T16:51:33.364 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'18 2021-01-15T16:51:33.365 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'19 2021-01-15T16:51:33.365 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'20 2021-01-15T16:51:33.365 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'21 2021-01-15T16:51:33.365 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'22 2021-01-15T16:51:33.365 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'23 2021-01-15T16:51:33.366 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'24 2021-01-15T16:51:33.366 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'25 2021-01-15T16:51:33.366 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'26 2021-01-15T16:51:33.366 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'27 2021-01-15T16:51:33.366 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'28 2021-01-15T16:51:33.367 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'29 2021-01-15T16:51:33.367 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'30 2021-01-15T16:51:33.367 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'31 2021-01-15T16:51:33.367 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'32 2021-01-15T16:51:33.367 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'33 2021-01-15T16:51:33.368 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'34 2021-01-15T16:51:33.368 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'35 2021-01-15T16:51:33.368 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'36 2021-01-15T16:51:33.368 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'37 2021-01-15T16:51:33.368 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'38 2021-01-15T16:51:33.369 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'39 2021-01-15T16:51:33.369 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'40 2021-01-15T16:51:33.369 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'41 2021-01-15T16:51:33.369 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'42 2021-01-15T16:51:33.369 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'43 2021-01-15T16:51:33.370 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'44 2021-01-15T16:51:33.370 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'45 2021-01-15T16:51:33.370 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'46 2021-01-15T16:51:33.370 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'47 2021-01-15T16:51:33.370 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'48 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'49 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'50 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'51 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'52 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'53 2021-01-15T16:51:33.371 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'54 2021-01-15T16:51:33.372 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'55 2021-01-15T16:51:33.372 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'56 2021-01-15T16:51:33.372 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'57 2021-01-15T16:51:33.372 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'58 2021-01-15T16:51:33.372 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'59 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'60 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'61 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'62 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'63 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'64 2021-01-15T16:51:33.373 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'65 2021-01-15T16:51:33.374 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'66 2021-01-15T16:51:33.374 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'67 2021-01-15T16:51:33.374 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'68 2021-01-15T16:51:33.374 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'69 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'70 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'71 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'72 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'73 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'74 2021-01-15T16:51:33.375 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'75 2021-01-15T16:51:33.376 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'76 2021-01-15T16:51:33.376 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'77 2021-01-15T16:51:33.376 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'78 2021-01-15T16:51:33.376 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'79 2021-01-15T16:51:33.376 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'80 2021-01-15T16:51:33.377 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'81 2021-01-15T16:51:33.377 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'82 2021-01-15T16:51:33.377 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'83 2021-01-15T16:51:33.377 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'84 2021-01-15T16:51:33.377 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'85 2021-01-15T16:51:33.378 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'86 2021-01-15T16:51:33.378 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'87 2021-01-15T16:51:33.378 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'88 2021-01-15T16:51:33.378 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'89 2021-01-15T16:51:33.378 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'90 2021-01-15T16:51:33.379 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'91 2021-01-15T16:51:33.379 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'92 2021-01-15T16:51:33.379 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'93 2021-01-15T16:51:33.379 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'94 2021-01-15T16:51:33.379 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'95 2021-01-15T16:51:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'96 2021-01-15T16:51:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'97 2021-01-15T16:51:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'98 2021-01-15T16:51:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'99 2021-01-15T16:51:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'100 2021-01-15T16:51:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'101 2021-01-15T16:51:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'102 2021-01-15T16:51:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'103 2021-01-15T16:51:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'104 2021-01-15T16:51:33.381 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'105 2021-01-15T16:51:33.382 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'106 2021-01-15T16:51:33.382 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'107 2021-01-15T16:51:33.382 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'108 2021-01-15T16:51:33.382 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'109 2021-01-15T16:51:33.382 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'110 2021-01-15T16:51:33.383 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'111 2021-01-15T16:51:33.383 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'112 2021-01-15T16:51:33.383 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'113 2021-01-15T16:51:33.383 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'114 2021-01-15T16:51:33.383 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'115 2021-01-15T16:51:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'116 2021-01-15T16:51:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'117 2021-01-15T16:51:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'118 2021-01-15T16:51:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'119 2021-01-15T16:51:33.384 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'120 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'121 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'122 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'123 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'124 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'125 2021-01-15T16:51:33.385 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'126 2021-01-15T16:51:33.386 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'127 2021-01-15T16:51:33.386 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'128 2021-01-15T16:51:33.386 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'129 2021-01-15T16:51:33.386 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'130 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'131 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'132 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'133 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'134 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'135 2021-01-15T16:51:33.387 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'136 2021-01-15T16:51:33.388 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'137 2021-01-15T16:51:33.388 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'138 2021-01-15T16:51:33.388 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'139 2021-01-15T16:51:33.388 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'140 2021-01-15T16:51:33.388 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 33'141 2021-01-15T16:51:33.389 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'142 2021-01-15T16:51:33.389 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'143 2021-01-15T16:51:33.389 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'144 2021-01-15T16:51:33.389 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'145 2021-01-15T16:51:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'146 2021-01-15T16:51:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'147 2021-01-15T16:51:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'148 2021-01-15T16:51:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'149 2021-01-15T16:51:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'150 2021-01-15T16:51:33.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'151 2021-01-15T16:51:33.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'152 2021-01-15T16:51:33.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'153 2021-01-15T16:51:33.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'154 2021-01-15T16:51:33.391 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'155 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'156 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'157 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'158 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'159 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'160 2021-01-15T16:51:33.392 INFO:tasks.workunit.client.0.smithi145.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-01-15T16:51:21.374+0000 7f7d273ff700 20 copy_up_to/copy_after copy dup from log version 37'161 2021-01-15T16:51:33.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:51:33.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=151 2021-01-15T16:51:33.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 151 '!=' 151 ']' 2021-01-15T16:51:33.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-01-15T16:51:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=10 2021-01-15T16:51:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 10 '!=' 10 ']' 2021-01-15T16:51:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-01-15T16:51:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-01-15T16:51:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-01-15T16:51:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-01-15T16:51:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:51:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:51:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:51:33.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:51:33.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:51:33.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:51:33.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:51:33.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:51:33.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:51:33.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:51:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:51:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:51:33.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:51:33.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:51:33.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:51:33.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:51:33.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:33.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:51:33.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:51:33.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:51:33.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:51:33.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T16:51:33.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-recovery-log 0 2021-01-15T16:51:33.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-01-15T16:51:33.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T16:51:33.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-01-15T16:51:33.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:51:33.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:51:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:51:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:51:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:51:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:51:33.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:51:33.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:51:33.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:51:33.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:51:33.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:51:33.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:51:33.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:51:33.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:51:33.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:51:33.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T16:51:33.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-01-15T16:51:33.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:51:33.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:33.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.119796 2021-01-15T16:51:33.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.119796 2021-01-15T16:51:33.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:51:33.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:51:33.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T16:51:33.407 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T16:51:33.407 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T16:51:33.446 INFO:tasks.workunit:Running workunit osd/osd-backfill-space.sh... 2021-01-15T16:51:33.447 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh 2021-01-15T16:51:33.508 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T16:51:33.519 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T16:51:33.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:51:33.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T16:51:33.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T16:51:33.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T16:51:33.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T16:51:33.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T16:51:33.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T16:51:33.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T16:51:33.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-space 2021-01-15T16:51:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:21: run: local dir=td/osd-backfill-space 2021-01-15T16:51:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:22: run: shift 2021-01-15T16:51:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:24: run: export CEPH_MON=127.0.0.1:7180 2021-01-15T16:51:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:24: run: CEPH_MON=127.0.0.1:7180 2021-01-15T16:51:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:25: run: export CEPH_ARGS 2021-01-15T16:51:33.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:26: run: uuidgen 2021-01-15T16:51:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:26: run: CEPH_ARGS+='--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none ' 2021-01-15T16:51:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7180 ' 2021-01-15T16:51:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:28: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T16:51:33.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:29: run: CEPH_ARGS+='--fake_statfs_for_testing=3686400 ' 2021-01-15T16:51:33.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:30: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2021-01-15T16:51:33.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:31: run: export objects=600 2021-01-15T16:51:33.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:31: run: objects=600 2021-01-15T16:51:33.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:32: run: export poolprefix=test 2021-01-15T16:51:33.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:32: run: poolprefix=test 2021-01-15T16:51:33.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: set 2021-01-15T16:51:33.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T16:51:33.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:34: run: local 'funcs=TEST_backfill_grow 2021-01-15T16:51:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_multi_partial 2021-01-15T16:51:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_test_multi 2021-01-15T16:51:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_test_sametarget 2021-01-15T16:51:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_test_simple 2021-01-15T16:51:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_ec_backfill_grow 2021-01-15T16:51:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_ec_backfill_multi 2021-01-15T16:51:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_ec_backfill_simple' 2021-01-15T16:51:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T16:51:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T16:51:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T16:51:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T16:51:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T16:51:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:51:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T16:51:33.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:51:33.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:51:33.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:51:33.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:51:33.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:51:33.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:51:33.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:51:33.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:51:33.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:51:33.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:51:33.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:51:33.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:51:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:51:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:51:33.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:51:33.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:51:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:51:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T16:51:33.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:51:33.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:33.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:33.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T16:51:33.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:51:33.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:51:33.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T16:51:33.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:51:33.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:33.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:33.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T16:51:33.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:51:33.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T16:51:33.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T16:51:33.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:51:33.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T16:51:33.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_grow td/osd-backfill-space 2021-01-15T16:51:33.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:557: TEST_backfill_grow: local dir=td/osd-backfill-space 2021-01-15T16:51:33.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:558: TEST_backfill_grow: local poolname=test 2021-01-15T16:51:33.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:559: TEST_backfill_grow: local OSDS=3 2021-01-15T16:51:33.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:561: TEST_backfill_grow: run_mon td/osd-backfill-space a 2021-01-15T16:51:33.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T16:51:33.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:51:33.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:51:33.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:51:33.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T16:51:33.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T16:51:33.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:51:33.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:51:33.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:51:33.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:51:33.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:33.857 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:33.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:33.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:51:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:51:34.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:51:34.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:51:34.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:51:34.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:51:34.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:51:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:51:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:51:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:51:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:51:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:34.044 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:34.044 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T16:51:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:51:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T16:51:34.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:51:34.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:51:34.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:51:34.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:51:34.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:51:34.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:51:34.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:51:34.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:51:34.147 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:51:34.147 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:34.148 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:34.148 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T16:51:34.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:51:34.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T16:51:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:562: TEST_backfill_grow: run_mgr td/osd-backfill-space x 2021-01-15T16:51:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T16:51:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:51:34.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:51:34.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:51:34.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T16:51:34.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:51:35.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:51:35.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:51:35.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:51:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:51:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:35.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:35.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:51:35.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:51:35.175 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: expr 3 - 1 2021-01-15T16:51:35.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: seq 0 2 2021-01-15T16:51:35.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:51:35.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 0 2021-01-15T16:51:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:51:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:51:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:51:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:51:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T16:51:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:51:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:51:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:51:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:51:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T16:51:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T16:51:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:51:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:51:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:51:35.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:51:35.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:51:35.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:51:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:51:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:35.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:51:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:51:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:51:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:51:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:51:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:51:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:51:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:51:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:51:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T16:51:35.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:51:35.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 2021-01-15T16:51:35.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9' 2021-01-15T16:51:35.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:51:35.188 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 2021-01-15T16:51:35.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAXyAFghLG9CxAASAx66tEw+u61SVJn6Mpzjg== 2021-01-15T16:51:35.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAXyAFghLG9CxAASAx66tEw+u61SVJn6Mpzjg=="}' 2021-01-15T16:51:35.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 -i td/osd-backfill-space/0/new.json 2021-01-15T16:51:35.541 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:51:35.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T16:51:35.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAXyAFghLG9CxAASAx66tEw+u61SVJn6Mpzjg== --osd-uuid ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 2021-01-15T16:51:35.580 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.579+0000 7f4198ec2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:35.582 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.581+0000 7f4198ec2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:35.583 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.583+0000 7f4198ec2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:35.584 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.584+0000 7f4198ec2080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:51:35.584 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.584+0000 7f4198ec2080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:51:35.585 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.584+0000 7f4198ec2080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:51:35.672 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:35.672+0000 7f4198ec2080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T16:51:40.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T16:51:40.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:51:40.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:51:40.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:51:40.273 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:51:40.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:51:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:51:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:51:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:51:40.796 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:51:40.819 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:40.819+0000 7f9c5d29f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:40.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:40.846+0000 7f9c5d29f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:40.849 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:40.848+0000 7f9c5d29f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:41.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:51:41.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:51:41.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:51:41.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:51:41.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:51:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:51:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:51:41.220 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:51:41.368 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:41.368+0000 7f9c5d29f080 -1 Falling back to public interface 2021-01-15T16:51:41.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:51:42.190 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:42.189+0000 7f9c5d29f080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:51:42.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:51:42.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:42.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:51:42.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:42.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:51:42.613 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:51:43.011 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/174059,v1:127.0.0.1:6803/174059] [v2:127.0.0.1:6804/174059,v1:127.0.0.1:6805/174059] exists,up ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 2021-01-15T16:51:43.012 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 ec2b626a-fb99-4d37-aea9-b62ecfc21c11 2021-01-15T16:51:43.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:51:43.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:51:43.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:51:43.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:51:43.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 1 2021-01-15T16:51:43.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:51:43.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:51:43.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:51:43.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:51:43.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T16:51:43.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:51:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:51:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:51:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:51:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T16:51:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T16:51:43.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:51:43.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:51:43.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:51:43.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:51:43.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:51:43.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:51:43.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:51:43.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:43.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:43.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:51:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:51:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:51:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:51:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:51:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:51:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:51:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:43.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:51:43.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:51:43.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T16:51:43.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:51:43.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ec2b626a-fb99-4d37-aea9-b62ecfc21c11 2021-01-15T16:51:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ec2b626a-fb99-4d37-aea9-b62ecfc21c11' 2021-01-15T16:51:43.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:51:43.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAfyAFg/+WvARAAW3SSxeaqXkJPUUtsMf/xrA== 2021-01-15T16:51:43.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAfyAFg/+WvARAAW3SSxeaqXkJPUUtsMf/xrA=="}' 2021-01-15T16:51:43.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ec2b626a-fb99-4d37-aea9-b62ecfc21c11 -i td/osd-backfill-space/1/new.json 2021-01-15T16:51:43.599 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:51:43.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T16:51:43.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAfyAFg/+WvARAAW3SSxeaqXkJPUUtsMf/xrA== --osd-uuid ec2b626a-fb99-4d37-aea9-b62ecfc21c11 2021-01-15T16:51:43.638 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.637+0000 7fd18b384080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:43.639 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.639+0000 7fd18b384080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:43.641 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.641+0000 7fd18b384080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:43.642 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.641+0000 7fd18b384080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:51:43.642 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.641+0000 7fd18b384080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:51:43.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.641+0000 7fd18b384080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:51:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:43.686+0000 7fd18b384080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T16:51:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T16:51:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:51:48.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:51:48.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:51:48.353 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:51:48.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:51:48.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:51:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:51:48.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:51:48.886 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:51:48.910 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:48.909+0000 7f5a17ed8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:48.934+0000 7f5a17ed8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:48.935 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:48.935+0000 7f5a17ed8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:49.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:51:49.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:51:49.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:51:49.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:51:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:51:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:51:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:49.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:51:49.308 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:51:49.451 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:49.451+0000 7f5a17ed8080 -1 Falling back to public interface 2021-01-15T16:51:49.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:51:50.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:50.237+0000 7f5a17ed8080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:51:50.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:51:50.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:50.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:51:50.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:50.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:51:50.703 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:51:51.146 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/175323,v1:127.0.0.1:6811/175323] [v2:127.0.0.1:6812/175323,v1:127.0.0.1:6813/175323] exists,up ec2b626a-fb99-4d37-aea9-b62ecfc21c11 2021-01-15T16:51:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:51:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:51:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:51:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:564: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:51:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:566: TEST_backfill_grow: run_osd td/osd-backfill-space 2 2021-01-15T16:51:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:51:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:51:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:51:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:51:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T16:51:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:51:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:51:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:51:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:51:51.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T16:51:51.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T16:51:51.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:51:51.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:51:51.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:51:51.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:51:51.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:51:51.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:51:51.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:51:51.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:51:51.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:51:51.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:51.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:51:51.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:51:51.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:51:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:51:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:51:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:51:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:51:51.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:51:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:51:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:51:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T16:51:51.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:51:51.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8af490b1-6d90-4187-a4a9-b6f73b41c329 2021-01-15T16:51:51.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8af490b1-6d90-4187-a4a9-b6f73b41c329' 2021-01-15T16:51:51.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:51:51.156 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 8af490b1-6d90-4187-a4a9-b6f73b41c329 2021-01-15T16:51:51.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAnyAFgDizwCRAAw/K8LG2zVoNvFnRNl4ZoEQ== 2021-01-15T16:51:51.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAnyAFgDizwCRAAw/K8LG2zVoNvFnRNl4ZoEQ=="}' 2021-01-15T16:51:51.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8af490b1-6d90-4187-a4a9-b6f73b41c329 -i td/osd-backfill-space/2/new.json 2021-01-15T16:51:51.690 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:51:51.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T16:51:51.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAnyAFgDizwCRAAw/K8LG2zVoNvFnRNl4ZoEQ== --osd-uuid 8af490b1-6d90-4187-a4a9-b6f73b41c329 2021-01-15T16:51:51.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.728+0000 7fe94c94c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:51.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.730+0000 7fe94c94c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:51.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.731+0000 7fe94c94c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:51.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.731+0000 7fe94c94c080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:51:51.732 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.731+0000 7fe94c94c080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:51:51.732 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.731+0000 7fe94c94c080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:51:51.776 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:51.776+0000 7fe94c94c080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T16:51:56.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T16:51:56.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:51:56.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:51:56.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:51:56.510 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:51:57.037 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:51:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:51:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:51:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:51:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:51:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:51:57.062 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:57.062+0000 7f8df2ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:57.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:57.064+0000 7f8df2ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:57.065 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:57.065+0000 7f8df2ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:51:57.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:51:57.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:51:57.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:51:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:51:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:51:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:51:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:57.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:51:57.481 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:51:57.581 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:57.580+0000 7f8df2ba8080 -1 Falling back to public interface 2021-01-15T16:51:57.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:51:58.369 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:51:58.369+0000 7f8df2ba8080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:51:58.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:51:58.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:51:58.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:51:58.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:51:58.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:51:58.883 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:51:59.343 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/176587,v1:127.0.0.1:6819/176587] [v2:127.0.0.1:6820/176587,v1:127.0.0.1:6821/176587] exists,up 8af490b1-6d90-4187-a4a9-b6f73b41c329 2021-01-15T16:51:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:51:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:51:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:51:59.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:569: TEST_backfill_grow: ceph osd set-backfillfull-ratio .85 2021-01-15T16:52:00.044 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T16:52:00.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:571: TEST_backfill_grow: create_pool test 1 1 2021-01-15T16:52:00.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T16:52:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T16:52:00.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:52:01.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:572: TEST_backfill_grow: ceph osd pool set test size 3 2021-01-15T16:52:02.425 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T16:52:02.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:573: TEST_backfill_grow: sleep 5 2021-01-15T16:52:07.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:575: TEST_backfill_grow: wait_for_clean 2021-01-15T16:52:07.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:52:07.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:52:07.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:52:07.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:52:07.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:52:07.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:52:07.440 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:52:07.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:52:07.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:52:07.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:52:07.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:52:07.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:52:07.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:52:07.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:52:07.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:52:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:52:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:52:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:52:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:52:07.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:52:07.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:52:08.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T16:52:08.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T16:52:08.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T16:52:08.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:52:08.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:52:08.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672966 2021-01-15T16:52:08.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672966 2021-01-15T16:52:08.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672966' 2021-01-15T16:52:08.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:52:08.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:52:08.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:52:08.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:52:08.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672966 2-64424509444' 2021-01-15T16:52:08.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:52:08.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T16:52:08.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:52:08.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:52:08.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T16:52:08.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:52:08.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T16:52:08.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T16:52:08.368 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T16:52:08.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:52:08.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2021-01-15T16:52:08.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:52:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:52:09.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:52:10.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836487 2021-01-15T16:52:10.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:52:10.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2021-01-15T16:52:10.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:52:10.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:52:10.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-01-15T16:52:10.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:52:10.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672966 2021-01-15T16:52:10.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-01-15T16:52:10.166 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672966 2021-01-15T16:52:10.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:52:10.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672966 2021-01-15T16:52:10.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:52:10.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:52:10.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:52:10.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:52:10.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:52:10.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:52:10.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:52:10.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:52:10.567 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:52:10.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:52:10.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T16:52:10.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:52:10.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:52:10.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:52:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:52:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:52:11.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:52:11.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:52:11.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:52:11.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:52:11.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:52:11.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:52:11.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:52:11.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:52:11.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:52:11.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:52:12.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:52:12.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:52:12.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:52:12.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:577: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/4kdata bs=1k count=4 2021-01-15T16:52:12.391 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T16:52:12.391 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T16:52:12.391 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.1291e-05 s, 66.8 MB/s 2021-01-15T16:52:12.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: seq 1 600 2021-01-15T16:52:12.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:12.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj1 td/osd-backfill-space/4kdata 2021-01-15T16:52:12.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:12.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj2 td/osd-backfill-space/4kdata 2021-01-15T16:52:12.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:12.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj3 td/osd-backfill-space/4kdata 2021-01-15T16:52:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:12.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj4 td/osd-backfill-space/4kdata 2021-01-15T16:52:12.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:12.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj5 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj6 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj7 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj8 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj9 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj10 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj11 td/osd-backfill-space/4kdata 2021-01-15T16:52:13.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:13.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj12 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj13 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj14 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj15 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj16 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj17 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj18 td/osd-backfill-space/4kdata 2021-01-15T16:52:14.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:14.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj19 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj20 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj21 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj22 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj23 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj24 td/osd-backfill-space/4kdata 2021-01-15T16:52:15.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:15.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj25 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj26 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj27 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj28 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj29 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj30 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj31 td/osd-backfill-space/4kdata 2021-01-15T16:52:16.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:16.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj32 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj33 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj34 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj35 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj36 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj37 td/osd-backfill-space/4kdata 2021-01-15T16:52:17.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:17.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj38 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj39 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj40 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj41 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj42 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj43 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj44 td/osd-backfill-space/4kdata 2021-01-15T16:52:18.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:18.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj45 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj46 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj47 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj48 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj49 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj50 td/osd-backfill-space/4kdata 2021-01-15T16:52:19.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:19.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj51 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj52 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj53 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj54 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj55 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj56 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj57 td/osd-backfill-space/4kdata 2021-01-15T16:52:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj58 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj59 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj60 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj61 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj62 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj63 td/osd-backfill-space/4kdata 2021-01-15T16:52:21.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:21.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj64 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj65 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj66 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj67 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj68 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj69 td/osd-backfill-space/4kdata 2021-01-15T16:52:22.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:22.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj70 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj71 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj72 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj73 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj74 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj75 td/osd-backfill-space/4kdata 2021-01-15T16:52:23.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:23.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj76 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj77 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj78 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj79 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj80 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj81 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj82 td/osd-backfill-space/4kdata 2021-01-15T16:52:24.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:24.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj83 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj84 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj85 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj86 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj87 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj88 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj89 td/osd-backfill-space/4kdata 2021-01-15T16:52:25.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:25.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj90 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj91 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj92 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj93 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj94 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj95 td/osd-backfill-space/4kdata 2021-01-15T16:52:26.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:26.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj96 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj97 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj98 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj99 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj100 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj101 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj102 td/osd-backfill-space/4kdata 2021-01-15T16:52:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj103 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj104 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj105 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj106 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj107 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj108 td/osd-backfill-space/4kdata 2021-01-15T16:52:28.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:28.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj109 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj110 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj111 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj112 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj113 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj114 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj115 td/osd-backfill-space/4kdata 2021-01-15T16:52:29.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:29.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj116 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj117 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj118 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj119 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj120 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj121 td/osd-backfill-space/4kdata 2021-01-15T16:52:30.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:30.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj122 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj123 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj124 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj125 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj126 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj127 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj128 td/osd-backfill-space/4kdata 2021-01-15T16:52:31.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:31.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj129 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj130 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj131 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj132 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj133 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj134 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj135 td/osd-backfill-space/4kdata 2021-01-15T16:52:32.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:32.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj136 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj137 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj138 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj139 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj140 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj141 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj142 td/osd-backfill-space/4kdata 2021-01-15T16:52:33.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:33.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj143 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj144 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj145 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj146 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj147 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj148 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj149 td/osd-backfill-space/4kdata 2021-01-15T16:52:34.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:34.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj150 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/4kdata 2021-01-15T16:52:35.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/4kdata 2021-01-15T16:52:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:37.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/4kdata 2021-01-15T16:52:37.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/4kdata 2021-01-15T16:52:38.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/4kdata 2021-01-15T16:52:38.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/4kdata 2021-01-15T16:52:38.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/4kdata 2021-01-15T16:52:38.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/4kdata 2021-01-15T16:52:38.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:38.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/4kdata 2021-01-15T16:52:39.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:39.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/4kdata 2021-01-15T16:52:40.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:40.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/4kdata 2021-01-15T16:52:41.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:41.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/4kdata 2021-01-15T16:52:42.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:42.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/4kdata 2021-01-15T16:52:43.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:43.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/4kdata 2021-01-15T16:52:44.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:44.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/4kdata 2021-01-15T16:52:45.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:45.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/4kdata 2021-01-15T16:52:46.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:46.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/4kdata 2021-01-15T16:52:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/4kdata 2021-01-15T16:52:48.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:48.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/4kdata 2021-01-15T16:52:49.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:49.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/4kdata 2021-01-15T16:52:50.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:50.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/4kdata 2021-01-15T16:52:51.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:51.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/4kdata 2021-01-15T16:52:52.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:52.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/4kdata 2021-01-15T16:52:53.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:53.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/4kdata 2021-01-15T16:52:54.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:54.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/4kdata 2021-01-15T16:52:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/4kdata 2021-01-15T16:52:56.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:56.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/4kdata 2021-01-15T16:52:57.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:57.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/4kdata 2021-01-15T16:52:58.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/4kdata 2021-01-15T16:52:59.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:52:59.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/4kdata 2021-01-15T16:53:00.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:00.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/4kdata 2021-01-15T16:53:01.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:01.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/4kdata 2021-01-15T16:53:02.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:02.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/4kdata 2021-01-15T16:53:03.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:03.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/4kdata 2021-01-15T16:53:04.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:04.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/4kdata 2021-01-15T16:53:05.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/4kdata 2021-01-15T16:53:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/4kdata 2021-01-15T16:53:07.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:07.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/4kdata 2021-01-15T16:53:08.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:08.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/4kdata 2021-01-15T16:53:09.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/4kdata 2021-01-15T16:53:09.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:09.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/4kdata 2021-01-15T16:53:09.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:09.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/4kdata 2021-01-15T16:53:09.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:09.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/4kdata 2021-01-15T16:53:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/4kdata 2021-01-15T16:53:10.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:10.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/4kdata 2021-01-15T16:53:11.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:11.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/4kdata 2021-01-15T16:53:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/4kdata 2021-01-15T16:53:13.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:13.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/4kdata 2021-01-15T16:53:14.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:14.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/4kdata 2021-01-15T16:53:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/4kdata 2021-01-15T16:53:16.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:16.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/4kdata 2021-01-15T16:53:17.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:17.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/4kdata 2021-01-15T16:53:18.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:18.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/4kdata 2021-01-15T16:53:19.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:19.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/4kdata 2021-01-15T16:53:20.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:20.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/4kdata 2021-01-15T16:53:21.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:21.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/4kdata 2021-01-15T16:53:22.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:22.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/4kdata 2021-01-15T16:53:23.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:23.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/4kdata 2021-01-15T16:53:24.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:24.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/4kdata 2021-01-15T16:53:25.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:25.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/4kdata 2021-01-15T16:53:26.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:26.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/4kdata 2021-01-15T16:53:27.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:27.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/4kdata 2021-01-15T16:53:28.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:28.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/4kdata 2021-01-15T16:53:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/4kdata 2021-01-15T16:53:30.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:30.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/4kdata 2021-01-15T16:53:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/4kdata 2021-01-15T16:53:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/4kdata 2021-01-15T16:53:33.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:33.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/4kdata 2021-01-15T16:53:34.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:34.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/4kdata 2021-01-15T16:53:35.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:35.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/4kdata 2021-01-15T16:53:36.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:36.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/4kdata 2021-01-15T16:53:37.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:37.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/4kdata 2021-01-15T16:53:38.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:38.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/4kdata 2021-01-15T16:53:39.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:39.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/4kdata 2021-01-15T16:53:40.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:40.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/4kdata 2021-01-15T16:53:41.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:41.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/4kdata 2021-01-15T16:53:41.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:578: TEST_backfill_grow: for i in $(seq 1 $objects) 2021-01-15T16:53:41.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:580: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/4kdata 2021-01-15T16:53:41.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:583: TEST_backfill_grow: get_pg test obj1 2021-01-15T16:53:41.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T16:53:41.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T16:53:41.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T16:53:41.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T16:53:41.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:583: TEST_backfill_grow: local PG=1.0 2021-01-15T16:53:41.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:585: TEST_backfill_grow: get_primary test obj1 2021-01-15T16:53:41.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T16:53:41.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T16:53:41.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T16:53:41.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T16:53:42.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:585: TEST_backfill_grow: local primary=1 2021-01-15T16:53:42.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:586: TEST_backfill_grow: get_not_primary test obj1 2021-01-15T16:53:42.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T16:53:42.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T16:53:42.248 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T16:53:42.248 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T16:53:42.248 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T16:53:42.249 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T16:53:42.249 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T16:53:42.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T16:53:42.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T16:53:42.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T16:53:43.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:586: TEST_backfill_grow: local otherosd=0 2021-01-15T16:53:43.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:588: TEST_backfill_grow: ceph osd set noout 2021-01-15T16:53:43.940 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T16:53:43.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:589: TEST_backfill_grow: kill_daemons td/osd-backfill-space TERM 0 2021-01-15T16:53:43.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:53:43.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:53:43.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:53:43.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:53:43.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:53:44.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:53:44.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:591: TEST_backfill_grow: expr 600 / 4 2021-01-15T16:53:44.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:591: TEST_backfill_grow: rmobjects=150 2021-01-15T16:53:44.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: seq 1 150 2021-01-15T16:53:44.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:44.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj1 2021-01-15T16:53:44.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:44.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj2 2021-01-15T16:53:44.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:44.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj3 2021-01-15T16:53:45.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj4 2021-01-15T16:53:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj5 2021-01-15T16:53:45.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj6 2021-01-15T16:53:45.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj7 2021-01-15T16:53:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj8 2021-01-15T16:53:45.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj9 2021-01-15T16:53:45.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj10 2021-01-15T16:53:45.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj11 2021-01-15T16:53:45.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:45.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj12 2021-01-15T16:53:46.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj13 2021-01-15T16:53:46.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj14 2021-01-15T16:53:46.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj15 2021-01-15T16:53:46.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj16 2021-01-15T16:53:46.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj17 2021-01-15T16:53:46.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj18 2021-01-15T16:53:46.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj19 2021-01-15T16:53:46.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj20 2021-01-15T16:53:46.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:46.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj21 2021-01-15T16:53:47.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj22 2021-01-15T16:53:47.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj23 2021-01-15T16:53:47.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj24 2021-01-15T16:53:47.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj25 2021-01-15T16:53:47.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj26 2021-01-15T16:53:47.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj27 2021-01-15T16:53:47.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj28 2021-01-15T16:53:47.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj29 2021-01-15T16:53:47.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj30 2021-01-15T16:53:47.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:47.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj31 2021-01-15T16:53:48.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj32 2021-01-15T16:53:48.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj33 2021-01-15T16:53:48.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj34 2021-01-15T16:53:48.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj35 2021-01-15T16:53:48.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj36 2021-01-15T16:53:48.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj37 2021-01-15T16:53:48.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj38 2021-01-15T16:53:48.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj39 2021-01-15T16:53:48.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:48.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj40 2021-01-15T16:53:49.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj41 2021-01-15T16:53:49.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj42 2021-01-15T16:53:49.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj43 2021-01-15T16:53:49.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj44 2021-01-15T16:53:49.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj45 2021-01-15T16:53:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj46 2021-01-15T16:53:49.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj47 2021-01-15T16:53:49.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj48 2021-01-15T16:53:49.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj49 2021-01-15T16:53:49.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:49.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj50 2021-01-15T16:53:50.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj51 2021-01-15T16:53:50.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj52 2021-01-15T16:53:50.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj53 2021-01-15T16:53:50.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj54 2021-01-15T16:53:50.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj55 2021-01-15T16:53:50.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj56 2021-01-15T16:53:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj57 2021-01-15T16:53:50.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj58 2021-01-15T16:53:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj59 2021-01-15T16:53:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj60 2021-01-15T16:53:51.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj61 2021-01-15T16:53:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj62 2021-01-15T16:53:51.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj63 2021-01-15T16:53:51.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj64 2021-01-15T16:53:51.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj65 2021-01-15T16:53:51.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj66 2021-01-15T16:53:51.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:51.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj67 2021-01-15T16:53:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj68 2021-01-15T16:53:52.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj69 2021-01-15T16:53:52.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj70 2021-01-15T16:53:52.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj71 2021-01-15T16:53:52.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj72 2021-01-15T16:53:52.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj73 2021-01-15T16:53:52.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj74 2021-01-15T16:53:52.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj75 2021-01-15T16:53:52.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj76 2021-01-15T16:53:52.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:52.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj77 2021-01-15T16:53:53.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj78 2021-01-15T16:53:53.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj79 2021-01-15T16:53:53.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj80 2021-01-15T16:53:53.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj81 2021-01-15T16:53:53.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj82 2021-01-15T16:53:53.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj83 2021-01-15T16:53:53.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj84 2021-01-15T16:53:53.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj85 2021-01-15T16:53:53.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:53.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj86 2021-01-15T16:53:54.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj87 2021-01-15T16:53:54.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj88 2021-01-15T16:53:54.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj89 2021-01-15T16:53:54.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj90 2021-01-15T16:53:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj91 2021-01-15T16:53:54.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj92 2021-01-15T16:53:54.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj93 2021-01-15T16:53:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj94 2021-01-15T16:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj95 2021-01-15T16:53:54.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:54.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj96 2021-01-15T16:53:55.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj97 2021-01-15T16:53:55.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj98 2021-01-15T16:53:55.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj99 2021-01-15T16:53:55.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj100 2021-01-15T16:53:55.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj101 2021-01-15T16:53:55.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj102 2021-01-15T16:53:55.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj103 2021-01-15T16:53:55.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj104 2021-01-15T16:53:55.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:55.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj105 2021-01-15T16:53:56.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj106 2021-01-15T16:53:56.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj107 2021-01-15T16:53:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj108 2021-01-15T16:53:56.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj109 2021-01-15T16:53:56.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj110 2021-01-15T16:53:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj111 2021-01-15T16:53:56.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj112 2021-01-15T16:53:56.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj113 2021-01-15T16:53:56.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:56.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj114 2021-01-15T16:53:57.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj115 2021-01-15T16:53:57.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj116 2021-01-15T16:53:57.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj117 2021-01-15T16:53:57.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj118 2021-01-15T16:53:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj119 2021-01-15T16:53:57.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj120 2021-01-15T16:53:57.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj121 2021-01-15T16:53:57.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj122 2021-01-15T16:53:57.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj123 2021-01-15T16:53:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj124 2021-01-15T16:53:58.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj125 2021-01-15T16:53:58.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj126 2021-01-15T16:53:58.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj127 2021-01-15T16:53:58.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj128 2021-01-15T16:53:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj129 2021-01-15T16:53:58.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj130 2021-01-15T16:53:58.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj131 2021-01-15T16:53:58.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj132 2021-01-15T16:53:58.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:58.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj133 2021-01-15T16:53:59.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj134 2021-01-15T16:53:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj135 2021-01-15T16:53:59.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj136 2021-01-15T16:53:59.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj137 2021-01-15T16:53:59.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj138 2021-01-15T16:53:59.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj139 2021-01-15T16:53:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj140 2021-01-15T16:53:59.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj141 2021-01-15T16:53:59.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj142 2021-01-15T16:53:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:53:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj143 2021-01-15T16:54:00.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj144 2021-01-15T16:54:00.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj145 2021-01-15T16:54:00.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj146 2021-01-15T16:54:00.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj147 2021-01-15T16:54:00.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj148 2021-01-15T16:54:00.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj149 2021-01-15T16:54:00.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:592: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T16:54:00.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:594: TEST_backfill_grow: rados -p test rm obj150 2021-01-15T16:54:00.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:597: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/6kdata bs=6k count=1 2021-01-15T16:54:00.816 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T16:54:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T16:54:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:6144 bytes (6.1 kB, 6.0 KiB) copied, 8.6999e-05 s, 70.6 MB/s 2021-01-15T16:54:00.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: expr 150 + 1 2021-01-15T16:54:00.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: seq 151 600 2021-01-15T16:54:00.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:00.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/6kdata 2021-01-15T16:54:01.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:01.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/6kdata 2021-01-15T16:54:02.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:02.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/6kdata 2021-01-15T16:54:03.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:03.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/6kdata 2021-01-15T16:54:04.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:04.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/6kdata 2021-01-15T16:54:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/6kdata 2021-01-15T16:54:06.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:06.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/6kdata 2021-01-15T16:54:07.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:07.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/6kdata 2021-01-15T16:54:08.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:08.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/6kdata 2021-01-15T16:54:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/6kdata 2021-01-15T16:54:10.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:10.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/6kdata 2021-01-15T16:54:11.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:11.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/6kdata 2021-01-15T16:54:12.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:12.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/6kdata 2021-01-15T16:54:12.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:12.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/6kdata 2021-01-15T16:54:12.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:12.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/6kdata 2021-01-15T16:54:12.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:12.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/6kdata 2021-01-15T16:54:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/6kdata 2021-01-15T16:54:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:13.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/6kdata 2021-01-15T16:54:14.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:14.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/6kdata 2021-01-15T16:54:15.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:15.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/6kdata 2021-01-15T16:54:16.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:16.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/6kdata 2021-01-15T16:54:17.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:17.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/6kdata 2021-01-15T16:54:18.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:18.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/6kdata 2021-01-15T16:54:19.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:19.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/6kdata 2021-01-15T16:54:20.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:20.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/6kdata 2021-01-15T16:54:21.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:21.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/6kdata 2021-01-15T16:54:22.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/6kdata 2021-01-15T16:54:23.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:23.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/6kdata 2021-01-15T16:54:24.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:24.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/6kdata 2021-01-15T16:54:25.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/6kdata 2021-01-15T16:54:26.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:26.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/6kdata 2021-01-15T16:54:27.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:27.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/6kdata 2021-01-15T16:54:28.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:28.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/6kdata 2021-01-15T16:54:29.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:29.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/6kdata 2021-01-15T16:54:30.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:30.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/6kdata 2021-01-15T16:54:31.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:31.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/6kdata 2021-01-15T16:54:32.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:32.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/6kdata 2021-01-15T16:54:33.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:33.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/6kdata 2021-01-15T16:54:34.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:34.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/6kdata 2021-01-15T16:54:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/6kdata 2021-01-15T16:54:36.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:36.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/6kdata 2021-01-15T16:54:37.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:37.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/6kdata 2021-01-15T16:54:38.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:38.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/6kdata 2021-01-15T16:54:39.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:39.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/6kdata 2021-01-15T16:54:40.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:40.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/6kdata 2021-01-15T16:54:41.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:41.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/6kdata 2021-01-15T16:54:42.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:42.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/6kdata 2021-01-15T16:54:43.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:43.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/6kdata 2021-01-15T16:54:43.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:43.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/6kdata 2021-01-15T16:54:43.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:43.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/6kdata 2021-01-15T16:54:43.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:43.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/6kdata 2021-01-15T16:54:43.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:43.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/6kdata 2021-01-15T16:54:44.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:44.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/6kdata 2021-01-15T16:54:45.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:45.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/6kdata 2021-01-15T16:54:46.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:46.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/6kdata 2021-01-15T16:54:47.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:47.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/6kdata 2021-01-15T16:54:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/6kdata 2021-01-15T16:54:49.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:49.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/6kdata 2021-01-15T16:54:50.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:50.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/6kdata 2021-01-15T16:54:51.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:51.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/6kdata 2021-01-15T16:54:52.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:52.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/6kdata 2021-01-15T16:54:53.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:53.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/6kdata 2021-01-15T16:54:54.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:54.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/6kdata 2021-01-15T16:54:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/6kdata 2021-01-15T16:54:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/6kdata 2021-01-15T16:54:57.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:57.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/6kdata 2021-01-15T16:54:58.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:58.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/6kdata 2021-01-15T16:54:59.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:54:59.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/6kdata 2021-01-15T16:55:00.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:55:00.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/6kdata 2021-01-15T16:55:00.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:55:00.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/6kdata 2021-01-15T16:55:00.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:598: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2021-01-15T16:55:00.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:600: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/6kdata 2021-01-15T16:55:00.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:603: TEST_backfill_grow: activate_osd td/osd-backfill-space 0 2021-01-15T16:55:00.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T16:55:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:55:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T16:55:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:55:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T16:55:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:55:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:55:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:55:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:55:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T16:55:00.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T16:55:00.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:55:00.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:55:00.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:55:00.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:55:00.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:55:00.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:55:00.390 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:55:00.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:55:00.391 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:55:00.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:55:00.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:55:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:55:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:55:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:55:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:55:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:55:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:55:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:55:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:55:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T16:55:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T16:55:00.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:55:00.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/0/whoami 2021-01-15T16:55:00.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T16:55:00.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:55:00.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:55:00.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:55:00.395 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:55:00.411 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:55:00.411+0000 7f1ccc236080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:55:00.412 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:55:00.412+0000 7f1ccc236080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:55:00.415 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:55:00.414+0000 7f1ccc236080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:55:00.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T16:55:00.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:55:00.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:55:00.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:55:00.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:55:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:55:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:55:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:55:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:55:00.786 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:55:00.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:55:00.929+0000 7f1ccc236080 -1 Falling back to public interface 2021-01-15T16:55:01.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:55:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:55:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:55:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:55:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:55:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:55:02.179 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:55:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:55:02.183+0000 7f1ccc236080 -1 osd.0 23 log_to_monitors {default=true} 2021-01-15T16:55:02.578 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 26 last_clean_interval [5,23) [v2:127.0.0.1:6802/201717,v1:127.0.0.1:6803/201717] [v2:127.0.0.1:6804/201717,v1:127.0.0.1:6805/201717] exists,up ad97470d-fe4a-4b7c-bdb5-ed161f3dfed9 2021-01-15T16:55:02.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:55:02.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:55:02.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:55:02.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:605: TEST_backfill_grow: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T16:55:02.708 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T16:55:02.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:607: TEST_backfill_grow: sleep 2 2021-01-15T16:55:04.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:609: TEST_backfill_grow: wait_for_clean 2021-01-15T16:55:04.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:55:04.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:55:04.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:55:04.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:55:04.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:55:04.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:55:04.722 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:55:04.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:55:04.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:55:04.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:55:04.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:55:04.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:55:04.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:55:04.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:55:04.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:55:05.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:55:05.225 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:55:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:55:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:55:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:55:05.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:55:05.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084291 2021-01-15T16:55:05.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084291 2021-01-15T16:55:05.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291' 2021-01-15T16:55:05.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:55:05.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:55:05.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673003 2021-01-15T16:55:05.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673003 2021-01-15T16:55:05.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291 1-42949673003' 2021-01-15T16:55:05.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:55:05.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:55:05.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509482 2021-01-15T16:55:05.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509482 2021-01-15T16:55:05.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-120259084291 1-42949673003 2-64424509482' 2021-01-15T16:55:05.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:55:05.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-120259084291 2021-01-15T16:55:05.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:55:05.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:55:05.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-120259084291 2021-01-15T16:55:05.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:55:05.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084291 2021-01-15T16:55:05.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2021-01-15T16:55:05.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:55:05.650 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 120259084291 2021-01-15T16:55:06.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 120259084291 2021-01-15T16:55:06.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:55:07.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:55:07.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:55:07.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084291 -lt 120259084291 2021-01-15T16:55:07.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:55:07.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673003 2021-01-15T16:55:07.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:55:07.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:55:07.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673003 2021-01-15T16:55:07.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:55:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673003 2021-01-15T16:55:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2021-01-15T16:55:07.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:55:07.447 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673003 2021-01-15T16:55:07.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673004 -lt 42949673003 2021-01-15T16:55:07.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:55:07.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509482 2021-01-15T16:55:07.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:55:07.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:55:07.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509482 2021-01-15T16:55:07.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:55:07.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509482 2021-01-15T16:55:07.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2021-01-15T16:55:07.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:55:07.850 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509482 2021-01-15T16:55:08.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509482 -lt 64424509482 2021-01-15T16:55:08.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:55:08.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:08.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:08.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T16:55:08.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:08.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:08.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:08.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:08.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:08.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:08.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:09.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:09.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:09.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:09.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:09.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:09.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T16:55:09.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:55:09.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T16:55:09.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:09.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:09.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:09.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:09.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:09.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:09.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:09.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:09.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:09.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:10.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:10.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:10.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:10.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:10.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:10.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:10.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:10.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:10.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:10.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:11.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9012 2021-01-15T16:55:11.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9012 '!=' null 2021-01-15T16:55:11.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:11.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:11.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:11.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:11.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:11.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:11.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:11.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:11.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:11.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:11.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:11.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:11.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:11.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:11.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:12.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:12.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:12.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:12.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9012 2021-01-15T16:55:12.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9012 '!=' null 2021-01-15T16:55:12.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:12.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:12.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:12.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:12.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:12.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:12.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:12.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:12.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:12.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:12.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:13.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:13.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:13.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:13.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:13.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:13.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:13.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:13.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:13.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:13.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:13.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:13.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:14.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25225 2021-01-15T16:55:14.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25225 '!=' null 2021-01-15T16:55:14.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:14.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:14.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:14.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:14.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:14.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:14.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:14.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:14.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:14.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:14.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:14.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:14.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:14.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:15.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:15.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:15.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:15.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:15.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:15.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21363 2021-01-15T16:55:15.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21363 '!=' null 2021-01-15T16:55:15.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:15.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:15.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:15.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:15.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:15.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:15.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:15.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:15.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:15.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:15.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:16.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:16.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:16.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:16.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:16.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:16.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:16.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:16.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:16.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:16.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:16.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:16.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:16.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:17.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=32255 2021-01-15T16:55:17.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 32255 '!=' null 2021-01-15T16:55:17.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:17.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:17.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:17.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:17.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:17.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:17.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:17.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:17.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:17.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:17.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:18.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:18.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:18.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:18.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=32255 2021-01-15T16:55:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 32255 '!=' null 2021-01-15T16:55:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:19.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:19.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:19.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:19.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:19.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:19.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:19.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:19.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:19.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:19.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:19.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:19.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:19.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:19.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:19.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:19.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:20.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26623 2021-01-15T16:55:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26623 '!=' null 2021-01-15T16:55:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:20.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:20.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:20.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:20.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:20.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:20.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:20.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:20.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:21.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:21.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:21.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:21.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:21.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:21.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:22.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26623 2021-01-15T16:55:22.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26623 '!=' null 2021-01-15T16:55:22.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:22.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:22.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:22.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:22.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:22.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:22.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:22.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:22.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:22.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:22.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:22.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:22.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:22.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:22.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:23.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:23.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:23.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:23.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:23.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40958 2021-01-15T16:55:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40958 '!=' null 2021-01-15T16:55:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:23.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:23.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:23.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:24.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:24.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:24.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:24.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:24.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:24.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:24.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:24.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:24.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:24.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:24.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:24.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:24.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:25.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27135 2021-01-15T16:55:25.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27135 '!=' null 2021-01-15T16:55:25.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:25.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:25.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:25.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:25.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:25.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:25.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:25.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:25.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:25.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:25.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:25.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:25.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:25.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:26.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:26.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:26.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:26.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27135 2021-01-15T16:55:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27135 '!=' null 2021-01-15T16:55:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:26.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:26.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:26.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:26.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:26.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:26.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:26.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:26.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:26.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:26.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:27.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:27.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:27.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:27.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:27.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:28.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=40447 2021-01-15T16:55:28.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 40447 '!=' null 2021-01-15T16:55:28.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:28.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:28.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:28.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:28.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:28.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:28.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:28.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:28.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:28.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:28.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:28.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:28.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:29.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:29.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:29.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:29.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:29.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:29.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:29.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:29.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:29.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:29.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:29.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:29.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:29.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:29.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:29.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:30.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:30.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:30.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:30.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:30.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:30.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:30.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:30.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:30.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:30.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:30.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:30.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:30.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:31.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:31.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:31.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:31.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:31.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:31.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:31.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:31.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:31.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:31.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:31.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:31.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:31.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:32.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:32.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:32.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:32.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:32.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:32.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:32.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:32.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:32.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:32.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:32.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:32.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:32.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:32.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:32.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:32.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:32.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:32.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:32.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:32.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:32.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:32.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:33.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:33.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:33.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:33.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:33.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:33.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:33.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:33.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:34.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41983 2021-01-15T16:55:34.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41983 '!=' null 2021-01-15T16:55:34.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:34.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:34.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:34.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:34.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:34.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:34.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:34.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:34.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:35.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:35.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:35.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:35.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:35.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:35.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:35.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:35.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:35.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:35.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:35.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:35.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:35.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:35.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:35.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:35.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:35.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:35.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:35.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:35.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:35.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:35.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:36.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:36.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:36.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:36.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:36.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:36.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:36.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:36.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:36.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:36.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:36.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:36.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:36.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:37.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42494 2021-01-15T16:55:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42494 '!=' null 2021-01-15T16:55:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:37.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:37.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:37.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:37.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:37.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:37.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:37.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:37.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:37.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:37.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:38.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:38.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:38.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:38.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:38.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:38.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:38.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:38.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:38.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:38.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42494 2021-01-15T16:55:38.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42494 '!=' null 2021-01-15T16:55:38.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:38.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:38.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:39.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:39.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:39.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:39.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:39.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:39.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:39.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:39.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:39.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:39.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:39.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:39.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:39.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:39.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:39.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:39.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:39.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:39.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:39.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:39.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:39.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:40.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=29183 2021-01-15T16:55:40.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 29183 '!=' null 2021-01-15T16:55:40.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:40.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:40.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:40.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:40.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:40.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:40.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:40.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:40.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:40.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:40.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:40.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:40.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:40.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:40.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:41.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:41.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:41.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:41.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:42.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=29183 2021-01-15T16:55:42.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 29183 '!=' null 2021-01-15T16:55:42.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:42.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:42.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:42.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:42.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:42.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:42.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:42.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:42.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:42.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:42.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:42.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:42.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:42.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:42.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:43.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:43.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:43.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:43.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:43.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:43.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:43.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:43.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:43.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:43.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=43007 2021-01-15T16:55:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 43007 '!=' null 2021-01-15T16:55:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:43.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:43.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:43.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:43.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:43.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:43.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:43.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:43.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:44.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:44.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:44.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:44.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:44.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:44.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:44.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:44.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:44.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:44.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:44.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:44.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:44.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:45.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=43007 2021-01-15T16:55:45.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 43007 '!=' null 2021-01-15T16:55:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:45.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:45.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:45.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:45.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:45.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:45.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:45.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:45.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:45.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:45.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:45.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:45.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:46.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:46.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:46.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:46.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:55:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:55:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:46.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:46.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:46.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:46.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:46.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:46.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:46.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:46.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:47.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:47.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:47.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:47.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:47.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:47.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:47.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:47.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:47.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:47.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:47.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:47.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:48.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42495 2021-01-15T16:55:48.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42495 '!=' null 2021-01-15T16:55:48.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:48.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:48.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:48.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:48.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:48.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:48.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:48.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:48.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:48.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:48.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:48.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:48.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:49.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:49.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:49.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:49.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:49.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:49.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:49.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:49.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:49.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:49.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:49.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:49.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:49.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:49.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:49.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:49.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:49.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:49.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:50.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:50.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:50.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:50.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:50.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:50.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:50.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:50.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:50.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:51.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:51.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:51.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:51.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:51.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:51.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:51.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:51.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:51.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:51.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:51.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:51.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:51.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:51.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:51.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:51.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:51.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:52.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:52.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:52.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:52.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:52.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:52.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:52.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:52.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:52.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:52.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:55:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:55:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:52.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:52.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:53.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:53.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:53.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:53.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:53.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:53.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:53.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:53.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:53.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:53.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:53.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:53.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:53.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:54.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41982 2021-01-15T16:55:54.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41982 '!=' null 2021-01-15T16:55:54.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:54.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:54.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:54.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:54.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:54.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:54.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:54.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:54.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:54.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:54.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:54.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:54.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:54.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:55.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:55.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:55.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:55.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:55.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28159 2021-01-15T16:55:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28159 '!=' null 2021-01-15T16:55:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:56.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:56.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:56.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:56.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:56.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:56.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:56.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:56.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:56.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:56.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:56.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:56.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:56.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:56.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:56.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:56.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:57.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42494 2021-01-15T16:55:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42494 '!=' null 2021-01-15T16:55:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:57.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:57.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:57.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:57.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:57.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:57.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:57.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:57.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:57.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:57.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:57.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:55:58.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:55:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:55:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:55:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:55:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:55:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:55:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:55:58.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:55:58.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:55:59.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42494 2021-01-15T16:55:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42494 '!=' null 2021-01-15T16:55:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:55:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:55:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:55:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:55:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:55:59.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:55:59.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:55:59.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:55:59.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:55:59.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:55:59.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:55:59.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:55:59.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:55:59.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:55:59.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:00.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:00.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:00.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:00.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28670 2021-01-15T16:56:00.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28670 '!=' null 2021-01-15T16:56:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:00.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:00.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:00.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:00.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:00.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:00.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:01.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:01.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:01.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:01.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:01.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:01.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:02.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:56:02.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:56:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:02.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:02.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:02.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:02.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:02.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:02.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:02.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:02.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:02.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:02.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:02.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:02.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:03.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:03.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:03.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:03.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42495 2021-01-15T16:56:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42495 '!=' null 2021-01-15T16:56:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:03.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:03.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:03.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:03.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:03.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:03.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:03.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:03.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:03.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:03.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:04.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:04.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:04.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:04.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:04.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:04.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:04.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:04.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:04.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28159 2021-01-15T16:56:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28159 '!=' null 2021-01-15T16:56:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:05.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:05.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:05.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:05.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:05.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:05.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:05.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:05.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:05.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:05.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:05.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:05.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:06.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:06.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:06.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:06.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:06.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:06.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:06.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28159 2021-01-15T16:56:06.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28159 '!=' null 2021-01-15T16:56:06.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:06.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:06.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:06.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:06.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:06.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:06.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:06.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:06.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:06.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:06.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:07.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:07.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:07.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:07.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:07.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:07.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:07.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:07.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:07.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:07.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:07.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:07.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:08.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=43007 2021-01-15T16:56:08.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 43007 '!=' null 2021-01-15T16:56:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:08.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:08.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:08.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:08.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:08.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:08.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:08.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:08.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:08.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:09.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:09.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:09.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:09.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:09.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:09.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:09.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:09.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:09.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:56:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:56:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:09.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:09.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:09.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:09.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:09.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:10.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:10.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:10.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:10.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:10.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:10.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:10.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:10.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:11.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:56:11.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:56:11.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:11.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:11.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:11.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:11.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:11.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:11.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:11.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:11.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:11.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:11.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:11.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:11.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:11.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:11.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:12.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:12.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:12.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:12.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:12.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:12.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:12.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:56:12.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:56:12.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:12.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:12.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:12.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:12.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:12.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:12.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:12.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:12.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:12.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:13.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:13.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:13.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:13.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:13.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:13.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:13.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:13.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:13.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42495 2021-01-15T16:56:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42495 '!=' null 2021-01-15T16:56:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:14.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:14.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:14.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:14.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:14.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:14.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:14.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:14.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:14.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:14.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:14.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:14.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:14.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:15.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:15.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:15.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:15.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28671 2021-01-15T16:56:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28671 '!=' null 2021-01-15T16:56:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:15.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:16.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:16.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:16.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:16.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:16.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:16.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:16.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:16.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:16.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:16.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:16.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:16.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:16.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:16.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:16.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:16.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:16.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:16.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:16.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:16.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:16.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:17.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42495 2021-01-15T16:56:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42495 '!=' null 2021-01-15T16:56:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:17.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:17.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:17.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:17.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:17.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:17.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:17.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:17.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:18.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:18.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:18.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:18.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:18.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:18.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:19.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42495 2021-01-15T16:56:19.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42495 '!=' null 2021-01-15T16:56:19.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:19.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:19.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:19.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:19.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:19.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:19.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:19.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:19.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:19.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:19.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:19.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:19.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:20.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:20.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:20.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:20.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:20.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:20.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:20.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:56:20.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:56:20.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:20.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:20.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:20.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:20.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:20.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:20.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:20.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:20.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:20.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:20.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:21.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:21.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:21.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:21.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:21.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:21.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:21.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:21.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:21.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:21.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:22.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27647 2021-01-15T16:56:22.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27647 '!=' null 2021-01-15T16:56:22.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:22.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:22.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:22.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:22.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:22.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:22.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:22.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:22.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:22.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:22.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:22.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:23.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:23.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:23.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:23.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:23.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41983 2021-01-15T16:56:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41983 '!=' null 2021-01-15T16:56:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:23.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:23.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:23.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:23.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:23.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:23.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:23.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:23.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:24.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:24.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:24.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:24.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:24.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:24.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:24.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:24.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:24.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:25.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28159 2021-01-15T16:56:25.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28159 '!=' null 2021-01-15T16:56:25.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:25.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:25.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:25.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:25.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:25.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:25.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:25.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:25.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:25.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:25.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:25.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:25.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:25.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:25.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:26.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:26.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:26.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:26.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:26.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:26.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:26.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:26.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:26.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:26.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=28159 2021-01-15T16:56:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 28159 '!=' null 2021-01-15T16:56:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:26.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:26.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:26.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:26.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:26.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:26.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:26.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:26.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:27.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:27.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:27.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:27.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:27.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:27.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:27.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:27.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:27.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:27.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:27.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:27.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:28.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=41983 2021-01-15T16:56:28.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 41983 '!=' null 2021-01-15T16:56:28.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:28.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:28.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:28.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:28.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:28.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:28.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:28.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:28.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:28.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:28.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:28.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:28.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:28.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:28.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:29.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:29.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:29.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:29.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:29.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:29.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:29.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:29.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:29.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:29.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=42239 2021-01-15T16:56:29.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 42239 '!=' null 2021-01-15T16:56:29.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:29.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:29.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:29.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:29.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:29.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:29.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:29.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:29.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:29.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:29.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:30.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:30.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:30.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:30.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:30.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:30.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:30.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:30.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:30.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:30.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:30.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:30.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:30.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:31.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20735 2021-01-15T16:56:31.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20735 '!=' null 2021-01-15T16:56:31.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:31.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:31.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:31.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:31.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:31.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:31.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:31.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:31.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:31.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:31.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:31.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T16:56:31.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:31.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:31.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:32.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T16:56:32.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T16:56:32.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T16:56:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T16:56:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T16:56:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T16:56:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T16:56:32.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T16:56:32.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T16:56:32.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20735 2021-01-15T16:56:32.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20735 '!=' null 2021-01-15T16:56:32.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T16:56:32.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:56:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:56:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:56:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:56:32.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:56:32.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:56:32.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:56:32.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:56:32.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:56:32.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:56:33.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T16:56:33.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:56:33.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:56:33.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:56:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T16:56:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:56:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:56:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:611: TEST_backfill_grow: delete_pool test 2021-01-15T16:56:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T16:56:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T16:56:34.398 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T16:56:34.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:612: TEST_backfill_grow: kill_daemons td/osd-backfill-space 2021-01-15T16:56:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:56:34.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:56:34.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:56:34.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:56:34.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:56:34.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:56:34.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:613: TEST_backfill_grow: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2021-01-15T16:56:34.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T16:56:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T16:56:34.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:56:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T16:56:34.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:56:34.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:56:34.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:56:34.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:56:34.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:56:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:56:34.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:56:34.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:56:34.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:56:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:56:34.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:56:34.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:56:34.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:56:34.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:56:34.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:56:34.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:56:34.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:56:34.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:56:34.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T16:56:34.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:56:34.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:34.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:34.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T16:56:34.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:56:34.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:56:34.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T16:56:34.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T16:56:34.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T16:56:34.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T16:56:34.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T16:56:34.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T16:56:34.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T16:56:34.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:56:34.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:56:34.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:56:34.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:56:34.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:56:34.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:56:34.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T16:56:34.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T16:56:34.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T16:56:34.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T16:56:34.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T16:56:34.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T16:56:34.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:56:34.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T16:56:34.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T16:56:34.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T16:56:34.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T16:56:34.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T16:56:34.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T16:56:34.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T16:56:34.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:34.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:34.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T16:56:34.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T16:56:34.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T16:56:34.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T16:56:34.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T16:56:34.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:34.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:34.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T16:56:34.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T16:56:34.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T16:56:34.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T16:56:34.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T16:56:34.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_multi_partial td/osd-backfill-space 2021-01-15T16:56:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:429: TEST_backfill_multi_partial: local dir=td/osd-backfill-space 2021-01-15T16:56:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:430: TEST_backfill_multi_partial: local EC= 2021-01-15T16:56:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:431: TEST_backfill_multi_partial: local pools=2 2021-01-15T16:56:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:432: TEST_backfill_multi_partial: local OSDS=3 2021-01-15T16:56:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:434: TEST_backfill_multi_partial: run_mon td/osd-backfill-space a 2021-01-15T16:56:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T16:56:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T16:56:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T16:56:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T16:56:34.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T16:56:34.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T16:56:34.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T16:56:34.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:56:34.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:56:34.908 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:56:34.908 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:34.909 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:34.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:34.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T16:56:35.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T16:56:35.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T16:56:35.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:56:35.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:56:35.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T16:56:35.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T16:56:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:56:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:56:35.095 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:56:35.095 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:56:35.095 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:35.095 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:35.095 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T16:56:35.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:56:35.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T16:56:35.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T16:56:35.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T16:56:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T16:56:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T16:56:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T16:56:35.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T16:56:35.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T16:56:35.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T16:56:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T16:56:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:35.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T16:56:35.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T16:56:35.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T16:56:35.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:435: TEST_backfill_multi_partial: run_mgr td/osd-backfill-space x 2021-01-15T16:56:35.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T16:56:35.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T16:56:35.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T16:56:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T16:56:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T16:56:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T16:56:36.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T16:56:36.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:56:36.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:56:36.195 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:56:36.195 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:36.195 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:36.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:36.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:56:36.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T16:56:36.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:436: TEST_backfill_multi_partial: export CEPH_ARGS 2021-01-15T16:56:36.220 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: expr 3 - 1 2021-01-15T16:56:36.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: seq 0 2 2021-01-15T16:56:36.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:56:36.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 0 2021-01-15T16:56:36.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:56:36.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:56:36.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T16:56:36.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:56:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T16:56:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:56:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:56:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:56:36.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:56:36.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T16:56:36.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T16:56:36.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:56:36.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:56:36.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:56:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:56:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:56:36.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:56:36.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:56:36.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:36.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:36.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:36.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:36.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:56:36.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:56:36.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:56:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:56:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:56:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:56:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:56:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:56:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:56:36.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T16:56:36.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:56:36.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d63a9ef1-0498-4cb7-964a-2ce69b80b192 2021-01-15T16:56:36.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d63a9ef1-0498-4cb7-964a-2ce69b80b192' 2021-01-15T16:56:36.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:56:36.233 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d63a9ef1-0498-4cb7-964a-2ce69b80b192 2021-01-15T16:56:36.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBEyQFgrodsDhAANjIGVLLV/t1J21dByQqDng== 2021-01-15T16:56:36.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBEyQFgrodsDhAANjIGVLLV/t1J21dByQqDng=="}' 2021-01-15T16:56:36.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d63a9ef1-0498-4cb7-964a-2ce69b80b192 -i td/osd-backfill-space/0/new.json 2021-01-15T16:56:36.619 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:56:36.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T16:56:36.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEyQFgrodsDhAANjIGVLLV/t1J21dByQqDng== --osd-uuid d63a9ef1-0498-4cb7-964a-2ce69b80b192 2021-01-15T16:56:36.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.657+0000 7f468d04d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:36.659 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.659+0000 7f468d04d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:36.660 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.660+0000 7f468d04d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:36.661 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.661+0000 7f468d04d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:56:36.661 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.661+0000 7f468d04d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:56:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.661+0000 7f468d04d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T16:56:36.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:36.699+0000 7f468d04d080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T16:56:41.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T16:56:41.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:56:41.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T16:56:41.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:56:41.300 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T16:56:41.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T16:56:41.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:41.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:56:41.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:56:41.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:56:41.830 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T16:56:41.854 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:41.854+0000 7fd23fb41080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:41.856 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:41.856+0000 7fd23fb41080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:41.857 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:41.857+0000 7fd23fb41080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:42.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T16:56:42.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:56:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T16:56:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:56:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:56:42.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:42.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:56:42.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:42.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:56:42.245 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:56:42.377 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:42.376+0000 7fd23fb41080 -1 Falling back to public interface 2021-01-15T16:56:42.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:56:43.192 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:43.192+0000 7fd23fb41080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T16:56:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:56:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:56:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T16:56:43.636 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:56:44.066 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/208057,v1:127.0.0.1:6803/208057] [v2:127.0.0.1:6804/208057,v1:127.0.0.1:6805/208057] exists,up d63a9ef1-0498-4cb7-964a-2ce69b80b192 2021-01-15T16:56:44.066 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T16:56:44.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:56:44.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:56:44.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:56:44.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:56:44.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 1 2021-01-15T16:56:44.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:56:44.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:56:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T16:56:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:56:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T16:56:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:56:44.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:56:44.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:56:44.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:56:44.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T16:56:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T16:56:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:56:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:56:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:56:44.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:56:44.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:56:44.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:56:44.072 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:56:44.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:44.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:44.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:44.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:56:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:56:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:56:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:56:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:56:44.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:56:44.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:56:44.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:44.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:56:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:56:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T16:56:44.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:56:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T16:56:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 ba9894f0-c157-464b-ad69-8661d42507f9' 2021-01-15T16:56:44.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:56:44.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBMyQFg+z+hBBAAJgxfA4IcwiE93UwI5gwU3g== 2021-01-15T16:56:44.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBMyQFg+z+hBBAAJgxfA4IcwiE93UwI5gwU3g=="}' 2021-01-15T16:56:44.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ba9894f0-c157-464b-ad69-8661d42507f9 -i td/osd-backfill-space/1/new.json 2021-01-15T16:56:44.568 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:56:44.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T16:56:44.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBMyQFg+z+hBBAAJgxfA4IcwiE93UwI5gwU3g== --osd-uuid ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T16:56:44.606 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.606+0000 7fba6b3c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:44.608 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.608+0000 7fba6b3c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:44.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.610+0000 7fba6b3c8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:44.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.610+0000 7fba6b3c8080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:56:44.611 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.610+0000 7fba6b3c8080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:56:44.611 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.610+0000 7fba6b3c8080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T16:56:44.663 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:44.663+0000 7fba6b3c8080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T16:56:49.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T16:56:49.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:56:49.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T16:56:49.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:56:49.238 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T16:56:49.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T16:56:49.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:56:49.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:49.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:56:49.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:56:49.777 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:56:49.800 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:49.800+0000 7f9d1848d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:49.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:49.811+0000 7f9d1848d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:49.813 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:49.813+0000 7f9d1848d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:50.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T16:56:50.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:56:50.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:56:50.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:56:50.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:56:50.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:50.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:56:50.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:50.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:56:50.193 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:56:50.329 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:50.329+0000 7f9d1848d080 -1 Falling back to public interface 2021-01-15T16:56:50.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:56:51.120 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:51.120+0000 7f9d1848d080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T16:56:51.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:56:51.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:51.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:56:51.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:51.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:56:51.588 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:56:52.017 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/209321,v1:127.0.0.1:6811/209321] [v2:127.0.0.1:6812/209321,v1:127.0.0.1:6813/209321] exists,up ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T16:56:52.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:56:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:56:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:56:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:438: TEST_backfill_multi_partial: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T16:56:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:440: TEST_backfill_multi_partial: run_osd td/osd-backfill-space 2 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T16:56:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T16:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T16:56:52.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T16:56:52.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T16:56:52.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:56:52.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T16:56:52.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:56:52.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:56:52.022 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:56:52.022 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:56:52.022 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:56:52.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:56:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:56:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T16:56:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T16:56:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:56:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:56:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:56:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:56:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T16:56:52.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T16:56:52.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T16:56:52.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T16:56:52.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=da24b4ca-22e9-4a54-afb6-b97afd81ecc4 2021-01-15T16:56:52.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 da24b4ca-22e9-4a54-afb6-b97afd81ecc4' 2021-01-15T16:56:52.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T16:56:52.026 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 da24b4ca-22e9-4a54-afb6-b97afd81ecc4 2021-01-15T16:56:52.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBUyQFgTNUSAhAA1eE5yhVDuLtwkLxjuROMyw== 2021-01-15T16:56:52.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBUyQFgTNUSAhAA1eE5yhVDuLtwkLxjuROMyw=="}' 2021-01-15T16:56:52.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new da24b4ca-22e9-4a54-afb6-b97afd81ecc4 -i td/osd-backfill-space/2/new.json 2021-01-15T16:56:52.644 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T16:56:52.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T16:56:52.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUyQFgTNUSAhAA1eE5yhVDuLtwkLxjuROMyw== --osd-uuid da24b4ca-22e9-4a54-afb6-b97afd81ecc4 2021-01-15T16:56:52.685 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.684+0000 7fac106ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:52.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.686+0000 7fac106ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:52.687 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.687+0000 7fac106ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:52.688 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.688+0000 7fac106ef080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:56:52.688 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.688+0000 7fac106ef080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:56:52.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.688+0000 7fac106ef080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T16:56:52.736 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:52.736+0000 7fac106ef080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T16:56:57.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T16:56:57.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T16:56:57.320 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T16:56:57.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T16:56:57.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T16:56:57.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T16:56:57.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:56:57.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T16:56:57.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T16:56:57.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T16:56:57.847 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T16:56:57.870 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:57.870+0000 7fbb2e485080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:57.893 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:57.893+0000 7fbb2e485080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:57.895 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:57.894+0000 7fbb2e485080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:56:58.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T16:56:58.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:56:58.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:56:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:56:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:56:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:56:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:58.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:56:58.253 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:56:58.411 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:58.411+0000 7fbb2e485080 -1 Falling back to public interface 2021-01-15T16:56:58.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:56:59.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:56:59.202+0000 7fbb2e485080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T16:56:59.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:56:59.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:56:59.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:56:59.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:56:59.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:56:59.651 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:57:00.060 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/210585,v1:127.0.0.1:6819/210585] [v2:127.0.0.1:6820/210585,v1:127.0.0.1:6821/210585] exists,up da24b4ca-22e9-4a54-afb6-b97afd81ecc4 2021-01-15T16:57:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:57:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:57:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:57:00.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:443: TEST_backfill_multi_partial: ceph osd set-backfillfull-ratio .85 2021-01-15T16:57:00.871 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T16:57:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:445: TEST_backfill_multi_partial: ceph osd set-require-min-compat-client luminous 2021-01-15T16:57:01.631 INFO:tasks.workunit.client.0.smithi145.stderr:set require_min_compat_client to luminous 2021-01-15T16:57:01.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:446: TEST_backfill_multi_partial: create_pool fillpool 1 1 2021-01-15T16:57:01.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2021-01-15T16:57:02.245 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' already exists 2021-01-15T16:57:02.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:57:03.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:447: TEST_backfill_multi_partial: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2021-01-15T16:57:04.132 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T16:57:04.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: seq 1 2 2021-01-15T16:57:04.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T16:57:04.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:450: TEST_backfill_multi_partial: create_pool test1 1 1 2021-01-15T16:57:04.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T16:57:04.737 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T16:57:04.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:57:05.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:451: TEST_backfill_multi_partial: ceph osd pool set test1 size 2 2021-01-15T16:57:06.521 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T16:57:06.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:448: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T16:57:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:450: TEST_backfill_multi_partial: create_pool test2 1 1 2021-01-15T16:57:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T16:57:07.230 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T16:57:07.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T16:57:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:451: TEST_backfill_multi_partial: ceph osd pool set test2 size 2 2021-01-15T16:57:09.008 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T16:57:09.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:454: TEST_backfill_multi_partial: wait_for_clean 2021-01-15T16:57:09.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T16:57:09.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T16:57:09.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T16:57:09.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T16:57:09.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T16:57:09.020 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T16:57:09.020 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T16:57:09.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T16:57:09.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T16:57:09.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T16:57:09.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T16:57:09.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T16:57:09.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T16:57:09.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T16:57:09.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T16:57:09.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T16:57:09.542 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:57:09.542 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T16:57:09.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T16:57:09.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:57:09.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T16:57:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2021-01-15T16:57:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2021-01-15T16:57:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2021-01-15T16:57:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:57:09.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T16:57:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672967 2021-01-15T16:57:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672967 2021-01-15T16:57:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2021-01-15T16:57:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T16:57:09.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T16:57:09.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T16:57:09.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T16:57:09.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509444' 2021-01-15T16:57:09.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:57:09.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2021-01-15T16:57:09.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:57:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T16:57:09.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-01-15T16:57:09.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:57:09.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2021-01-15T16:57:09.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-01-15T16:57:09.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T16:57:09.973 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836488 2021-01-15T16:57:10.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836488 2021-01-15T16:57:10.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:57:10.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672967 2021-01-15T16:57:10.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:57:10.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T16:57:10.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672967 2021-01-15T16:57:10.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:57:10.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672967 2021-01-15T16:57:10.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-01-15T16:57:10.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T16:57:10.400 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672967 2021-01-15T16:57:10.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672967 -lt 42949672967 2021-01-15T16:57:10.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T16:57:10.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T16:57:10.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T16:57:10.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T16:57:10.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T16:57:10.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T16:57:10.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T16:57:10.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T16:57:10.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:57:10.826 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T16:57:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509442 -lt 64424509444 2021-01-15T16:57:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T16:57:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T16:57:12.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T16:57:12.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509445 -lt 64424509444 2021-01-15T16:57:12.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T16:57:12.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:57:12.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:57:13.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T16:57:13.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:57:13.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:57:13.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:57:13.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:57:13.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:57:13.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:57:13.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:57:13.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T16:57:13.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:57:13.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:57:13.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:57:14.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T16:57:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' -1 2021-01-15T16:57:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T16:57:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=2 2021-01-15T16:57:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T16:57:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T16:57:14.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T16:57:14.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T16:57:14.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T16:57:14.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T16:57:14.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T16:57:14.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T16:57:14.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T16:57:14.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T16:57:14.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T16:57:14.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T16:57:14.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T16:57:14.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T16:57:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T16:57:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T16:57:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T16:57:15.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:461: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=2611 count=1 2021-01-15T16:57:15.048 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T16:57:15.048 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T16:57:15.049 INFO:tasks.workunit.client.0.smithi145.stderr:2611 bytes (2.6 kB, 2.5 KiB) copied, 5.4637e-05 s, 47.8 MB/s 2021-01-15T16:57:15.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: seq 1 600 2021-01-15T16:57:15.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-1 td/osd-backfill-space/datafile 2021-01-15T16:57:15.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-2 td/osd-backfill-space/datafile 2021-01-15T16:57:15.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-3 td/osd-backfill-space/datafile 2021-01-15T16:57:15.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-4 td/osd-backfill-space/datafile 2021-01-15T16:57:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-5 td/osd-backfill-space/datafile 2021-01-15T16:57:15.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-6 td/osd-backfill-space/datafile 2021-01-15T16:57:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-7 td/osd-backfill-space/datafile 2021-01-15T16:57:15.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-8 td/osd-backfill-space/datafile 2021-01-15T16:57:15.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-9 td/osd-backfill-space/datafile 2021-01-15T16:57:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-10 td/osd-backfill-space/datafile 2021-01-15T16:57:15.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:15.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-11 td/osd-backfill-space/datafile 2021-01-15T16:57:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-12 td/osd-backfill-space/datafile 2021-01-15T16:57:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-13 td/osd-backfill-space/datafile 2021-01-15T16:57:16.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-14 td/osd-backfill-space/datafile 2021-01-15T16:57:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-15 td/osd-backfill-space/datafile 2021-01-15T16:57:16.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-16 td/osd-backfill-space/datafile 2021-01-15T16:57:16.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-17 td/osd-backfill-space/datafile 2021-01-15T16:57:16.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-18 td/osd-backfill-space/datafile 2021-01-15T16:57:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-19 td/osd-backfill-space/datafile 2021-01-15T16:57:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-20 td/osd-backfill-space/datafile 2021-01-15T16:57:16.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-21 td/osd-backfill-space/datafile 2021-01-15T16:57:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-22 td/osd-backfill-space/datafile 2021-01-15T16:57:17.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-23 td/osd-backfill-space/datafile 2021-01-15T16:57:17.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-24 td/osd-backfill-space/datafile 2021-01-15T16:57:17.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-25 td/osd-backfill-space/datafile 2021-01-15T16:57:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-26 td/osd-backfill-space/datafile 2021-01-15T16:57:17.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-27 td/osd-backfill-space/datafile 2021-01-15T16:57:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-28 td/osd-backfill-space/datafile 2021-01-15T16:57:17.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-29 td/osd-backfill-space/datafile 2021-01-15T16:57:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-30 td/osd-backfill-space/datafile 2021-01-15T16:57:17.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-31 td/osd-backfill-space/datafile 2021-01-15T16:57:17.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-32 td/osd-backfill-space/datafile 2021-01-15T16:57:17.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-33 td/osd-backfill-space/datafile 2021-01-15T16:57:17.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-34 td/osd-backfill-space/datafile 2021-01-15T16:57:17.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:17.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-35 td/osd-backfill-space/datafile 2021-01-15T16:57:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-36 td/osd-backfill-space/datafile 2021-01-15T16:57:18.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-37 td/osd-backfill-space/datafile 2021-01-15T16:57:18.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-38 td/osd-backfill-space/datafile 2021-01-15T16:57:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-39 td/osd-backfill-space/datafile 2021-01-15T16:57:18.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-40 td/osd-backfill-space/datafile 2021-01-15T16:57:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-41 td/osd-backfill-space/datafile 2021-01-15T16:57:18.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-42 td/osd-backfill-space/datafile 2021-01-15T16:57:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-43 td/osd-backfill-space/datafile 2021-01-15T16:57:18.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-44 td/osd-backfill-space/datafile 2021-01-15T16:57:18.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-45 td/osd-backfill-space/datafile 2021-01-15T16:57:18.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:18.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-46 td/osd-backfill-space/datafile 2021-01-15T16:57:19.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-47 td/osd-backfill-space/datafile 2021-01-15T16:57:19.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-48 td/osd-backfill-space/datafile 2021-01-15T16:57:19.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-49 td/osd-backfill-space/datafile 2021-01-15T16:57:19.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-50 td/osd-backfill-space/datafile 2021-01-15T16:57:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-51 td/osd-backfill-space/datafile 2021-01-15T16:57:19.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-52 td/osd-backfill-space/datafile 2021-01-15T16:57:19.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-53 td/osd-backfill-space/datafile 2021-01-15T16:57:19.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-54 td/osd-backfill-space/datafile 2021-01-15T16:57:19.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-55 td/osd-backfill-space/datafile 2021-01-15T16:57:19.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-56 td/osd-backfill-space/datafile 2021-01-15T16:57:19.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-57 td/osd-backfill-space/datafile 2021-01-15T16:57:19.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-58 td/osd-backfill-space/datafile 2021-01-15T16:57:19.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:19.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-59 td/osd-backfill-space/datafile 2021-01-15T16:57:20.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-60 td/osd-backfill-space/datafile 2021-01-15T16:57:20.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-61 td/osd-backfill-space/datafile 2021-01-15T16:57:20.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-62 td/osd-backfill-space/datafile 2021-01-15T16:57:20.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-63 td/osd-backfill-space/datafile 2021-01-15T16:57:20.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-64 td/osd-backfill-space/datafile 2021-01-15T16:57:20.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-65 td/osd-backfill-space/datafile 2021-01-15T16:57:20.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-66 td/osd-backfill-space/datafile 2021-01-15T16:57:20.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-67 td/osd-backfill-space/datafile 2021-01-15T16:57:20.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-68 td/osd-backfill-space/datafile 2021-01-15T16:57:20.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-69 td/osd-backfill-space/datafile 2021-01-15T16:57:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-70 td/osd-backfill-space/datafile 2021-01-15T16:57:21.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-71 td/osd-backfill-space/datafile 2021-01-15T16:57:21.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-72 td/osd-backfill-space/datafile 2021-01-15T16:57:21.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-73 td/osd-backfill-space/datafile 2021-01-15T16:57:21.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-74 td/osd-backfill-space/datafile 2021-01-15T16:57:21.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-75 td/osd-backfill-space/datafile 2021-01-15T16:57:21.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-76 td/osd-backfill-space/datafile 2021-01-15T16:57:21.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-77 td/osd-backfill-space/datafile 2021-01-15T16:57:21.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-78 td/osd-backfill-space/datafile 2021-01-15T16:57:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-79 td/osd-backfill-space/datafile 2021-01-15T16:57:21.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-80 td/osd-backfill-space/datafile 2021-01-15T16:57:21.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-81 td/osd-backfill-space/datafile 2021-01-15T16:57:21.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:21.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-82 td/osd-backfill-space/datafile 2021-01-15T16:57:22.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-83 td/osd-backfill-space/datafile 2021-01-15T16:57:22.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-84 td/osd-backfill-space/datafile 2021-01-15T16:57:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-85 td/osd-backfill-space/datafile 2021-01-15T16:57:22.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-86 td/osd-backfill-space/datafile 2021-01-15T16:57:22.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-87 td/osd-backfill-space/datafile 2021-01-15T16:57:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-88 td/osd-backfill-space/datafile 2021-01-15T16:57:22.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-89 td/osd-backfill-space/datafile 2021-01-15T16:57:22.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-90 td/osd-backfill-space/datafile 2021-01-15T16:57:22.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-91 td/osd-backfill-space/datafile 2021-01-15T16:57:22.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-92 td/osd-backfill-space/datafile 2021-01-15T16:57:22.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-93 td/osd-backfill-space/datafile 2021-01-15T16:57:22.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-94 td/osd-backfill-space/datafile 2021-01-15T16:57:23.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-95 td/osd-backfill-space/datafile 2021-01-15T16:57:23.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-96 td/osd-backfill-space/datafile 2021-01-15T16:57:23.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-97 td/osd-backfill-space/datafile 2021-01-15T16:57:23.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-98 td/osd-backfill-space/datafile 2021-01-15T16:57:23.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-99 td/osd-backfill-space/datafile 2021-01-15T16:57:23.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-100 td/osd-backfill-space/datafile 2021-01-15T16:57:23.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-101 td/osd-backfill-space/datafile 2021-01-15T16:57:23.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-102 td/osd-backfill-space/datafile 2021-01-15T16:57:23.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-103 td/osd-backfill-space/datafile 2021-01-15T16:57:23.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-104 td/osd-backfill-space/datafile 2021-01-15T16:57:23.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:23.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-105 td/osd-backfill-space/datafile 2021-01-15T16:57:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-106 td/osd-backfill-space/datafile 2021-01-15T16:57:24.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-107 td/osd-backfill-space/datafile 2021-01-15T16:57:24.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-108 td/osd-backfill-space/datafile 2021-01-15T16:57:24.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-109 td/osd-backfill-space/datafile 2021-01-15T16:57:24.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-110 td/osd-backfill-space/datafile 2021-01-15T16:57:24.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-111 td/osd-backfill-space/datafile 2021-01-15T16:57:24.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-112 td/osd-backfill-space/datafile 2021-01-15T16:57:24.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-113 td/osd-backfill-space/datafile 2021-01-15T16:57:24.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-114 td/osd-backfill-space/datafile 2021-01-15T16:57:24.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-115 td/osd-backfill-space/datafile 2021-01-15T16:57:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-116 td/osd-backfill-space/datafile 2021-01-15T16:57:24.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:24.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-117 td/osd-backfill-space/datafile 2021-01-15T16:57:25.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-118 td/osd-backfill-space/datafile 2021-01-15T16:57:25.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-119 td/osd-backfill-space/datafile 2021-01-15T16:57:25.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-120 td/osd-backfill-space/datafile 2021-01-15T16:57:25.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-121 td/osd-backfill-space/datafile 2021-01-15T16:57:25.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-122 td/osd-backfill-space/datafile 2021-01-15T16:57:25.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-123 td/osd-backfill-space/datafile 2021-01-15T16:57:25.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-124 td/osd-backfill-space/datafile 2021-01-15T16:57:25.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-125 td/osd-backfill-space/datafile 2021-01-15T16:57:25.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-126 td/osd-backfill-space/datafile 2021-01-15T16:57:25.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-127 td/osd-backfill-space/datafile 2021-01-15T16:57:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-128 td/osd-backfill-space/datafile 2021-01-15T16:57:25.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:25.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-129 td/osd-backfill-space/datafile 2021-01-15T16:57:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-130 td/osd-backfill-space/datafile 2021-01-15T16:57:26.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-131 td/osd-backfill-space/datafile 2021-01-15T16:57:26.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-132 td/osd-backfill-space/datafile 2021-01-15T16:57:26.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-133 td/osd-backfill-space/datafile 2021-01-15T16:57:26.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-134 td/osd-backfill-space/datafile 2021-01-15T16:57:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-135 td/osd-backfill-space/datafile 2021-01-15T16:57:26.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-136 td/osd-backfill-space/datafile 2021-01-15T16:57:26.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-137 td/osd-backfill-space/datafile 2021-01-15T16:57:26.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-138 td/osd-backfill-space/datafile 2021-01-15T16:57:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-139 td/osd-backfill-space/datafile 2021-01-15T16:57:26.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-140 td/osd-backfill-space/datafile 2021-01-15T16:57:26.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-141 td/osd-backfill-space/datafile 2021-01-15T16:57:26.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:26.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-142 td/osd-backfill-space/datafile 2021-01-15T16:57:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-143 td/osd-backfill-space/datafile 2021-01-15T16:57:27.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-144 td/osd-backfill-space/datafile 2021-01-15T16:57:27.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-145 td/osd-backfill-space/datafile 2021-01-15T16:57:27.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-146 td/osd-backfill-space/datafile 2021-01-15T16:57:27.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-147 td/osd-backfill-space/datafile 2021-01-15T16:57:27.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-148 td/osd-backfill-space/datafile 2021-01-15T16:57:27.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-149 td/osd-backfill-space/datafile 2021-01-15T16:57:27.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-150 td/osd-backfill-space/datafile 2021-01-15T16:57:27.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-151 td/osd-backfill-space/datafile 2021-01-15T16:57:27.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-152 td/osd-backfill-space/datafile 2021-01-15T16:57:27.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-153 td/osd-backfill-space/datafile 2021-01-15T16:57:27.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:27.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-154 td/osd-backfill-space/datafile 2021-01-15T16:57:28.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-155 td/osd-backfill-space/datafile 2021-01-15T16:57:28.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-156 td/osd-backfill-space/datafile 2021-01-15T16:57:28.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-157 td/osd-backfill-space/datafile 2021-01-15T16:57:28.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-158 td/osd-backfill-space/datafile 2021-01-15T16:57:28.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-159 td/osd-backfill-space/datafile 2021-01-15T16:57:28.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-160 td/osd-backfill-space/datafile 2021-01-15T16:57:28.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-161 td/osd-backfill-space/datafile 2021-01-15T16:57:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-162 td/osd-backfill-space/datafile 2021-01-15T16:57:28.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-163 td/osd-backfill-space/datafile 2021-01-15T16:57:28.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:28.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-164 td/osd-backfill-space/datafile 2021-01-15T16:57:29.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-165 td/osd-backfill-space/datafile 2021-01-15T16:57:29.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-166 td/osd-backfill-space/datafile 2021-01-15T16:57:29.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-167 td/osd-backfill-space/datafile 2021-01-15T16:57:29.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-168 td/osd-backfill-space/datafile 2021-01-15T16:57:29.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-169 td/osd-backfill-space/datafile 2021-01-15T16:57:29.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-170 td/osd-backfill-space/datafile 2021-01-15T16:57:29.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-171 td/osd-backfill-space/datafile 2021-01-15T16:57:29.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-172 td/osd-backfill-space/datafile 2021-01-15T16:57:29.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-173 td/osd-backfill-space/datafile 2021-01-15T16:57:29.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-174 td/osd-backfill-space/datafile 2021-01-15T16:57:29.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-175 td/osd-backfill-space/datafile 2021-01-15T16:57:29.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-176 td/osd-backfill-space/datafile 2021-01-15T16:57:29.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:29.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-177 td/osd-backfill-space/datafile 2021-01-15T16:57:30.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-178 td/osd-backfill-space/datafile 2021-01-15T16:57:30.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-179 td/osd-backfill-space/datafile 2021-01-15T16:57:30.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-180 td/osd-backfill-space/datafile 2021-01-15T16:57:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-181 td/osd-backfill-space/datafile 2021-01-15T16:57:30.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-182 td/osd-backfill-space/datafile 2021-01-15T16:57:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-183 td/osd-backfill-space/datafile 2021-01-15T16:57:30.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-184 td/osd-backfill-space/datafile 2021-01-15T16:57:30.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-185 td/osd-backfill-space/datafile 2021-01-15T16:57:30.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-186 td/osd-backfill-space/datafile 2021-01-15T16:57:30.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:30.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-187 td/osd-backfill-space/datafile 2021-01-15T16:57:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-188 td/osd-backfill-space/datafile 2021-01-15T16:57:31.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-189 td/osd-backfill-space/datafile 2021-01-15T16:57:31.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-190 td/osd-backfill-space/datafile 2021-01-15T16:57:31.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-191 td/osd-backfill-space/datafile 2021-01-15T16:57:31.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-192 td/osd-backfill-space/datafile 2021-01-15T16:57:31.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-193 td/osd-backfill-space/datafile 2021-01-15T16:57:31.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-194 td/osd-backfill-space/datafile 2021-01-15T16:57:31.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-195 td/osd-backfill-space/datafile 2021-01-15T16:57:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-196 td/osd-backfill-space/datafile 2021-01-15T16:57:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-197 td/osd-backfill-space/datafile 2021-01-15T16:57:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-198 td/osd-backfill-space/datafile 2021-01-15T16:57:31.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-199 td/osd-backfill-space/datafile 2021-01-15T16:57:31.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:31.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-200 td/osd-backfill-space/datafile 2021-01-15T16:57:32.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-201 td/osd-backfill-space/datafile 2021-01-15T16:57:32.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-202 td/osd-backfill-space/datafile 2021-01-15T16:57:32.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-203 td/osd-backfill-space/datafile 2021-01-15T16:57:32.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-204 td/osd-backfill-space/datafile 2021-01-15T16:57:32.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-205 td/osd-backfill-space/datafile 2021-01-15T16:57:32.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-206 td/osd-backfill-space/datafile 2021-01-15T16:57:32.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-207 td/osd-backfill-space/datafile 2021-01-15T16:57:32.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-208 td/osd-backfill-space/datafile 2021-01-15T16:57:32.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-209 td/osd-backfill-space/datafile 2021-01-15T16:57:32.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-210 td/osd-backfill-space/datafile 2021-01-15T16:57:32.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-211 td/osd-backfill-space/datafile 2021-01-15T16:57:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-212 td/osd-backfill-space/datafile 2021-01-15T16:57:33.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-213 td/osd-backfill-space/datafile 2021-01-15T16:57:33.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-214 td/osd-backfill-space/datafile 2021-01-15T16:57:33.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-215 td/osd-backfill-space/datafile 2021-01-15T16:57:33.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-216 td/osd-backfill-space/datafile 2021-01-15T16:57:33.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-217 td/osd-backfill-space/datafile 2021-01-15T16:57:33.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-218 td/osd-backfill-space/datafile 2021-01-15T16:57:33.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-219 td/osd-backfill-space/datafile 2021-01-15T16:57:33.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-220 td/osd-backfill-space/datafile 2021-01-15T16:57:33.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-221 td/osd-backfill-space/datafile 2021-01-15T16:57:33.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-222 td/osd-backfill-space/datafile 2021-01-15T16:57:33.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-223 td/osd-backfill-space/datafile 2021-01-15T16:57:33.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:33.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-224 td/osd-backfill-space/datafile 2021-01-15T16:57:34.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-225 td/osd-backfill-space/datafile 2021-01-15T16:57:34.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-226 td/osd-backfill-space/datafile 2021-01-15T16:57:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-227 td/osd-backfill-space/datafile 2021-01-15T16:57:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-228 td/osd-backfill-space/datafile 2021-01-15T16:57:34.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-229 td/osd-backfill-space/datafile 2021-01-15T16:57:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-230 td/osd-backfill-space/datafile 2021-01-15T16:57:34.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-231 td/osd-backfill-space/datafile 2021-01-15T16:57:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-232 td/osd-backfill-space/datafile 2021-01-15T16:57:34.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-233 td/osd-backfill-space/datafile 2021-01-15T16:57:34.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-234 td/osd-backfill-space/datafile 2021-01-15T16:57:34.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:34.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-235 td/osd-backfill-space/datafile 2021-01-15T16:57:35.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-236 td/osd-backfill-space/datafile 2021-01-15T16:57:35.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-237 td/osd-backfill-space/datafile 2021-01-15T16:57:35.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-238 td/osd-backfill-space/datafile 2021-01-15T16:57:35.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-239 td/osd-backfill-space/datafile 2021-01-15T16:57:35.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-240 td/osd-backfill-space/datafile 2021-01-15T16:57:35.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-241 td/osd-backfill-space/datafile 2021-01-15T16:57:35.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-242 td/osd-backfill-space/datafile 2021-01-15T16:57:35.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-243 td/osd-backfill-space/datafile 2021-01-15T16:57:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-244 td/osd-backfill-space/datafile 2021-01-15T16:57:35.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-245 td/osd-backfill-space/datafile 2021-01-15T16:57:35.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-246 td/osd-backfill-space/datafile 2021-01-15T16:57:35.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-247 td/osd-backfill-space/datafile 2021-01-15T16:57:35.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:35.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-248 td/osd-backfill-space/datafile 2021-01-15T16:57:36.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-249 td/osd-backfill-space/datafile 2021-01-15T16:57:36.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-250 td/osd-backfill-space/datafile 2021-01-15T16:57:36.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-251 td/osd-backfill-space/datafile 2021-01-15T16:57:36.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-252 td/osd-backfill-space/datafile 2021-01-15T16:57:36.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-253 td/osd-backfill-space/datafile 2021-01-15T16:57:36.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-254 td/osd-backfill-space/datafile 2021-01-15T16:57:36.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-255 td/osd-backfill-space/datafile 2021-01-15T16:57:36.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-256 td/osd-backfill-space/datafile 2021-01-15T16:57:36.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-257 td/osd-backfill-space/datafile 2021-01-15T16:57:36.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-258 td/osd-backfill-space/datafile 2021-01-15T16:57:36.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:36.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-259 td/osd-backfill-space/datafile 2021-01-15T16:57:37.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-260 td/osd-backfill-space/datafile 2021-01-15T16:57:37.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-261 td/osd-backfill-space/datafile 2021-01-15T16:57:37.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-262 td/osd-backfill-space/datafile 2021-01-15T16:57:37.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-263 td/osd-backfill-space/datafile 2021-01-15T16:57:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-264 td/osd-backfill-space/datafile 2021-01-15T16:57:37.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-265 td/osd-backfill-space/datafile 2021-01-15T16:57:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-266 td/osd-backfill-space/datafile 2021-01-15T16:57:37.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-267 td/osd-backfill-space/datafile 2021-01-15T16:57:37.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-268 td/osd-backfill-space/datafile 2021-01-15T16:57:37.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-269 td/osd-backfill-space/datafile 2021-01-15T16:57:37.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-270 td/osd-backfill-space/datafile 2021-01-15T16:57:37.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-271 td/osd-backfill-space/datafile 2021-01-15T16:57:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-272 td/osd-backfill-space/datafile 2021-01-15T16:57:38.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-273 td/osd-backfill-space/datafile 2021-01-15T16:57:38.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-274 td/osd-backfill-space/datafile 2021-01-15T16:57:38.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-275 td/osd-backfill-space/datafile 2021-01-15T16:57:38.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-276 td/osd-backfill-space/datafile 2021-01-15T16:57:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-277 td/osd-backfill-space/datafile 2021-01-15T16:57:38.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:38.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-278 td/osd-backfill-space/datafile 2021-01-15T16:57:39.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-279 td/osd-backfill-space/datafile 2021-01-15T16:57:39.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-280 td/osd-backfill-space/datafile 2021-01-15T16:57:39.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-281 td/osd-backfill-space/datafile 2021-01-15T16:57:39.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-282 td/osd-backfill-space/datafile 2021-01-15T16:57:39.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-283 td/osd-backfill-space/datafile 2021-01-15T16:57:39.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-284 td/osd-backfill-space/datafile 2021-01-15T16:57:39.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-285 td/osd-backfill-space/datafile 2021-01-15T16:57:39.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-286 td/osd-backfill-space/datafile 2021-01-15T16:57:39.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-287 td/osd-backfill-space/datafile 2021-01-15T16:57:39.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-288 td/osd-backfill-space/datafile 2021-01-15T16:57:39.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-289 td/osd-backfill-space/datafile 2021-01-15T16:57:39.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-290 td/osd-backfill-space/datafile 2021-01-15T16:57:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-291 td/osd-backfill-space/datafile 2021-01-15T16:57:40.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-292 td/osd-backfill-space/datafile 2021-01-15T16:57:40.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-293 td/osd-backfill-space/datafile 2021-01-15T16:57:40.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-294 td/osd-backfill-space/datafile 2021-01-15T16:57:40.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-295 td/osd-backfill-space/datafile 2021-01-15T16:57:40.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-296 td/osd-backfill-space/datafile 2021-01-15T16:57:40.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-297 td/osd-backfill-space/datafile 2021-01-15T16:57:40.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-298 td/osd-backfill-space/datafile 2021-01-15T16:57:40.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-299 td/osd-backfill-space/datafile 2021-01-15T16:57:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-300 td/osd-backfill-space/datafile 2021-01-15T16:57:40.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:40.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-301 td/osd-backfill-space/datafile 2021-01-15T16:57:41.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-302 td/osd-backfill-space/datafile 2021-01-15T16:57:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-303 td/osd-backfill-space/datafile 2021-01-15T16:57:41.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-304 td/osd-backfill-space/datafile 2021-01-15T16:57:41.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-305 td/osd-backfill-space/datafile 2021-01-15T16:57:41.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-306 td/osd-backfill-space/datafile 2021-01-15T16:57:41.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-307 td/osd-backfill-space/datafile 2021-01-15T16:57:41.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-308 td/osd-backfill-space/datafile 2021-01-15T16:57:41.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-309 td/osd-backfill-space/datafile 2021-01-15T16:57:41.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-310 td/osd-backfill-space/datafile 2021-01-15T16:57:41.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-311 td/osd-backfill-space/datafile 2021-01-15T16:57:41.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-312 td/osd-backfill-space/datafile 2021-01-15T16:57:41.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-313 td/osd-backfill-space/datafile 2021-01-15T16:57:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-314 td/osd-backfill-space/datafile 2021-01-15T16:57:42.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-315 td/osd-backfill-space/datafile 2021-01-15T16:57:42.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-316 td/osd-backfill-space/datafile 2021-01-15T16:57:42.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-317 td/osd-backfill-space/datafile 2021-01-15T16:57:42.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-318 td/osd-backfill-space/datafile 2021-01-15T16:57:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-319 td/osd-backfill-space/datafile 2021-01-15T16:57:42.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-320 td/osd-backfill-space/datafile 2021-01-15T16:57:42.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-321 td/osd-backfill-space/datafile 2021-01-15T16:57:42.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-322 td/osd-backfill-space/datafile 2021-01-15T16:57:42.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-323 td/osd-backfill-space/datafile 2021-01-15T16:57:42.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-324 td/osd-backfill-space/datafile 2021-01-15T16:57:42.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:42.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-325 td/osd-backfill-space/datafile 2021-01-15T16:57:43.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-326 td/osd-backfill-space/datafile 2021-01-15T16:57:43.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-327 td/osd-backfill-space/datafile 2021-01-15T16:57:43.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-328 td/osd-backfill-space/datafile 2021-01-15T16:57:43.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-329 td/osd-backfill-space/datafile 2021-01-15T16:57:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-330 td/osd-backfill-space/datafile 2021-01-15T16:57:43.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-331 td/osd-backfill-space/datafile 2021-01-15T16:57:43.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-332 td/osd-backfill-space/datafile 2021-01-15T16:57:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-333 td/osd-backfill-space/datafile 2021-01-15T16:57:43.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-334 td/osd-backfill-space/datafile 2021-01-15T16:57:43.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-335 td/osd-backfill-space/datafile 2021-01-15T16:57:43.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-336 td/osd-backfill-space/datafile 2021-01-15T16:57:43.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:43.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-337 td/osd-backfill-space/datafile 2021-01-15T16:57:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-338 td/osd-backfill-space/datafile 2021-01-15T16:57:44.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-339 td/osd-backfill-space/datafile 2021-01-15T16:57:44.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-340 td/osd-backfill-space/datafile 2021-01-15T16:57:44.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-341 td/osd-backfill-space/datafile 2021-01-15T16:57:44.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-342 td/osd-backfill-space/datafile 2021-01-15T16:57:44.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-343 td/osd-backfill-space/datafile 2021-01-15T16:57:44.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-344 td/osd-backfill-space/datafile 2021-01-15T16:57:44.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-345 td/osd-backfill-space/datafile 2021-01-15T16:57:44.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-346 td/osd-backfill-space/datafile 2021-01-15T16:57:44.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-347 td/osd-backfill-space/datafile 2021-01-15T16:57:44.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:44.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-348 td/osd-backfill-space/datafile 2021-01-15T16:57:45.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-349 td/osd-backfill-space/datafile 2021-01-15T16:57:45.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-350 td/osd-backfill-space/datafile 2021-01-15T16:57:45.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-351 td/osd-backfill-space/datafile 2021-01-15T16:57:45.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-352 td/osd-backfill-space/datafile 2021-01-15T16:57:45.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-353 td/osd-backfill-space/datafile 2021-01-15T16:57:45.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-354 td/osd-backfill-space/datafile 2021-01-15T16:57:45.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-355 td/osd-backfill-space/datafile 2021-01-15T16:57:45.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-356 td/osd-backfill-space/datafile 2021-01-15T16:57:45.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-357 td/osd-backfill-space/datafile 2021-01-15T16:57:45.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-358 td/osd-backfill-space/datafile 2021-01-15T16:57:45.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-359 td/osd-backfill-space/datafile 2021-01-15T16:57:45.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-360 td/osd-backfill-space/datafile 2021-01-15T16:57:45.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:45.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-361 td/osd-backfill-space/datafile 2021-01-15T16:57:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-362 td/osd-backfill-space/datafile 2021-01-15T16:57:46.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-363 td/osd-backfill-space/datafile 2021-01-15T16:57:46.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-364 td/osd-backfill-space/datafile 2021-01-15T16:57:46.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-365 td/osd-backfill-space/datafile 2021-01-15T16:57:46.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-366 td/osd-backfill-space/datafile 2021-01-15T16:57:46.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-367 td/osd-backfill-space/datafile 2021-01-15T16:57:46.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-368 td/osd-backfill-space/datafile 2021-01-15T16:57:46.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-369 td/osd-backfill-space/datafile 2021-01-15T16:57:46.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-370 td/osd-backfill-space/datafile 2021-01-15T16:57:46.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-371 td/osd-backfill-space/datafile 2021-01-15T16:57:46.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-372 td/osd-backfill-space/datafile 2021-01-15T16:57:46.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:46.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-373 td/osd-backfill-space/datafile 2021-01-15T16:57:47.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-374 td/osd-backfill-space/datafile 2021-01-15T16:57:47.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-375 td/osd-backfill-space/datafile 2021-01-15T16:57:47.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-376 td/osd-backfill-space/datafile 2021-01-15T16:57:47.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-377 td/osd-backfill-space/datafile 2021-01-15T16:57:47.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-378 td/osd-backfill-space/datafile 2021-01-15T16:57:47.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-379 td/osd-backfill-space/datafile 2021-01-15T16:57:47.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-380 td/osd-backfill-space/datafile 2021-01-15T16:57:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-381 td/osd-backfill-space/datafile 2021-01-15T16:57:47.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-382 td/osd-backfill-space/datafile 2021-01-15T16:57:47.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-383 td/osd-backfill-space/datafile 2021-01-15T16:57:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-384 td/osd-backfill-space/datafile 2021-01-15T16:57:47.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-385 td/osd-backfill-space/datafile 2021-01-15T16:57:47.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:47.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-386 td/osd-backfill-space/datafile 2021-01-15T16:57:48.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-387 td/osd-backfill-space/datafile 2021-01-15T16:57:48.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-388 td/osd-backfill-space/datafile 2021-01-15T16:57:48.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-389 td/osd-backfill-space/datafile 2021-01-15T16:57:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-390 td/osd-backfill-space/datafile 2021-01-15T16:57:48.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-391 td/osd-backfill-space/datafile 2021-01-15T16:57:48.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-392 td/osd-backfill-space/datafile 2021-01-15T16:57:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-393 td/osd-backfill-space/datafile 2021-01-15T16:57:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-394 td/osd-backfill-space/datafile 2021-01-15T16:57:48.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-395 td/osd-backfill-space/datafile 2021-01-15T16:57:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-396 td/osd-backfill-space/datafile 2021-01-15T16:57:48.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-397 td/osd-backfill-space/datafile 2021-01-15T16:57:49.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-398 td/osd-backfill-space/datafile 2021-01-15T16:57:49.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-399 td/osd-backfill-space/datafile 2021-01-15T16:57:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-400 td/osd-backfill-space/datafile 2021-01-15T16:57:49.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-401 td/osd-backfill-space/datafile 2021-01-15T16:57:49.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-402 td/osd-backfill-space/datafile 2021-01-15T16:57:49.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-403 td/osd-backfill-space/datafile 2021-01-15T16:57:49.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-404 td/osd-backfill-space/datafile 2021-01-15T16:57:49.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-405 td/osd-backfill-space/datafile 2021-01-15T16:57:49.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-406 td/osd-backfill-space/datafile 2021-01-15T16:57:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-407 td/osd-backfill-space/datafile 2021-01-15T16:57:49.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-408 td/osd-backfill-space/datafile 2021-01-15T16:57:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-409 td/osd-backfill-space/datafile 2021-01-15T16:57:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-410 td/osd-backfill-space/datafile 2021-01-15T16:57:50.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-411 td/osd-backfill-space/datafile 2021-01-15T16:57:50.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-412 td/osd-backfill-space/datafile 2021-01-15T16:57:50.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-413 td/osd-backfill-space/datafile 2021-01-15T16:57:50.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-414 td/osd-backfill-space/datafile 2021-01-15T16:57:50.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-415 td/osd-backfill-space/datafile 2021-01-15T16:57:50.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-416 td/osd-backfill-space/datafile 2021-01-15T16:57:50.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-417 td/osd-backfill-space/datafile 2021-01-15T16:57:50.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-418 td/osd-backfill-space/datafile 2021-01-15T16:57:50.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-419 td/osd-backfill-space/datafile 2021-01-15T16:57:50.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:50.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-420 td/osd-backfill-space/datafile 2021-01-15T16:57:51.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-421 td/osd-backfill-space/datafile 2021-01-15T16:57:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-422 td/osd-backfill-space/datafile 2021-01-15T16:57:51.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-423 td/osd-backfill-space/datafile 2021-01-15T16:57:51.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-424 td/osd-backfill-space/datafile 2021-01-15T16:57:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-425 td/osd-backfill-space/datafile 2021-01-15T16:57:51.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-426 td/osd-backfill-space/datafile 2021-01-15T16:57:51.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-427 td/osd-backfill-space/datafile 2021-01-15T16:57:51.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-428 td/osd-backfill-space/datafile 2021-01-15T16:57:51.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-429 td/osd-backfill-space/datafile 2021-01-15T16:57:51.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-430 td/osd-backfill-space/datafile 2021-01-15T16:57:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-431 td/osd-backfill-space/datafile 2021-01-15T16:57:51.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:51.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-432 td/osd-backfill-space/datafile 2021-01-15T16:57:52.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-433 td/osd-backfill-space/datafile 2021-01-15T16:57:52.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-434 td/osd-backfill-space/datafile 2021-01-15T16:57:52.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-435 td/osd-backfill-space/datafile 2021-01-15T16:57:52.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-436 td/osd-backfill-space/datafile 2021-01-15T16:57:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-437 td/osd-backfill-space/datafile 2021-01-15T16:57:52.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-438 td/osd-backfill-space/datafile 2021-01-15T16:57:52.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-439 td/osd-backfill-space/datafile 2021-01-15T16:57:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-440 td/osd-backfill-space/datafile 2021-01-15T16:57:52.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-441 td/osd-backfill-space/datafile 2021-01-15T16:57:52.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-442 td/osd-backfill-space/datafile 2021-01-15T16:57:52.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:52.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-443 td/osd-backfill-space/datafile 2021-01-15T16:57:53.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-444 td/osd-backfill-space/datafile 2021-01-15T16:57:53.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-445 td/osd-backfill-space/datafile 2021-01-15T16:57:53.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-446 td/osd-backfill-space/datafile 2021-01-15T16:57:53.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-447 td/osd-backfill-space/datafile 2021-01-15T16:57:53.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-448 td/osd-backfill-space/datafile 2021-01-15T16:57:53.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-449 td/osd-backfill-space/datafile 2021-01-15T16:57:53.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-450 td/osd-backfill-space/datafile 2021-01-15T16:57:53.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-451 td/osd-backfill-space/datafile 2021-01-15T16:57:53.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-452 td/osd-backfill-space/datafile 2021-01-15T16:57:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-453 td/osd-backfill-space/datafile 2021-01-15T16:57:53.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-454 td/osd-backfill-space/datafile 2021-01-15T16:57:53.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:53.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-455 td/osd-backfill-space/datafile 2021-01-15T16:57:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-456 td/osd-backfill-space/datafile 2021-01-15T16:57:54.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-457 td/osd-backfill-space/datafile 2021-01-15T16:57:54.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-458 td/osd-backfill-space/datafile 2021-01-15T16:57:54.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-459 td/osd-backfill-space/datafile 2021-01-15T16:57:54.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-460 td/osd-backfill-space/datafile 2021-01-15T16:57:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-461 td/osd-backfill-space/datafile 2021-01-15T16:57:54.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-462 td/osd-backfill-space/datafile 2021-01-15T16:57:54.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-463 td/osd-backfill-space/datafile 2021-01-15T16:57:54.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-464 td/osd-backfill-space/datafile 2021-01-15T16:57:54.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-465 td/osd-backfill-space/datafile 2021-01-15T16:57:54.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-466 td/osd-backfill-space/datafile 2021-01-15T16:57:54.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:54.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-467 td/osd-backfill-space/datafile 2021-01-15T16:57:55.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-468 td/osd-backfill-space/datafile 2021-01-15T16:57:55.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-469 td/osd-backfill-space/datafile 2021-01-15T16:57:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-470 td/osd-backfill-space/datafile 2021-01-15T16:57:55.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-471 td/osd-backfill-space/datafile 2021-01-15T16:57:55.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-472 td/osd-backfill-space/datafile 2021-01-15T16:57:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-473 td/osd-backfill-space/datafile 2021-01-15T16:57:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-474 td/osd-backfill-space/datafile 2021-01-15T16:57:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-475 td/osd-backfill-space/datafile 2021-01-15T16:57:55.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-476 td/osd-backfill-space/datafile 2021-01-15T16:57:55.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-477 td/osd-backfill-space/datafile 2021-01-15T16:57:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-478 td/osd-backfill-space/datafile 2021-01-15T16:57:55.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:55.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-479 td/osd-backfill-space/datafile 2021-01-15T16:57:56.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-480 td/osd-backfill-space/datafile 2021-01-15T16:57:56.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-481 td/osd-backfill-space/datafile 2021-01-15T16:57:56.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-482 td/osd-backfill-space/datafile 2021-01-15T16:57:56.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-483 td/osd-backfill-space/datafile 2021-01-15T16:57:56.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-484 td/osd-backfill-space/datafile 2021-01-15T16:57:56.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-485 td/osd-backfill-space/datafile 2021-01-15T16:57:56.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-486 td/osd-backfill-space/datafile 2021-01-15T16:57:56.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-487 td/osd-backfill-space/datafile 2021-01-15T16:57:56.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-488 td/osd-backfill-space/datafile 2021-01-15T16:57:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-489 td/osd-backfill-space/datafile 2021-01-15T16:57:56.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-490 td/osd-backfill-space/datafile 2021-01-15T16:57:56.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:56.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-491 td/osd-backfill-space/datafile 2021-01-15T16:57:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-492 td/osd-backfill-space/datafile 2021-01-15T16:57:57.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-493 td/osd-backfill-space/datafile 2021-01-15T16:57:57.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-494 td/osd-backfill-space/datafile 2021-01-15T16:57:57.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-495 td/osd-backfill-space/datafile 2021-01-15T16:57:57.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-496 td/osd-backfill-space/datafile 2021-01-15T16:57:57.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-497 td/osd-backfill-space/datafile 2021-01-15T16:57:57.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-498 td/osd-backfill-space/datafile 2021-01-15T16:57:57.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-499 td/osd-backfill-space/datafile 2021-01-15T16:57:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-500 td/osd-backfill-space/datafile 2021-01-15T16:57:57.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-501 td/osd-backfill-space/datafile 2021-01-15T16:57:57.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-502 td/osd-backfill-space/datafile 2021-01-15T16:57:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:57.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-503 td/osd-backfill-space/datafile 2021-01-15T16:57:58.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-504 td/osd-backfill-space/datafile 2021-01-15T16:57:58.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-505 td/osd-backfill-space/datafile 2021-01-15T16:57:58.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-506 td/osd-backfill-space/datafile 2021-01-15T16:57:58.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-507 td/osd-backfill-space/datafile 2021-01-15T16:57:58.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-508 td/osd-backfill-space/datafile 2021-01-15T16:57:58.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-509 td/osd-backfill-space/datafile 2021-01-15T16:57:58.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-510 td/osd-backfill-space/datafile 2021-01-15T16:57:58.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-511 td/osd-backfill-space/datafile 2021-01-15T16:57:58.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-512 td/osd-backfill-space/datafile 2021-01-15T16:57:58.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-513 td/osd-backfill-space/datafile 2021-01-15T16:57:58.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:58.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-514 td/osd-backfill-space/datafile 2021-01-15T16:57:59.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-515 td/osd-backfill-space/datafile 2021-01-15T16:57:59.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-516 td/osd-backfill-space/datafile 2021-01-15T16:57:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-517 td/osd-backfill-space/datafile 2021-01-15T16:57:59.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-518 td/osd-backfill-space/datafile 2021-01-15T16:57:59.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-519 td/osd-backfill-space/datafile 2021-01-15T16:57:59.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-520 td/osd-backfill-space/datafile 2021-01-15T16:57:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-521 td/osd-backfill-space/datafile 2021-01-15T16:57:59.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-522 td/osd-backfill-space/datafile 2021-01-15T16:57:59.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-523 td/osd-backfill-space/datafile 2021-01-15T16:57:59.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-524 td/osd-backfill-space/datafile 2021-01-15T16:57:59.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-525 td/osd-backfill-space/datafile 2021-01-15T16:57:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-526 td/osd-backfill-space/datafile 2021-01-15T16:57:59.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:57:59.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-527 td/osd-backfill-space/datafile 2021-01-15T16:58:00.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-528 td/osd-backfill-space/datafile 2021-01-15T16:58:00.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-529 td/osd-backfill-space/datafile 2021-01-15T16:58:00.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-530 td/osd-backfill-space/datafile 2021-01-15T16:58:00.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-531 td/osd-backfill-space/datafile 2021-01-15T16:58:00.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-532 td/osd-backfill-space/datafile 2021-01-15T16:58:00.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-533 td/osd-backfill-space/datafile 2021-01-15T16:58:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-534 td/osd-backfill-space/datafile 2021-01-15T16:58:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-535 td/osd-backfill-space/datafile 2021-01-15T16:58:00.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-536 td/osd-backfill-space/datafile 2021-01-15T16:58:00.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:00.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-537 td/osd-backfill-space/datafile 2021-01-15T16:58:01.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-538 td/osd-backfill-space/datafile 2021-01-15T16:58:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-539 td/osd-backfill-space/datafile 2021-01-15T16:58:01.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-540 td/osd-backfill-space/datafile 2021-01-15T16:58:01.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-541 td/osd-backfill-space/datafile 2021-01-15T16:58:01.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-542 td/osd-backfill-space/datafile 2021-01-15T16:58:01.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-543 td/osd-backfill-space/datafile 2021-01-15T16:58:01.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-544 td/osd-backfill-space/datafile 2021-01-15T16:58:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-545 td/osd-backfill-space/datafile 2021-01-15T16:58:01.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-546 td/osd-backfill-space/datafile 2021-01-15T16:58:01.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-547 td/osd-backfill-space/datafile 2021-01-15T16:58:01.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-548 td/osd-backfill-space/datafile 2021-01-15T16:58:01.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-549 td/osd-backfill-space/datafile 2021-01-15T16:58:01.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:01.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-550 td/osd-backfill-space/datafile 2021-01-15T16:58:02.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-551 td/osd-backfill-space/datafile 2021-01-15T16:58:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-552 td/osd-backfill-space/datafile 2021-01-15T16:58:02.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-553 td/osd-backfill-space/datafile 2021-01-15T16:58:02.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-554 td/osd-backfill-space/datafile 2021-01-15T16:58:02.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-555 td/osd-backfill-space/datafile 2021-01-15T16:58:02.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-556 td/osd-backfill-space/datafile 2021-01-15T16:58:02.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-557 td/osd-backfill-space/datafile 2021-01-15T16:58:02.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-558 td/osd-backfill-space/datafile 2021-01-15T16:58:02.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-559 td/osd-backfill-space/datafile 2021-01-15T16:58:02.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-560 td/osd-backfill-space/datafile 2021-01-15T16:58:02.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:02.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-561 td/osd-backfill-space/datafile 2021-01-15T16:58:03.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-562 td/osd-backfill-space/datafile 2021-01-15T16:58:03.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-563 td/osd-backfill-space/datafile 2021-01-15T16:58:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-564 td/osd-backfill-space/datafile 2021-01-15T16:58:03.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-565 td/osd-backfill-space/datafile 2021-01-15T16:58:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-566 td/osd-backfill-space/datafile 2021-01-15T16:58:03.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-567 td/osd-backfill-space/datafile 2021-01-15T16:58:03.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-568 td/osd-backfill-space/datafile 2021-01-15T16:58:03.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-569 td/osd-backfill-space/datafile 2021-01-15T16:58:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-570 td/osd-backfill-space/datafile 2021-01-15T16:58:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-571 td/osd-backfill-space/datafile 2021-01-15T16:58:03.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-572 td/osd-backfill-space/datafile 2021-01-15T16:58:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-573 td/osd-backfill-space/datafile 2021-01-15T16:58:03.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:03.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-574 td/osd-backfill-space/datafile 2021-01-15T16:58:04.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-575 td/osd-backfill-space/datafile 2021-01-15T16:58:04.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-576 td/osd-backfill-space/datafile 2021-01-15T16:58:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-577 td/osd-backfill-space/datafile 2021-01-15T16:58:04.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-578 td/osd-backfill-space/datafile 2021-01-15T16:58:04.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-579 td/osd-backfill-space/datafile 2021-01-15T16:58:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-580 td/osd-backfill-space/datafile 2021-01-15T16:58:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-581 td/osd-backfill-space/datafile 2021-01-15T16:58:04.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-582 td/osd-backfill-space/datafile 2021-01-15T16:58:04.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-583 td/osd-backfill-space/datafile 2021-01-15T16:58:04.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-584 td/osd-backfill-space/datafile 2021-01-15T16:58:04.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:04.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-585 td/osd-backfill-space/datafile 2021-01-15T16:58:05.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-586 td/osd-backfill-space/datafile 2021-01-15T16:58:05.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-587 td/osd-backfill-space/datafile 2021-01-15T16:58:05.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-588 td/osd-backfill-space/datafile 2021-01-15T16:58:05.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-589 td/osd-backfill-space/datafile 2021-01-15T16:58:05.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-590 td/osd-backfill-space/datafile 2021-01-15T16:58:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-591 td/osd-backfill-space/datafile 2021-01-15T16:58:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-592 td/osd-backfill-space/datafile 2021-01-15T16:58:05.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-593 td/osd-backfill-space/datafile 2021-01-15T16:58:05.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-594 td/osd-backfill-space/datafile 2021-01-15T16:58:05.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-595 td/osd-backfill-space/datafile 2021-01-15T16:58:05.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-596 td/osd-backfill-space/datafile 2021-01-15T16:58:05.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-597 td/osd-backfill-space/datafile 2021-01-15T16:58:05.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:05.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-598 td/osd-backfill-space/datafile 2021-01-15T16:58:06.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:06.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-599 td/osd-backfill-space/datafile 2021-01-15T16:58:06.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:462: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:06.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:464: TEST_backfill_multi_partial: rados -p fillpool put obj-fill-600 td/osd-backfill-space/datafile 2021-01-15T16:58:06.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:467: TEST_backfill_multi_partial: get_primary fillpool obj-fill-1 2021-01-15T16:58:06.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2021-01-15T16:58:06.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj-fill-1 2021-01-15T16:58:06.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj-fill-1 2021-01-15T16:58:06.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T16:58:06.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:467: TEST_backfill_multi_partial: local fillosd=1 2021-01-15T16:58:06.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:468: TEST_backfill_multi_partial: expr 1 + 1 2021-01-15T16:58:06.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:468: TEST_backfill_multi_partial: osd=2 2021-01-15T16:58:06.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:469: TEST_backfill_multi_partial: '[' 2 = 3 ']' 2021-01-15T16:58:06.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:473: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2021-01-15T16:58:06.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2021-01-15T16:58:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=209321 2021-01-15T16:58:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2021-01-15T16:58:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2021-01-15T16:58:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2021-01-15T16:58:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T16:58:06.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 209321 2021-01-15T16:58:06.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2021-01-15T16:58:06.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2021-01-15T16:58:06.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2021-01-15T16:58:06.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T16:58:06.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 209321 2021-01-15T16:58:06.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2021-01-15T16:58:06.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2021-01-15T16:58:06.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2021-01-15T16:58:06.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:474: TEST_backfill_multi_partial: ceph osd out osd.1 2021-01-15T16:58:07.615 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T16:58:07.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:476: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op export-remove --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2021-01-15T16:58:07.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2021-01-15T16:58:07.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:58:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:58:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:58:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2021-01-15T16:58:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op export-remove --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2021-01-15T16:58:09.363 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 1.0 info 1.0( v 35'600 (35'500,35'600] local-lis/les=24/25 n=600 ec=21/21 lis/c=24/24 les/c/f=25/25/0 sis=24) 2021-01-15T16:58:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0002d87b:::obj-fill-318:head# 2021-01-15T16:58:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:00676868:::obj-fill-583:head# 2021-01-15T16:58:09.376 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:00f4514e:::obj-fill-547:head# 2021-01-15T16:58:09.377 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:01596a3c:::obj-fill-533:head# 2021-01-15T16:58:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0285e89c:::obj-fill-190:head# 2021-01-15T16:58:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:02bd591e:::obj-fill-463:head# 2021-01-15T16:58:09.379 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:02c01ffe:::obj-fill-28:head# 2021-01-15T16:58:09.379 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:03ada31e:::obj-fill-285:head# 2021-01-15T16:58:09.380 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:03b07c3f:::obj-fill-568:head# 2021-01-15T16:58:09.383 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:03b85770:::obj-fill-573:head# 2021-01-15T16:58:09.384 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:03dcbe89:::obj-fill-375:head# 2021-01-15T16:58:09.390 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04231826:::obj-fill-594:head# 2021-01-15T16:58:09.391 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:04d48684:::obj-fill-421:head# 2021-01-15T16:58:09.392 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:05385fcc:::obj-fill-299:head# 2021-01-15T16:58:09.395 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:05785d38:::obj-fill-159:head# 2021-01-15T16:58:09.395 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:057ec231:::obj-fill-138:head# 2021-01-15T16:58:09.396 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:06058b5e:::obj-fill-569:head# 2021-01-15T16:58:09.396 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:06d2a3d9:::obj-fill-284:head# 2021-01-15T16:58:09.397 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:07597b63:::obj-fill-459:head# 2021-01-15T16:58:09.398 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:075f4069:::obj-fill-550:head# 2021-01-15T16:58:09.398 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:08234052:::obj-fill-17:head# 2021-01-15T16:58:09.399 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:091ce2ec:::obj-fill-472:head# 2021-01-15T16:58:09.400 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:095fd509:::obj-fill-210:head# 2021-01-15T16:58:09.401 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0a52a594:::obj-fill-116:head# 2021-01-15T16:58:09.402 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0a63716b:::obj-fill-225:head# 2021-01-15T16:58:09.402 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0b0fa6fb:::obj-fill-541:head# 2021-01-15T16:58:09.403 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0b673f72:::obj-fill-232:head# 2021-01-15T16:58:09.404 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0bbcd1b2:::obj-fill-582:head# 2021-01-15T16:58:09.404 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0caf8ce3:::obj-fill-214:head# 2021-01-15T16:58:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0d94b9fb:::obj-fill-415:head# 2021-01-15T16:58:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0de4e6f7:::obj-fill-94:head# 2021-01-15T16:58:09.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0ef75768:::obj-fill-489:head# 2021-01-15T16:58:09.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0f67284f:::obj-fill-213:head# 2021-01-15T16:58:09.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:0f823df2:::obj-fill-234:head# 2021-01-15T16:58:09.407 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1005c08d:::obj-fill-490:head# 2021-01-15T16:58:09.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1039525f:::obj-fill-369:head# 2021-01-15T16:58:09.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:105fb36c:::obj-fill-226:head# 2021-01-15T16:58:09.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:111abdeb:::obj-fill-554:head# 2021-01-15T16:58:09.409 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:11392f3d:::obj-fill-453:head# 2021-01-15T16:58:09.411 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1169976c:::obj-fill-417:head# 2021-01-15T16:58:09.412 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:11840a2a:::obj-fill-477:head# 2021-01-15T16:58:09.412 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:11a4832e:::obj-fill-279:head# 2021-01-15T16:58:09.413 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:11a4d2f6:::obj-fill-256:head# 2021-01-15T16:58:09.414 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:12d52a31:::obj-fill-22:head# 2021-01-15T16:58:09.414 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:133c45db:::obj-fill-257:head# 2021-01-15T16:58:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13563256:::obj-fill-150:head# 2021-01-15T16:58:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13c09e00:::obj-fill-71:head# 2021-01-15T16:58:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13e9fbed:::obj-fill-242:head# 2021-01-15T16:58:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:13fae6c2:::obj-fill-163:head# 2021-01-15T16:58:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:145025e0:::obj-fill-543:head# 2021-01-15T16:58:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:14517c84:::obj-fill-13:head# 2021-01-15T16:58:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:14802e9c:::obj-fill-127:head# 2021-01-15T16:58:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:150f1c96:::obj-fill-62:head# 2021-01-15T16:58:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:155e90a2:::obj-fill-537:head# 2021-01-15T16:58:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1632c776:::obj-fill-262:head# 2021-01-15T16:58:09.420 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:163bde28:::obj-fill-523:head# 2021-01-15T16:58:09.420 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:169c4ad0:::obj-fill-85:head# 2021-01-15T16:58:09.421 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:16c4da1d:::obj-fill-436:head# 2021-01-15T16:58:09.421 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17b28c32:::obj-fill-345:head# 2021-01-15T16:58:09.422 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17b34255:::obj-fill-577:head# 2021-01-15T16:58:09.422 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17d2541f:::obj-fill-426:head# 2021-01-15T16:58:09.423 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:17ea19f2:::obj-fill-542:head# 2021-01-15T16:58:09.423 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:189155ea:::obj-fill-496:head# 2021-01-15T16:58:09.424 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:191a6d81:::obj-fill-548:head# 2021-01-15T16:58:09.424 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1a964993:::obj-fill-576:head# 2021-01-15T16:58:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1aa623dc:::obj-fill-356:head# 2021-01-15T16:58:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1abe5c38:::obj-fill-561:head# 2021-01-15T16:58:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1bddc06b:::obj-fill-7:head# 2021-01-15T16:58:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1be86b71:::obj-fill-295:head# 2021-01-15T16:58:09.427 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1c499bca:::obj-fill-240:head# 2021-01-15T16:58:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1c545e27:::obj-fill-509:head# 2021-01-15T16:58:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1c63c8f4:::obj-fill-4:head# 2021-01-15T16:58:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1d5a8f59:::obj-fill-377:head# 2021-01-15T16:58:09.429 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1d85b5df:::obj-fill-201:head# 2021-01-15T16:58:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1d9a0d5b:::obj-fill-581:head# 2021-01-15T16:58:09.430 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1dcf5667:::obj-fill-488:head# 2021-01-15T16:58:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1fa3090d:::obj-fill-458:head# 2021-01-15T16:58:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1fe937f6:::obj-fill-337:head# 2021-01-15T16:58:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:1ff54542:::obj-fill-349:head# 2021-01-15T16:58:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:20d9e0ca:::obj-fill-367:head# 2021-01-15T16:58:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:21b8e3bb:::obj-fill-538:head# 2021-01-15T16:58:09.433 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:227183d4:::obj-fill-235:head# 2021-01-15T16:58:09.434 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:229e5e5d:::obj-fill-522:head# 2021-01-15T16:58:09.435 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:234f8861:::obj-fill-476:head# 2021-01-15T16:58:09.435 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:23ac2c9d:::obj-fill-471:head# 2021-01-15T16:58:09.436 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:25db1742:::obj-fill-200:head# 2021-01-15T16:58:09.436 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:264f8545:::obj-fill-230:head# 2021-01-15T16:58:09.437 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:267ba97f:::obj-fill-493:head# 2021-01-15T16:58:09.437 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:26b067e8:::obj-fill-208:head# 2021-01-15T16:58:09.438 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:26f521cb:::obj-fill-300:head# 2021-01-15T16:58:09.438 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:27e70a11:::obj-fill-368:head# 2021-01-15T16:58:09.438 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:29357d4a:::obj-fill-456:head# 2021-01-15T16:58:09.439 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:29f2567e:::obj-fill-414:head# 2021-01-15T16:58:09.439 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2a10cbb6:::obj-fill-228:head# 2021-01-15T16:58:09.440 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2a1519a6:::obj-fill-148:head# 2021-01-15T16:58:09.440 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2a686ca6:::obj-fill-411:head# 2021-01-15T16:58:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2ab3f384:::obj-fill-77:head# 2021-01-15T16:58:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2ae6a1d0:::obj-fill-187:head# 2021-01-15T16:58:09.442 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2b890959:::obj-fill-222:head# 2021-01-15T16:58:09.442 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2bcc26c4:::obj-fill-114:head# 2021-01-15T16:58:09.443 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2bd3a3a2:::obj-fill-281:head# 2021-01-15T16:58:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2c1a21d7:::obj-fill-505:head# 2021-01-15T16:58:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2c255347:::obj-fill-95:head# 2021-01-15T16:58:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2c2a3f62:::obj-fill-131:head# 2021-01-15T16:58:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2c7d92ce:::obj-fill-442:head# 2021-01-15T16:58:09.445 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2d1ad117:::obj-fill-443:head# 2021-01-15T16:58:09.446 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2d57b2f7:::obj-fill-192:head# 2021-01-15T16:58:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2d7b53ea:::obj-fill-79:head# 2021-01-15T16:58:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2db25266:::obj-fill-121:head# 2021-01-15T16:58:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2e27853e:::obj-fill-473:head# 2021-01-15T16:58:09.448 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2f12cb21:::obj-fill-386:head# 2021-01-15T16:58:09.448 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2f957332:::obj-fill-422:head# 2021-01-15T16:58:09.449 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:2feae493:::obj-fill-301:head# 2021-01-15T16:58:09.449 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:302c95d2:::obj-fill-110:head# 2021-01-15T16:58:09.450 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:302d6b76:::obj-fill-374:head# 2021-01-15T16:58:09.450 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:30581f34:::obj-fill-313:head# 2021-01-15T16:58:09.451 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:307e86ac:::obj-fill-381:head# 2021-01-15T16:58:09.451 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:30ab77bf:::obj-fill-275:head# 2021-01-15T16:58:09.452 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:31127e6d:::obj-fill-250:head# 2021-01-15T16:58:09.452 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:32c93500:::obj-fill-155:head# 2021-01-15T16:58:09.453 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:339880b7:::obj-fill-227:head# 2021-01-15T16:58:09.453 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:34069e1e:::obj-fill-292:head# 2021-01-15T16:58:09.454 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:346dff31:::obj-fill-157:head# 2021-01-15T16:58:09.454 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:34ade0f1:::obj-fill-575:head# 2021-01-15T16:58:09.455 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:34bbae43:::obj-fill-90:head# 2021-01-15T16:58:09.455 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:34e97cf8:::obj-fill-334:head# 2021-01-15T16:58:09.456 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:35d7ccc2:::obj-fill-510:head# 2021-01-15T16:58:09.456 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:35f519b6:::obj-fill-513:head# 2021-01-15T16:58:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:364727e8:::obj-fill-130:head# 2021-01-15T16:58:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:364c0361:::obj-fill-586:head# 2021-01-15T16:58:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:368e7baf:::obj-fill-546:head# 2021-01-15T16:58:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:36ffb224:::obj-fill-82:head# 2021-01-15T16:58:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:371cb800:::obj-fill-330:head# 2021-01-15T16:58:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:37d5378d:::obj-fill-175:head# 2021-01-15T16:58:09.460 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:385593f3:::obj-fill-325:head# 2021-01-15T16:58:09.460 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:388d5073:::obj-fill-317:head# 2021-01-15T16:58:09.460 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:394af079:::obj-fill-59:head# 2021-01-15T16:58:09.461 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3a41ad37:::obj-fill-409:head# 2021-01-15T16:58:09.461 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3b3c63d2:::obj-fill-136:head# 2021-01-15T16:58:09.462 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3bb50131:::obj-fill-512:head# 2021-01-15T16:58:09.462 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3bb52bb5:::obj-fill-188:head# 2021-01-15T16:58:09.463 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3c0683d1:::obj-fill-372:head# 2021-01-15T16:58:09.463 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3c67d3f4:::obj-fill-249:head# 2021-01-15T16:58:09.464 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3c6a4a4c:::obj-fill-495:head# 2021-01-15T16:58:09.464 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3c7fc8c9:::obj-fill-324:head# 2021-01-15T16:58:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3d940c50:::obj-fill-591:head# 2021-01-15T16:58:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3ed5508e:::obj-fill-558:head# 2021-01-15T16:58:09.466 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:3f598316:::obj-fill-389:head# 2021-01-15T16:58:09.466 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:401cd0ce:::obj-fill-251:head# 2021-01-15T16:58:09.467 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:40fae006:::obj-fill-14:head# 2021-01-15T16:58:09.468 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4120b10f:::obj-fill-165:head# 2021-01-15T16:58:09.468 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:41b382c2:::obj-fill-289:head# 2021-01-15T16:58:09.469 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4277cdea:::obj-fill-319:head# 2021-01-15T16:58:09.470 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:42978c3c:::obj-fill-174:head# 2021-01-15T16:58:09.470 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:42b08dc2:::obj-fill-553:head# 2021-01-15T16:58:09.471 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:42ec4bf6:::obj-fill-209:head# 2021-01-15T16:58:09.471 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:430c129d:::obj-fill-448:head# 2021-01-15T16:58:09.472 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4335d553:::obj-fill-600:head# 2021-01-15T16:58:09.472 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:44804f62:::obj-fill-532:head# 2021-01-15T16:58:09.472 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:44954d54:::obj-fill-2:head# 2021-01-15T16:58:09.473 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4502f904:::obj-fill-429:head# 2021-01-15T16:58:09.473 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4568ddcb:::obj-fill-382:head# 2021-01-15T16:58:09.474 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:45939974:::obj-fill-563:head# 2021-01-15T16:58:09.474 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:462a5bbd:::obj-fill-567:head# 2021-01-15T16:58:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:47773ae4:::obj-fill-307:head# 2021-01-15T16:58:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4802b263:::obj-fill-387:head# 2021-01-15T16:58:09.476 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:483d938e:::obj-fill-128:head# 2021-01-15T16:58:09.476 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:48a7ad29:::obj-fill-147:head# 2021-01-15T16:58:09.477 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:493b61ce:::obj-fill-454:head# 2021-01-15T16:58:09.477 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4afb70fe:::obj-fill-186:head# 2021-01-15T16:58:09.478 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4bff08e3:::obj-fill-352:head# 2021-01-15T16:58:09.479 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4c01eabe:::obj-fill-370:head# 2021-01-15T16:58:09.479 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4c131560:::obj-fill-6:head# 2021-01-15T16:58:09.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4c4c5e14:::obj-fill-350:head# 2021-01-15T16:58:09.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4c80dc2c:::obj-fill-357:head# 2021-01-15T16:58:09.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4c9dee9b:::obj-fill-540:head# 2021-01-15T16:58:09.481 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4ce40c18:::obj-fill-280:head# 2021-01-15T16:58:09.481 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4d854fc3:::obj-fill-585:head# 2021-01-15T16:58:09.482 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:4f615b53:::obj-fill-101:head# 2021-01-15T16:58:09.482 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5042b30f:::obj-fill-84:head# 2021-01-15T16:58:09.483 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:507dc08b:::obj-fill-336:head# 2021-01-15T16:58:09.484 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:512fcf5a:::obj-fill-169:head# 2021-01-15T16:58:09.484 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:51a0c8f8:::obj-fill-578:head# 2021-01-15T16:58:09.485 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:51fd05a2:::obj-fill-425:head# 2021-01-15T16:58:09.485 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:52400ebc:::obj-fill-81:head# 2021-01-15T16:58:09.486 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5288d40c:::obj-fill-16:head# 2021-01-15T16:58:09.487 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5295b0ee:::obj-fill-224:head# 2021-01-15T16:58:09.487 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:531e4c60:::obj-fill-103:head# 2021-01-15T16:58:09.488 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:54259f28:::obj-fill-584:head# 2021-01-15T16:58:09.488 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:542d619c:::obj-fill-528:head# 2021-01-15T16:58:09.488 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5503ca3e:::obj-fill-494:head# 2021-01-15T16:58:09.489 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5555c48b:::obj-fill-403:head# 2021-01-15T16:58:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:558d1ec7:::obj-fill-447:head# 2021-01-15T16:58:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:55aa2233:::obj-fill-21:head# 2021-01-15T16:58:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:55aaf8c4:::obj-fill-29:head# 2021-01-15T16:58:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:55e246e1:::obj-fill-65:head# 2021-01-15T16:58:09.492 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:563b14a8:::obj-fill-120:head# 2021-01-15T16:58:09.493 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:56e1df25:::obj-fill-305:head# 2021-01-15T16:58:09.493 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5702979c:::obj-fill-73:head# 2021-01-15T16:58:09.493 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:570722ba:::obj-fill-219:head# 2021-01-15T16:58:09.494 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:57f7a09b:::obj-fill-571:head# 2021-01-15T16:58:09.495 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:585309ba:::obj-fill-144:head# 2021-01-15T16:58:09.495 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:585c7c9c:::obj-fill-437:head# 2021-01-15T16:58:09.495 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:587c63db:::obj-fill-182:head# 2021-01-15T16:58:09.496 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:58f34d6b:::obj-fill-598:head# 2021-01-15T16:58:09.497 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:58f40088:::obj-fill-298:head# 2021-01-15T16:58:09.497 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:591e150c:::obj-fill-32:head# 2021-01-15T16:58:09.497 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:59971ddb:::obj-fill-559:head# 2021-01-15T16:58:09.498 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5adafca4:::obj-fill-56:head# 2021-01-15T16:58:09.498 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5b32b04f:::obj-fill-484:head# 2021-01-15T16:58:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5bdaeb16:::obj-fill-99:head# 2021-01-15T16:58:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5c1d6f4b:::obj-fill-133:head# 2021-01-15T16:58:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5c92b8f1:::obj-fill-255:head# 2021-01-15T16:58:09.500 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5ca4850f:::obj-fill-238:head# 2021-01-15T16:58:09.500 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5d928a5b:::obj-fill-395:head# 2021-01-15T16:58:09.501 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5dd49320:::obj-fill-423:head# 2021-01-15T16:58:09.502 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5e7bc368:::obj-fill-506:head# 2021-01-15T16:58:09.502 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5ebd5ee3:::obj-fill-322:head# 2021-01-15T16:58:09.502 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5edeedf1:::obj-fill-557:head# 2021-01-15T16:58:09.503 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:5facf8e2:::obj-fill-15:head# 2021-01-15T16:58:09.504 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:603a2c76:::obj-fill-439:head# 2021-01-15T16:58:09.504 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:60b30714:::obj-fill-53:head# 2021-01-15T16:58:09.505 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:612b4a8a:::obj-fill-412:head# 2021-01-15T16:58:09.505 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:61a111b6:::obj-fill-87:head# 2021-01-15T16:58:09.506 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:63a0e8a2:::obj-fill-266:head# 2021-01-15T16:58:09.507 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:63aa4d60:::obj-fill-68:head# 2021-01-15T16:58:09.507 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:63ac5b95:::obj-fill-405:head# 2021-01-15T16:58:09.507 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6415a9e3:::obj-fill-400:head# 2021-01-15T16:58:09.508 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:64ba03ae:::obj-fill-20:head# 2021-01-15T16:58:09.509 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6512f015:::obj-fill-597:head# 2021-01-15T16:58:09.509 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:65433104:::obj-fill-504:head# 2021-01-15T16:58:09.510 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:65950c58:::obj-fill-86:head# 2021-01-15T16:58:09.510 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:65b6b2d7:::obj-fill-39:head# 2021-01-15T16:58:09.511 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:661c8afd:::obj-fill-267:head# 2021-01-15T16:58:09.511 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:66f87c94:::obj-fill-229:head# 2021-01-15T16:58:09.512 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:67292cef:::obj-fill-93:head# 2021-01-15T16:58:09.512 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:67f0e12c:::obj-fill-363:head# 2021-01-15T16:58:09.513 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:696eb83b:::obj-fill-185:head# 2021-01-15T16:58:09.514 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6b33981c:::obj-fill-259:head# 2021-01-15T16:58:09.514 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6b5a527c:::obj-fill-485:head# 2021-01-15T16:58:09.515 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6b6f1852:::obj-fill-390:head# 2021-01-15T16:58:09.515 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6ba5c5f8:::obj-fill-304:head# 2021-01-15T16:58:09.516 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6bce929d:::obj-fill-518:head# 2021-01-15T16:58:09.516 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6c8ffc04:::obj-fill-61:head# 2021-01-15T16:58:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6ca7e551:::obj-fill-40:head# 2021-01-15T16:58:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6caff2cd:::obj-fill-332:head# 2021-01-15T16:58:09.518 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6ce4adc7:::obj-fill-574:head# 2021-01-15T16:58:09.518 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6d09d770:::obj-fill-74:head# 2021-01-15T16:58:09.519 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6e039822:::obj-fill-278:head# 2021-01-15T16:58:09.520 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6f2cd109:::obj-fill-297:head# 2021-01-15T16:58:09.520 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6f3754c4:::obj-fill-88:head# 2021-01-15T16:58:09.521 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6f86cfd2:::obj-fill-511:head# 2021-01-15T16:58:09.521 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:6fe3f3f7:::obj-fill-193:head# 2021-01-15T16:58:09.521 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7002c57d:::obj-fill-50:head# 2021-01-15T16:58:09.522 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7030b2a3:::obj-fill-217:head# 2021-01-15T16:58:09.523 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:70665087:::obj-fill-51:head# 2021-01-15T16:58:09.523 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:70a0f5e0:::obj-fill-11:head# 2021-01-15T16:58:09.523 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:71f4643d:::obj-fill-236:head# 2021-01-15T16:58:09.530 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:722fbd0f:::obj-fill-302:head# 2021-01-15T16:58:09.551 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:72584da9:::obj-fill-445:head# 2021-01-15T16:58:09.557 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:72d91d24:::obj-fill-218:head# 2021-01-15T16:58:09.563 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:72f7268b:::obj-fill-106:head# 2021-01-15T16:58:09.564 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:74077abf:::obj-fill-435:head# 2021-01-15T16:58:09.565 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:740ca050:::obj-fill-112:head# 2021-01-15T16:58:09.565 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7435e881:::obj-fill-78:head# 2021-01-15T16:58:09.566 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:74ba0bb7:::obj-fill-478:head# 2021-01-15T16:58:09.566 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:74d82091:::obj-fill-96:head# 2021-01-15T16:58:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7515f56c:::obj-fill-483:head# 2021-01-15T16:58:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7550d6fe:::obj-fill-286:head# 2021-01-15T16:58:09.568 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:75807edc:::obj-fill-223:head# 2021-01-15T16:58:09.568 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:762ba324:::obj-fill-172:head# 2021-01-15T16:58:09.569 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:76369bda:::obj-fill-570:head# 2021-01-15T16:58:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7667a5b0:::obj-fill-196:head# 2021-01-15T16:58:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:77736c0c:::obj-fill-566:head# 2021-01-15T16:58:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:77fb47a4:::obj-fill-314:head# 2021-01-15T16:58:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:785ad894:::obj-fill-80:head# 2021-01-15T16:58:09.572 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:785e3d14:::obj-fill-530:head# 2021-01-15T16:58:09.572 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:788efef4:::obj-fill-595:head# 2021-01-15T16:58:09.573 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:789ec5ab:::obj-fill-44:head# 2021-01-15T16:58:09.573 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:798a3fa8:::obj-fill-517:head# 2021-01-15T16:58:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:79a105ef:::obj-fill-315:head# 2021-01-15T16:58:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:79d02f04:::obj-fill-243:head# 2021-01-15T16:58:09.740 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7a00dfcb:::obj-fill-402:head# 2021-01-15T16:58:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7b982684:::obj-fill-168:head# 2021-01-15T16:58:09.742 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7bb36851:::obj-fill-440:head# 2021-01-15T16:58:09.742 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7bb49d8c:::obj-fill-55:head# 2021-01-15T16:58:09.743 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7c77b783:::obj-fill-580:head# 2021-01-15T16:58:09.743 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7c89d75b:::obj-fill-555:head# 2021-01-15T16:58:09.744 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7d12fb67:::obj-fill-399:head# 2021-01-15T16:58:09.745 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7d139961:::obj-fill-92:head# 2021-01-15T16:58:09.745 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7d39ffd0:::obj-fill-135:head# 2021-01-15T16:58:09.746 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7e534cc4:::obj-fill-282:head# 2021-01-15T16:58:09.746 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7e7a08e0:::obj-fill-108:head# 2021-01-15T16:58:09.747 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7e8e2b67:::obj-fill-244:head# 2021-01-15T16:58:09.748 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7eea215e:::obj-fill-139:head# 2021-01-15T16:58:09.749 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7efa3c6e:::obj-fill-408:head# 2021-01-15T16:58:09.749 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7f03a24b:::obj-fill-45:head# 2021-01-15T16:58:09.750 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7f6a2e2b:::obj-fill-427:head# 2021-01-15T16:58:09.751 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7fb0abd6:::obj-fill-479:head# 2021-01-15T16:58:09.755 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:7fb77de1:::obj-fill-167:head# 2021-01-15T16:58:09.756 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:80012a03:::obj-fill-105:head# 2021-01-15T16:58:09.757 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:80075ae5:::obj-fill-376:head# 2021-01-15T16:58:09.758 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8089baa5:::obj-fill-97:head# 2021-01-15T16:58:09.758 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:80d0b4c5:::obj-fill-123:head# 2021-01-15T16:58:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:810f319f:::obj-fill-552:head# 2021-01-15T16:58:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:81427ef2:::obj-fill-474:head# 2021-01-15T16:58:09.760 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:818d780c:::obj-fill-33:head# 2021-01-15T16:58:09.760 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:819f31b9:::obj-fill-366:head# 2021-01-15T16:58:09.761 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8205549d:::obj-fill-125:head# 2021-01-15T16:58:09.761 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:821f56e5:::obj-fill-339:head# 2021-01-15T16:58:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:822807c6:::obj-fill-69:head# 2021-01-15T16:58:09.763 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:822fcf61:::obj-fill-560:head# 2021-01-15T16:58:09.764 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:82ad696b:::obj-fill-149:head# 2021-01-15T16:58:09.765 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:835cc771:::obj-fill-446:head# 2021-01-15T16:58:09.766 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8386bee1:::obj-fill-137:head# 2021-01-15T16:58:09.766 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:83c1303a:::obj-fill-311:head# 2021-01-15T16:58:09.766 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:83cf1c2a:::obj-fill-360:head# 2021-01-15T16:58:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:83d70321:::obj-fill-351:head# 2021-01-15T16:58:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:83ef162d:::obj-fill-331:head# 2021-01-15T16:58:09.769 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:840f8f20:::obj-fill-398:head# 2021-01-15T16:58:09.769 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:84d2e456:::obj-fill-207:head# 2021-01-15T16:58:09.769 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:856697ff:::obj-fill-333:head# 2021-01-15T16:58:09.771 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:858be937:::obj-fill-418:head# 2021-01-15T16:58:09.771 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:85a0b302:::obj-fill-438:head# 2021-01-15T16:58:09.771 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:85fa942e:::obj-fill-272:head# 2021-01-15T16:58:09.772 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:863eb6f8:::obj-fill-320:head# 2021-01-15T16:58:09.772 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:868d4588:::obj-fill-19:head# 2021-01-15T16:58:09.773 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8759d023:::obj-fill-466:head# 2021-01-15T16:58:09.773 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:88072111:::obj-fill-312:head# 2021-01-15T16:58:09.773 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:886edcef:::obj-fill-380:head# 2021-01-15T16:58:09.773 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:88ce753e:::obj-fill-43:head# 2021-01-15T16:58:09.774 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:890e1d83:::obj-fill-590:head# 2021-01-15T16:58:09.774 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:891566a3:::obj-fill-341:head# 2021-01-15T16:58:09.775 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:891ca235:::obj-fill-177:head# 2021-01-15T16:58:09.776 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:897fa9d0:::obj-fill-283:head# 2021-01-15T16:58:09.776 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:89cd304b:::obj-fill-592:head# 2021-01-15T16:58:09.777 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:89d04c4a:::obj-fill-388:head# 2021-01-15T16:58:09.777 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:89db45d9:::obj-fill-247:head# 2021-01-15T16:58:09.778 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8ad879c5:::obj-fill-492:head# 2021-01-15T16:58:09.778 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8af4b5de:::obj-fill-37:head# 2021-01-15T16:58:09.779 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8b132e86:::obj-fill-35:head# 2021-01-15T16:58:09.780 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8b80a60a:::obj-fill-100:head# 2021-01-15T16:58:09.780 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8bb64fa6:::obj-fill-599:head# 2021-01-15T16:58:09.781 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8bed03f5:::obj-fill-5:head# 2021-01-15T16:58:09.781 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8c60e135:::obj-fill-470:head# 2021-01-15T16:58:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8c774623:::obj-fill-38:head# 2021-01-15T16:58:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8cc528ae:::obj-fill-519:head# 2021-01-15T16:58:09.783 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8cf73826:::obj-fill-460:head# 2021-01-15T16:58:09.783 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8d3564b3:::obj-fill-452:head# 2021-01-15T16:58:09.820 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8dfe1ca2:::obj-fill-469:head# 2021-01-15T16:58:09.822 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8e527672:::obj-fill-246:head# 2021-01-15T16:58:09.823 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8e96103f:::obj-fill-444:head# 2021-01-15T16:58:09.823 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8ed6417f:::obj-fill-146:head# 2021-01-15T16:58:09.824 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8eda2ffc:::obj-fill-539:head# 2021-01-15T16:58:09.824 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8f08dd48:::obj-fill-26:head# 2021-01-15T16:58:09.825 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8f111b25:::obj-fill-18:head# 2021-01-15T16:58:09.825 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:8fde63f8:::obj-fill-338:head# 2021-01-15T16:58:09.826 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9030746d:::obj-fill-468:head# 2021-01-15T16:58:09.826 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9115d8a1:::obj-fill-260:head# 2021-01-15T16:58:09.827 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:91649909:::obj-fill-140:head# 2021-01-15T16:58:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:91c1081d:::obj-fill-404:head# 2021-01-15T16:58:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:91f42503:::obj-fill-216:head# 2021-01-15T16:58:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:927b34a0:::obj-fill-63:head# 2021-01-15T16:58:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:92a00a91:::obj-fill-467:head# 2021-01-15T16:58:09.830 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:930a7268:::obj-fill-450:head# 2021-01-15T16:58:09.831 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:93c3d0bf:::obj-fill-118:head# 2021-01-15T16:58:09.831 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:93ca7c9e:::obj-fill-132:head# 2021-01-15T16:58:09.832 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:942435d5:::obj-fill-378:head# 2021-01-15T16:58:09.833 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9445f8c5:::obj-fill-245:head# 2021-01-15T16:58:09.834 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9457c8d9:::obj-fill-535:head# 2021-01-15T16:58:09.835 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:950dc56e:::obj-fill-348:head# 2021-01-15T16:58:09.835 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9559ee1d:::obj-fill-545:head# 2021-01-15T16:58:09.836 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:957c5cc4:::obj-fill-212:head# 2021-01-15T16:58:09.836 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:95bdac02:::obj-fill-588:head# 2021-01-15T16:58:09.837 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:95f202af:::obj-fill-126:head# 2021-01-15T16:58:09.837 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:96789003:::obj-fill-83:head# 2021-01-15T16:58:09.838 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:96d494a8:::obj-fill-316:head# 2021-01-15T16:58:09.838 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:97adb85b:::obj-fill-233:head# 2021-01-15T16:58:09.840 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:980d998c:::obj-fill-197:head# 2021-01-15T16:58:09.840 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:98235c44:::obj-fill-396:head# 2021-01-15T16:58:09.841 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:987d3b87:::obj-fill-327:head# 2021-01-15T16:58:09.842 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:990a5e24:::obj-fill-515:head# 2021-01-15T16:58:09.842 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:995fec05:::obj-fill-191:head# 2021-01-15T16:58:09.843 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9a769b75:::obj-fill-587:head# 2021-01-15T16:58:09.843 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9b141990:::obj-fill-394:head# 2021-01-15T16:58:09.845 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9bb108fb:::obj-fill-579:head# 2021-01-15T16:58:09.845 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9ce19d92:::obj-fill-361:head# 2021-01-15T16:58:09.846 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9d8c3f95:::obj-fill-572:head# 2021-01-15T16:58:09.846 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9e283e00:::obj-fill-254:head# 2021-01-15T16:58:09.847 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9e3a1a61:::obj-fill-58:head# 2021-01-15T16:58:09.848 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9e8f6029:::obj-fill-41:head# 2021-01-15T16:58:09.848 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9ef117aa:::obj-fill-520:head# 2021-01-15T16:58:09.849 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9f3bdfc2:::obj-fill-184:head# 2021-01-15T16:58:09.849 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9fd89320:::obj-fill-364:head# 2021-01-15T16:58:09.850 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:9feb7062:::obj-fill-419:head# 2021-01-15T16:58:09.850 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a07504c4:::obj-fill-516:head# 2021-01-15T16:58:09.851 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a1897e34:::obj-fill-153:head# 2021-01-15T16:58:09.851 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a19a9d76:::obj-fill-30:head# 2021-01-15T16:58:09.852 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a1a49c98:::obj-fill-98:head# 2021-01-15T16:58:09.853 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a1ea911a:::obj-fill-36:head# 2021-01-15T16:58:09.853 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a24bad98:::obj-fill-335:head# 2021-01-15T16:58:09.853 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a263b16f:::obj-fill-326:head# 2021-01-15T16:58:09.854 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a27d661e:::obj-fill-152:head# 2021-01-15T16:58:09.854 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a282a129:::obj-fill-451:head# 2021-01-15T16:58:09.854 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a28f05ba:::obj-fill-241:head# 2021-01-15T16:58:09.856 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a2cdc5cc:::obj-fill-180:head# 2021-01-15T16:58:09.857 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a30777ea:::obj-fill-486:head# 2021-01-15T16:58:09.857 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a3a65661:::obj-fill-151:head# 2021-01-15T16:58:09.857 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a3cb2967:::obj-fill-195:head# 2021-01-15T16:58:09.858 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a3ffc098:::obj-fill-293:head# 2021-01-15T16:58:09.858 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a4a31ccc:::obj-fill-413:head# 2021-01-15T16:58:09.858 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a51c3677:::obj-fill-384:head# 2021-01-15T16:58:09.859 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a5e941a7:::obj-fill-141:head# 2021-01-15T16:58:09.860 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a6cdd8b2:::obj-fill-342:head# 2021-01-15T16:58:09.860 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a6e20b86:::obj-fill-593:head# 2021-01-15T16:58:09.861 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a7a9ccf9:::obj-fill-129:head# 2021-01-15T16:58:09.861 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a7b702d3:::obj-fill-49:head# 2021-01-15T16:58:09.862 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a8daf8bd:::obj-fill-391:head# 2021-01-15T16:58:09.862 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:a99c3cef:::obj-fill-171:head# 2021-01-15T16:58:09.863 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:aabb5718:::obj-fill-521:head# 2021-01-15T16:58:09.863 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ab0cb851:::obj-fill-183:head# 2021-01-15T16:58:09.864 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ab7a05e4:::obj-fill-203:head# 2021-01-15T16:58:09.865 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ab7dda83:::obj-fill-113:head# 2021-01-15T16:58:09.865 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:abb3502c:::obj-fill-461:head# 2021-01-15T16:58:09.866 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ac1c41be:::obj-fill-162:head# 2021-01-15T16:58:09.866 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ac2114ba:::obj-fill-239:head# 2021-01-15T16:58:09.867 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ad355fca:::obj-fill-271:head# 2021-01-15T16:58:09.867 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ad4634a0:::obj-fill-487:head# 2021-01-15T16:58:09.867 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:add2bb13:::obj-fill-31:head# 2021-01-15T16:58:09.868 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:aeb6258d:::obj-fill-124:head# 2021-01-15T16:58:09.868 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:af17f7d7:::obj-fill-464:head# 2021-01-15T16:58:09.869 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:af1e1cf0:::obj-fill-237:head# 2021-01-15T16:58:09.870 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:afc439ce:::obj-fill-248:head# 2021-01-15T16:58:09.870 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:afceab4d:::obj-fill-527:head# 2021-01-15T16:58:09.871 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b03af40c:::obj-fill-143:head# 2021-01-15T16:58:09.871 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b04cf0db:::obj-fill-507:head# 2021-01-15T16:58:09.872 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b07896f0:::obj-fill-269:head# 2021-01-15T16:58:09.872 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b0b954d5:::obj-fill-524:head# 2021-01-15T16:58:09.873 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b1cbfcb7:::obj-fill-134:head# 2021-01-15T16:58:09.873 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b2978fd3:::obj-fill-27:head# 2021-01-15T16:58:09.874 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b394798f:::obj-fill-156:head# 2021-01-15T16:58:09.874 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b3c6f36d:::obj-fill-401:head# 2021-01-15T16:58:09.875 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b3f8e5ac:::obj-fill-290:head# 2021-01-15T16:58:09.875 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b41ed5f8:::obj-fill-76:head# 2021-01-15T16:58:09.876 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b4878e76:::obj-fill-253:head# 2021-01-15T16:58:09.876 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b4e19741:::obj-fill-503:head# 2021-01-15T16:58:09.877 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b53a3690:::obj-fill-264:head# 2021-01-15T16:58:09.877 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b545ab4f:::obj-fill-24:head# 2021-01-15T16:58:09.878 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b55af5e6:::obj-fill-491:head# 2021-01-15T16:58:09.878 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b57edf82:::obj-fill-526:head# 2021-01-15T16:58:09.879 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b589d2be:::obj-fill-323:head# 2021-01-15T16:58:09.879 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b6207f32:::obj-fill-344:head# 2021-01-15T16:58:09.880 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b757ee2e:::obj-fill-433:head# 2021-01-15T16:58:09.880 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b75cc479:::obj-fill-34:head# 2021-01-15T16:58:09.881 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b7b03117:::obj-fill-42:head# 2021-01-15T16:58:09.881 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b7e6205d:::obj-fill-501:head# 2021-01-15T16:58:09.881 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b8d18b7a:::obj-fill-353:head# 2021-01-15T16:58:09.882 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b9189c29:::obj-fill-328:head# 2021-01-15T16:58:09.883 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b98e07aa:::obj-fill-54:head# 2021-01-15T16:58:09.883 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:b9cd1e9a:::obj-fill-371:head# 2021-01-15T16:58:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bc3322bb:::obj-fill-340:head# 2021-01-15T16:58:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bcf43161:::obj-fill-294:head# 2021-01-15T16:58:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bd07fc4f:::obj-fill-549:head# 2021-01-15T16:58:09.885 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bdf48e47:::obj-fill-430:head# 2021-01-15T16:58:09.885 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:be01c111:::obj-fill-1:head# 2021-01-15T16:58:09.886 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:be3a4bc7:::obj-fill-102:head# 2021-01-15T16:58:09.886 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:be8e1fcd:::obj-fill-321:head# 2021-01-15T16:58:09.886 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bf5fcbac:::obj-fill-287:head# 2021-01-15T16:58:09.887 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bf806d14:::obj-fill-115:head# 2021-01-15T16:58:09.887 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bfb28823:::obj-fill-72:head# 2021-01-15T16:58:09.888 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:bfef805e:::obj-fill-119:head# 2021-01-15T16:58:09.888 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c02f8e10:::obj-fill-497:head# 2021-01-15T16:58:09.889 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c050cd6e:::obj-fill-8:head# 2021-01-15T16:58:09.889 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c218e297:::obj-fill-173:head# 2021-01-15T16:58:09.890 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c2bc66fe:::obj-fill-252:head# 2021-01-15T16:58:09.891 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c3b55903:::obj-fill-428:head# 2021-01-15T16:58:09.891 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c51907b0:::obj-fill-457:head# 2021-01-15T16:58:09.892 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c561dc83:::obj-fill-75:head# 2021-01-15T16:58:09.892 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c5d1e1a9:::obj-fill-268:head# 2021-01-15T16:58:09.892 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c60ab0d0:::obj-fill-303:head# 2021-01-15T16:58:09.893 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c644ce52:::obj-fill-514:head# 2021-01-15T16:58:09.893 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c6ab3a11:::obj-fill-407:head# 2021-01-15T16:58:09.894 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c74377bf:::obj-fill-432:head# 2021-01-15T16:58:09.894 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c767fff6:::obj-fill-531:head# 2021-01-15T16:58:09.895 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c7a9adad:::obj-fill-475:head# 2021-01-15T16:58:09.895 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c7e3a9c2:::obj-fill-465:head# 2021-01-15T16:58:09.896 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c850aed7:::obj-fill-424:head# 2021-01-15T16:58:09.896 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c8a0088a:::obj-fill-329:head# 2021-01-15T16:58:09.897 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:c9d419f9:::obj-fill-176:head# 2021-01-15T16:58:09.897 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:caf25ec2:::obj-fill-274:head# 2021-01-15T16:58:09.898 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cb625830:::obj-fill-70:head# 2021-01-15T16:58:09.898 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cbb65125:::obj-fill-529:head# 2021-01-15T16:58:09.899 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cbe2544c:::obj-fill-3:head# 2021-01-15T16:58:09.899 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cbfce4ee:::obj-fill-455:head# 2021-01-15T16:58:09.900 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cc62419c:::obj-fill-498:head# 2021-01-15T16:58:09.900 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cce831fa:::obj-fill-160:head# 2021-01-15T16:58:09.901 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cda71af4:::obj-fill-122:head# 2021-01-15T16:58:09.901 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cea7a728:::obj-fill-265:head# 2021-01-15T16:58:09.902 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cee19f0a:::obj-fill-142:head# 2021-01-15T16:58:09.903 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cf22117e:::obj-fill-499:head# 2021-01-15T16:58:09.903 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cfbfa66d:::obj-fill-66:head# 2021-01-15T16:58:09.904 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cfc2552c:::obj-fill-211:head# 2021-01-15T16:58:09.904 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:cfc563e8:::obj-fill-392:head# 2021-01-15T16:58:09.904 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d02d8880:::obj-fill-362:head# 2021-01-15T16:58:09.905 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d054947e:::obj-fill-67:head# 2021-01-15T16:58:09.905 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d061c977:::obj-fill-164:head# 2021-01-15T16:58:09.906 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d06a5f57:::obj-fill-296:head# 2021-01-15T16:58:09.906 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d110062a:::obj-fill-343:head# 2021-01-15T16:58:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d1b8ad47:::obj-fill-383:head# 2021-01-15T16:58:09.907 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d1e80db6:::obj-fill-117:head# 2021-01-15T16:58:09.912 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d210027e:::obj-fill-231:head# 2021-01-15T16:58:09.913 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d24428ed:::obj-fill-199:head# 2021-01-15T16:58:09.914 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d27c3c4b:::obj-fill-154:head# 2021-01-15T16:58:09.914 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d28957e3:::obj-fill-263:head# 2021-01-15T16:58:09.915 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d2d44da7:::obj-fill-544:head# 2021-01-15T16:58:09.915 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d31004a4:::obj-fill-434:head# 2021-01-15T16:58:09.916 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d32487c3:::obj-fill-385:head# 2021-01-15T16:58:09.917 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d340891c:::obj-fill-12:head# 2021-01-15T16:58:09.917 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d4135728:::obj-fill-310:head# 2021-01-15T16:58:09.918 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d5a43972:::obj-fill-9:head# 2021-01-15T16:58:09.918 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d5efeb15:::obj-fill-347:head# 2021-01-15T16:58:09.919 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d6463f70:::obj-fill-536:head# 2021-01-15T16:58:09.919 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d666d9b7:::obj-fill-420:head# 2021-01-15T16:58:09.920 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d6e86688:::obj-fill-166:head# 2021-01-15T16:58:09.921 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d730370d:::obj-fill-206:head# 2021-01-15T16:58:09.921 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d783551c:::obj-fill-109:head# 2021-01-15T16:58:09.922 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d7f77e5f:::obj-fill-551:head# 2021-01-15T16:58:09.922 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d843056c:::obj-fill-291:head# 2021-01-15T16:58:09.923 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d84eb173:::obj-fill-221:head# 2021-01-15T16:58:09.924 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d8dd069c:::obj-fill-60:head# 2021-01-15T16:58:09.924 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d8f2f750:::obj-fill-562:head# 2021-01-15T16:58:09.924 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d8f80ac2:::obj-fill-179:head# 2021-01-15T16:58:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:d9f904cf:::obj-fill-308:head# 2021-01-15T16:58:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:db0a534b:::obj-fill-410:head# 2021-01-15T16:58:09.926 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dc4403f4:::obj-fill-416:head# 2021-01-15T16:58:09.927 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dc66c3ca:::obj-fill-220:head# 2021-01-15T16:58:09.927 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dc8198dc:::obj-fill-596:head# 2021-01-15T16:58:09.928 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dcbade2b:::obj-fill-10:head# 2021-01-15T16:58:09.928 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ddb787ea:::obj-fill-556:head# 2021-01-15T16:58:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:de58065b:::obj-fill-306:head# 2021-01-15T16:58:09.929 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:deaa6fab:::obj-fill-194:head# 2021-01-15T16:58:09.930 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dfee7714:::obj-fill-202:head# 2021-01-15T16:58:09.931 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:dff5df4a:::obj-fill-565:head# 2021-01-15T16:58:09.931 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e2053442:::obj-fill-107:head# 2021-01-15T16:58:09.932 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e32055b8:::obj-fill-57:head# 2021-01-15T16:58:09.932 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e37fac1e:::obj-fill-161:head# 2021-01-15T16:58:09.933 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e388f5aa:::obj-fill-273:head# 2021-01-15T16:58:09.934 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e38b1f4d:::obj-fill-393:head# 2021-01-15T16:58:09.934 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e4792474:::obj-fill-158:head# 2021-01-15T16:58:09.935 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e4899cb0:::obj-fill-270:head# 2021-01-15T16:58:09.936 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e62ae9f1:::obj-fill-288:head# 2021-01-15T16:58:09.937 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e64fd95a:::obj-fill-449:head# 2021-01-15T16:58:09.937 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e6682cfc:::obj-fill-373:head# 2021-01-15T16:58:09.938 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e716717b:::obj-fill-178:head# 2021-01-15T16:58:09.938 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e80226af:::obj-fill-354:head# 2021-01-15T16:58:09.939 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e83a5940:::obj-fill-564:head# 2021-01-15T16:58:09.939 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e854bfbe:::obj-fill-589:head# 2021-01-15T16:58:09.940 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e8b197bc:::obj-fill-89:head# 2021-01-15T16:58:09.940 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e92fca88:::obj-fill-397:head# 2021-01-15T16:58:09.941 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e94d8f7b:::obj-fill-261:head# 2021-01-15T16:58:09.942 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:e98cc0a4:::obj-fill-525:head# 2021-01-15T16:58:09.942 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ea5a9459:::obj-fill-431:head# 2021-01-15T16:58:09.943 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ebc54d47:::obj-fill-258:head# 2021-01-15T16:58:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ec0002dd:::obj-fill-205:head# 2021-01-15T16:58:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ec740b5c:::obj-fill-46:head# 2021-01-15T16:58:09.945 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ed35dda4:::obj-fill-170:head# 2021-01-15T16:58:09.945 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ed51b8f7:::obj-fill-91:head# 2021-01-15T16:58:09.946 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:edd96161:::obj-fill-48:head# 2021-01-15T16:58:09.946 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ee3f0d4b:::obj-fill-47:head# 2021-01-15T16:58:09.947 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ee4784dd:::obj-fill-480:head# 2021-01-15T16:58:09.947 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ee5a8271:::obj-fill-198:head# 2021-01-15T16:58:09.948 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ef745b66:::obj-fill-145:head# 2021-01-15T16:58:09.948 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:efa5309d:::obj-fill-215:head# 2021-01-15T16:58:09.949 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:efbdd3a1:::obj-fill-276:head# 2021-01-15T16:58:09.949 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f059d4c0:::obj-fill-482:head# 2021-01-15T16:58:09.950 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f0f9fb7a:::obj-fill-379:head# 2021-01-15T16:58:09.950 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f16a3656:::obj-fill-462:head# 2021-01-15T16:58:09.951 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f1b395ae:::obj-fill-25:head# 2021-01-15T16:58:09.951 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5876dad:::obj-fill-359:head# 2021-01-15T16:58:09.952 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5a283cd:::obj-fill-534:head# 2021-01-15T16:58:09.953 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5cd905a:::obj-fill-358:head# 2021-01-15T16:58:09.953 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5e48302:::obj-fill-406:head# 2021-01-15T16:58:09.954 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f5ea65fc:::obj-fill-52:head# 2021-01-15T16:58:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f642de6b:::obj-fill-204:head# 2021-01-15T16:58:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f716eafd:::obj-fill-502:head# 2021-01-15T16:58:09.956 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f752e9cb:::obj-fill-23:head# 2021-01-15T16:58:09.956 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f85fe42c:::obj-fill-346:head# 2021-01-15T16:58:09.957 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:f882fd3b:::obj-fill-355:head# 2021-01-15T16:58:09.957 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fa133332:::obj-fill-309:head# 2021-01-15T16:58:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fa875e52:::obj-fill-64:head# 2021-01-15T16:58:09.958 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fad91a22:::obj-fill-481:head# 2021-01-15T16:58:09.959 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fbbc3122:::obj-fill-111:head# 2021-01-15T16:58:09.959 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fc539ddb:::obj-fill-181:head# 2021-01-15T16:58:09.960 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fc54128d:::obj-fill-104:head# 2021-01-15T16:58:09.960 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fe00aded:::obj-fill-365:head# 2021-01-15T16:58:09.960 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fe1e9971:::obj-fill-189:head# 2021-01-15T16:58:09.961 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ff0e0563:::obj-fill-508:head# 2021-01-15T16:58:09.962 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ff4e156c:::obj-fill-441:head# 2021-01-15T16:58:09.962 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:ffa355f7:::obj-fill-500:head# 2021-01-15T16:58:09.963 INFO:tasks.workunit.client.0.smithi145.stderr:Read #1:fff6c93b:::obj-fill-277:head# 2021-01-15T16:58:09.963 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2021-01-15T16:58:09.964 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2021-01-15T16:58:09.964 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2021-01-15T16:58:09.964 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-01-15T16:58:11.142 INFO:tasks.workunit.client.0.smithi145.stderr:Remove successful 2021-01-15T16:58:11.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:477: TEST_backfill_multi_partial: activate_osd td/osd-backfill-space 1 2021-01-15T16:58:11.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T16:58:11.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:58:11.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T16:58:11.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:58:11.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T16:58:11.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:58:11.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:58:11.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:58:11.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:58:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T16:58:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T16:58:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:58:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:58:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:58:11.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:58:11.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:58:11.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:58:11.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:58:11.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:58:11.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:58:11.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:58:11.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:58:11.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:58:11.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:58:11.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:58:11.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:58:11.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:58:11.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:58:11.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:58:11.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:58:11.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T16:58:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T16:58:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:58:11.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2021-01-15T16:58:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T16:58:11.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:58:11.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:58:11.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:58:11.496 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T16:58:11.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:58:11.515+0000 7f9aba435080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:58:11.517 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:58:11.517+0000 7f9aba435080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:58:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:58:11.519+0000 7f9aba435080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:58:11.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T16:58:11.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:58:11.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T16:58:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:58:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:58:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:58:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:58:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:58:11.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:58:11.894 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T16:58:12.036 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:58:12.035+0000 7f9aba435080 -1 Falling back to public interface 2021-01-15T16:58:12.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:58:12.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:58:12.981+0000 7f9aba435080 -1 osd.1 35 log_to_monitors {default=true} 2021-01-15T16:58:13.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:58:13.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:58:13.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:58:13.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:58:13.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T16:58:13.288 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T16:58:13.684 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up out weight 0 up_from 40 up_thru 33 down_at 36 last_clean_interval [10,35) [v2:127.0.0.1:6810/224142,v1:127.0.0.1:6811/224142] [v2:127.0.0.1:6812/224142,v1:127.0.0.1:6813/224142] exists,up ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T16:58:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:58:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:58:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:58:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:479: TEST_backfill_multi_partial: ceph pg dump pgs 2021-01-15T16:58:14.087 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:58:14.088 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:58:09.112776+0000 0'0 39:22 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:07.000190+0000 0'0 2021-01-15T16:57:07.000190+0000 0 2021-01-15T16:58:14.088 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 554 0 0 0 0 1446494 0 0 54 54 stale+active+clean 2021-01-15T16:57:04.288669+0000 35'554 35:582 [1] 1 [1] 1 0'0 2021-01-15T16:57:01.996196+0000 0'0 2021-01-15T16:57:01.996196+0000 0 2021-01-15T16:58:14.088 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:58:09.112982+0000 0'0 39:42 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:04.485874+0000 0'0 2021-01-15T16:57:04.485874+0000 0 2021-01-15T16:58:14.089 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:58:14.089 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:58:14.090 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:58:14.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:481: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=1 2021-01-15T16:58:14.099 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T16:58:14.100 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T16:58:14.100 INFO:tasks.workunit.client.0.smithi145.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 7.4439e-05 s, 13.8 MB/s 2021-01-15T16:58:14.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: seq 1 600 2021-01-15T16:58:14.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-1 td/osd-backfill-space/datafile 2021-01-15T16:58:14.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-2 td/osd-backfill-space/datafile 2021-01-15T16:58:14.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-3 td/osd-backfill-space/datafile 2021-01-15T16:58:14.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-4 td/osd-backfill-space/datafile 2021-01-15T16:58:14.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-5 td/osd-backfill-space/datafile 2021-01-15T16:58:14.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-6 td/osd-backfill-space/datafile 2021-01-15T16:58:14.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:14.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-7 td/osd-backfill-space/datafile 2021-01-15T16:58:15.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-8 td/osd-backfill-space/datafile 2021-01-15T16:58:15.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-9 td/osd-backfill-space/datafile 2021-01-15T16:58:15.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-10 td/osd-backfill-space/datafile 2021-01-15T16:58:15.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-11 td/osd-backfill-space/datafile 2021-01-15T16:58:15.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-12 td/osd-backfill-space/datafile 2021-01-15T16:58:15.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-13 td/osd-backfill-space/datafile 2021-01-15T16:58:15.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:15.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-14 td/osd-backfill-space/datafile 2021-01-15T16:58:16.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-15 td/osd-backfill-space/datafile 2021-01-15T16:58:16.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-16 td/osd-backfill-space/datafile 2021-01-15T16:58:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-17 td/osd-backfill-space/datafile 2021-01-15T16:58:16.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-18 td/osd-backfill-space/datafile 2021-01-15T16:58:16.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-19 td/osd-backfill-space/datafile 2021-01-15T16:58:16.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-20 td/osd-backfill-space/datafile 2021-01-15T16:58:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:16.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-21 td/osd-backfill-space/datafile 2021-01-15T16:58:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-22 td/osd-backfill-space/datafile 2021-01-15T16:58:17.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-23 td/osd-backfill-space/datafile 2021-01-15T16:58:17.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-24 td/osd-backfill-space/datafile 2021-01-15T16:58:17.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-25 td/osd-backfill-space/datafile 2021-01-15T16:58:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-26 td/osd-backfill-space/datafile 2021-01-15T16:58:17.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-27 td/osd-backfill-space/datafile 2021-01-15T16:58:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-28 td/osd-backfill-space/datafile 2021-01-15T16:58:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:17.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-29 td/osd-backfill-space/datafile 2021-01-15T16:58:18.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-30 td/osd-backfill-space/datafile 2021-01-15T16:58:18.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-31 td/osd-backfill-space/datafile 2021-01-15T16:58:18.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-32 td/osd-backfill-space/datafile 2021-01-15T16:58:18.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-33 td/osd-backfill-space/datafile 2021-01-15T16:58:18.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-34 td/osd-backfill-space/datafile 2021-01-15T16:58:18.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-35 td/osd-backfill-space/datafile 2021-01-15T16:58:18.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:18.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-36 td/osd-backfill-space/datafile 2021-01-15T16:58:19.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-37 td/osd-backfill-space/datafile 2021-01-15T16:58:19.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-38 td/osd-backfill-space/datafile 2021-01-15T16:58:19.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-39 td/osd-backfill-space/datafile 2021-01-15T16:58:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-40 td/osd-backfill-space/datafile 2021-01-15T16:58:19.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-41 td/osd-backfill-space/datafile 2021-01-15T16:58:19.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-42 td/osd-backfill-space/datafile 2021-01-15T16:58:19.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:19.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-43 td/osd-backfill-space/datafile 2021-01-15T16:58:20.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-44 td/osd-backfill-space/datafile 2021-01-15T16:58:20.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-45 td/osd-backfill-space/datafile 2021-01-15T16:58:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-46 td/osd-backfill-space/datafile 2021-01-15T16:58:20.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-47 td/osd-backfill-space/datafile 2021-01-15T16:58:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-48 td/osd-backfill-space/datafile 2021-01-15T16:58:20.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-49 td/osd-backfill-space/datafile 2021-01-15T16:58:20.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:20.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-50 td/osd-backfill-space/datafile 2021-01-15T16:58:21.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-51 td/osd-backfill-space/datafile 2021-01-15T16:58:21.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-52 td/osd-backfill-space/datafile 2021-01-15T16:58:21.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-53 td/osd-backfill-space/datafile 2021-01-15T16:58:21.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-54 td/osd-backfill-space/datafile 2021-01-15T16:58:21.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-55 td/osd-backfill-space/datafile 2021-01-15T16:58:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-56 td/osd-backfill-space/datafile 2021-01-15T16:58:21.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-57 td/osd-backfill-space/datafile 2021-01-15T16:58:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-58 td/osd-backfill-space/datafile 2021-01-15T16:58:22.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-59 td/osd-backfill-space/datafile 2021-01-15T16:58:22.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-60 td/osd-backfill-space/datafile 2021-01-15T16:58:22.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-61 td/osd-backfill-space/datafile 2021-01-15T16:58:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-62 td/osd-backfill-space/datafile 2021-01-15T16:58:22.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-63 td/osd-backfill-space/datafile 2021-01-15T16:58:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-64 td/osd-backfill-space/datafile 2021-01-15T16:58:22.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:22.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-65 td/osd-backfill-space/datafile 2021-01-15T16:58:23.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-66 td/osd-backfill-space/datafile 2021-01-15T16:58:23.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-67 td/osd-backfill-space/datafile 2021-01-15T16:58:23.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-68 td/osd-backfill-space/datafile 2021-01-15T16:58:23.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-69 td/osd-backfill-space/datafile 2021-01-15T16:58:23.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-70 td/osd-backfill-space/datafile 2021-01-15T16:58:23.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-71 td/osd-backfill-space/datafile 2021-01-15T16:58:23.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-72 td/osd-backfill-space/datafile 2021-01-15T16:58:23.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:23.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-73 td/osd-backfill-space/datafile 2021-01-15T16:58:24.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-74 td/osd-backfill-space/datafile 2021-01-15T16:58:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-75 td/osd-backfill-space/datafile 2021-01-15T16:58:24.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-76 td/osd-backfill-space/datafile 2021-01-15T16:58:24.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-77 td/osd-backfill-space/datafile 2021-01-15T16:58:24.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-78 td/osd-backfill-space/datafile 2021-01-15T16:58:24.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-79 td/osd-backfill-space/datafile 2021-01-15T16:58:24.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:24.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-80 td/osd-backfill-space/datafile 2021-01-15T16:58:25.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-81 td/osd-backfill-space/datafile 2021-01-15T16:58:25.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-82 td/osd-backfill-space/datafile 2021-01-15T16:58:25.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-83 td/osd-backfill-space/datafile 2021-01-15T16:58:25.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-84 td/osd-backfill-space/datafile 2021-01-15T16:58:25.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-85 td/osd-backfill-space/datafile 2021-01-15T16:58:25.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-86 td/osd-backfill-space/datafile 2021-01-15T16:58:25.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:25.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-87 td/osd-backfill-space/datafile 2021-01-15T16:58:26.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-88 td/osd-backfill-space/datafile 2021-01-15T16:58:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-89 td/osd-backfill-space/datafile 2021-01-15T16:58:26.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-90 td/osd-backfill-space/datafile 2021-01-15T16:58:26.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-91 td/osd-backfill-space/datafile 2021-01-15T16:58:26.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-92 td/osd-backfill-space/datafile 2021-01-15T16:58:26.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-93 td/osd-backfill-space/datafile 2021-01-15T16:58:26.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-94 td/osd-backfill-space/datafile 2021-01-15T16:58:26.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:26.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-95 td/osd-backfill-space/datafile 2021-01-15T16:58:27.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-96 td/osd-backfill-space/datafile 2021-01-15T16:58:27.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-97 td/osd-backfill-space/datafile 2021-01-15T16:58:27.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-98 td/osd-backfill-space/datafile 2021-01-15T16:58:27.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-99 td/osd-backfill-space/datafile 2021-01-15T16:58:27.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-100 td/osd-backfill-space/datafile 2021-01-15T16:58:27.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-101 td/osd-backfill-space/datafile 2021-01-15T16:58:27.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:27.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-102 td/osd-backfill-space/datafile 2021-01-15T16:58:28.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-103 td/osd-backfill-space/datafile 2021-01-15T16:58:28.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-104 td/osd-backfill-space/datafile 2021-01-15T16:58:28.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-105 td/osd-backfill-space/datafile 2021-01-15T16:58:28.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-106 td/osd-backfill-space/datafile 2021-01-15T16:58:28.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-107 td/osd-backfill-space/datafile 2021-01-15T16:58:28.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-108 td/osd-backfill-space/datafile 2021-01-15T16:58:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:28.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-109 td/osd-backfill-space/datafile 2021-01-15T16:58:29.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-110 td/osd-backfill-space/datafile 2021-01-15T16:58:29.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-111 td/osd-backfill-space/datafile 2021-01-15T16:58:29.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-112 td/osd-backfill-space/datafile 2021-01-15T16:58:29.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-113 td/osd-backfill-space/datafile 2021-01-15T16:58:29.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-114 td/osd-backfill-space/datafile 2021-01-15T16:58:29.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-115 td/osd-backfill-space/datafile 2021-01-15T16:58:29.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:29.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-116 td/osd-backfill-space/datafile 2021-01-15T16:58:30.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-117 td/osd-backfill-space/datafile 2021-01-15T16:58:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-118 td/osd-backfill-space/datafile 2021-01-15T16:58:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-119 td/osd-backfill-space/datafile 2021-01-15T16:58:30.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-120 td/osd-backfill-space/datafile 2021-01-15T16:58:30.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-121 td/osd-backfill-space/datafile 2021-01-15T16:58:30.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-122 td/osd-backfill-space/datafile 2021-01-15T16:58:30.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:30.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-123 td/osd-backfill-space/datafile 2021-01-15T16:58:31.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-124 td/osd-backfill-space/datafile 2021-01-15T16:58:31.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-125 td/osd-backfill-space/datafile 2021-01-15T16:58:31.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-126 td/osd-backfill-space/datafile 2021-01-15T16:58:31.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-127 td/osd-backfill-space/datafile 2021-01-15T16:58:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-128 td/osd-backfill-space/datafile 2021-01-15T16:58:31.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-129 td/osd-backfill-space/datafile 2021-01-15T16:58:31.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:31.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-130 td/osd-backfill-space/datafile 2021-01-15T16:58:32.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-131 td/osd-backfill-space/datafile 2021-01-15T16:58:32.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-132 td/osd-backfill-space/datafile 2021-01-15T16:58:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-133 td/osd-backfill-space/datafile 2021-01-15T16:58:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-134 td/osd-backfill-space/datafile 2021-01-15T16:58:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-135 td/osd-backfill-space/datafile 2021-01-15T16:58:32.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-136 td/osd-backfill-space/datafile 2021-01-15T16:58:32.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-137 td/osd-backfill-space/datafile 2021-01-15T16:58:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-138 td/osd-backfill-space/datafile 2021-01-15T16:58:33.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-139 td/osd-backfill-space/datafile 2021-01-15T16:58:33.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-140 td/osd-backfill-space/datafile 2021-01-15T16:58:33.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-141 td/osd-backfill-space/datafile 2021-01-15T16:58:33.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-142 td/osd-backfill-space/datafile 2021-01-15T16:58:33.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-143 td/osd-backfill-space/datafile 2021-01-15T16:58:33.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-144 td/osd-backfill-space/datafile 2021-01-15T16:58:33.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-145 td/osd-backfill-space/datafile 2021-01-15T16:58:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-146 td/osd-backfill-space/datafile 2021-01-15T16:58:34.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-147 td/osd-backfill-space/datafile 2021-01-15T16:58:34.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-148 td/osd-backfill-space/datafile 2021-01-15T16:58:34.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-149 td/osd-backfill-space/datafile 2021-01-15T16:58:34.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-150 td/osd-backfill-space/datafile 2021-01-15T16:58:34.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-151 td/osd-backfill-space/datafile 2021-01-15T16:58:34.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:34.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-152 td/osd-backfill-space/datafile 2021-01-15T16:58:35.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-153 td/osd-backfill-space/datafile 2021-01-15T16:58:35.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-154 td/osd-backfill-space/datafile 2021-01-15T16:58:35.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-155 td/osd-backfill-space/datafile 2021-01-15T16:58:35.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-156 td/osd-backfill-space/datafile 2021-01-15T16:58:35.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-157 td/osd-backfill-space/datafile 2021-01-15T16:58:35.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-158 td/osd-backfill-space/datafile 2021-01-15T16:58:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-159 td/osd-backfill-space/datafile 2021-01-15T16:58:35.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:35.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-160 td/osd-backfill-space/datafile 2021-01-15T16:58:36.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-161 td/osd-backfill-space/datafile 2021-01-15T16:58:36.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-162 td/osd-backfill-space/datafile 2021-01-15T16:58:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-163 td/osd-backfill-space/datafile 2021-01-15T16:58:36.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-164 td/osd-backfill-space/datafile 2021-01-15T16:58:36.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-165 td/osd-backfill-space/datafile 2021-01-15T16:58:36.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-166 td/osd-backfill-space/datafile 2021-01-15T16:58:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:36.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-167 td/osd-backfill-space/datafile 2021-01-15T16:58:37.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-168 td/osd-backfill-space/datafile 2021-01-15T16:58:37.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-169 td/osd-backfill-space/datafile 2021-01-15T16:58:37.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-170 td/osd-backfill-space/datafile 2021-01-15T16:58:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-171 td/osd-backfill-space/datafile 2021-01-15T16:58:37.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-172 td/osd-backfill-space/datafile 2021-01-15T16:58:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-173 td/osd-backfill-space/datafile 2021-01-15T16:58:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-174 td/osd-backfill-space/datafile 2021-01-15T16:58:37.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:37.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-175 td/osd-backfill-space/datafile 2021-01-15T16:58:38.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-176 td/osd-backfill-space/datafile 2021-01-15T16:58:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-177 td/osd-backfill-space/datafile 2021-01-15T16:58:38.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-178 td/osd-backfill-space/datafile 2021-01-15T16:58:38.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-179 td/osd-backfill-space/datafile 2021-01-15T16:58:38.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-180 td/osd-backfill-space/datafile 2021-01-15T16:58:38.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-181 td/osd-backfill-space/datafile 2021-01-15T16:58:38.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:38.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-182 td/osd-backfill-space/datafile 2021-01-15T16:58:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-183 td/osd-backfill-space/datafile 2021-01-15T16:58:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-184 td/osd-backfill-space/datafile 2021-01-15T16:58:39.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-185 td/osd-backfill-space/datafile 2021-01-15T16:58:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-186 td/osd-backfill-space/datafile 2021-01-15T16:58:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-187 td/osd-backfill-space/datafile 2021-01-15T16:58:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-188 td/osd-backfill-space/datafile 2021-01-15T16:58:39.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-189 td/osd-backfill-space/datafile 2021-01-15T16:58:39.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:39.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-190 td/osd-backfill-space/datafile 2021-01-15T16:58:40.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:40.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-191 td/osd-backfill-space/datafile 2021-01-15T16:58:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-192 td/osd-backfill-space/datafile 2021-01-15T16:58:40.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:40.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-193 td/osd-backfill-space/datafile 2021-01-15T16:58:40.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:40.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-194 td/osd-backfill-space/datafile 2021-01-15T16:58:40.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:40.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-195 td/osd-backfill-space/datafile 2021-01-15T16:58:41.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-196 td/osd-backfill-space/datafile 2021-01-15T16:58:41.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-197 td/osd-backfill-space/datafile 2021-01-15T16:58:41.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-198 td/osd-backfill-space/datafile 2021-01-15T16:58:41.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-199 td/osd-backfill-space/datafile 2021-01-15T16:58:41.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-200 td/osd-backfill-space/datafile 2021-01-15T16:58:41.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-201 td/osd-backfill-space/datafile 2021-01-15T16:58:41.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:41.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-202 td/osd-backfill-space/datafile 2021-01-15T16:58:42.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-203 td/osd-backfill-space/datafile 2021-01-15T16:58:42.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-204 td/osd-backfill-space/datafile 2021-01-15T16:58:42.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-205 td/osd-backfill-space/datafile 2021-01-15T16:58:42.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-206 td/osd-backfill-space/datafile 2021-01-15T16:58:42.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-207 td/osd-backfill-space/datafile 2021-01-15T16:58:42.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-208 td/osd-backfill-space/datafile 2021-01-15T16:58:42.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:42.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-209 td/osd-backfill-space/datafile 2021-01-15T16:58:43.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-210 td/osd-backfill-space/datafile 2021-01-15T16:58:43.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-211 td/osd-backfill-space/datafile 2021-01-15T16:58:43.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-212 td/osd-backfill-space/datafile 2021-01-15T16:58:43.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-213 td/osd-backfill-space/datafile 2021-01-15T16:58:43.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-214 td/osd-backfill-space/datafile 2021-01-15T16:58:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-215 td/osd-backfill-space/datafile 2021-01-15T16:58:43.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-216 td/osd-backfill-space/datafile 2021-01-15T16:58:43.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:43.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-217 td/osd-backfill-space/datafile 2021-01-15T16:58:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-218 td/osd-backfill-space/datafile 2021-01-15T16:58:44.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-219 td/osd-backfill-space/datafile 2021-01-15T16:58:44.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-220 td/osd-backfill-space/datafile 2021-01-15T16:58:44.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-221 td/osd-backfill-space/datafile 2021-01-15T16:58:44.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-222 td/osd-backfill-space/datafile 2021-01-15T16:58:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-223 td/osd-backfill-space/datafile 2021-01-15T16:58:44.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:44.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-224 td/osd-backfill-space/datafile 2021-01-15T16:58:45.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-225 td/osd-backfill-space/datafile 2021-01-15T16:58:45.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-226 td/osd-backfill-space/datafile 2021-01-15T16:58:45.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-227 td/osd-backfill-space/datafile 2021-01-15T16:58:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-228 td/osd-backfill-space/datafile 2021-01-15T16:58:45.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-229 td/osd-backfill-space/datafile 2021-01-15T16:58:45.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-230 td/osd-backfill-space/datafile 2021-01-15T16:58:45.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:45.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-231 td/osd-backfill-space/datafile 2021-01-15T16:58:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-232 td/osd-backfill-space/datafile 2021-01-15T16:58:46.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-233 td/osd-backfill-space/datafile 2021-01-15T16:58:46.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-234 td/osd-backfill-space/datafile 2021-01-15T16:58:46.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-235 td/osd-backfill-space/datafile 2021-01-15T16:58:46.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-236 td/osd-backfill-space/datafile 2021-01-15T16:58:46.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-237 td/osd-backfill-space/datafile 2021-01-15T16:58:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-238 td/osd-backfill-space/datafile 2021-01-15T16:58:46.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:46.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-239 td/osd-backfill-space/datafile 2021-01-15T16:58:47.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-240 td/osd-backfill-space/datafile 2021-01-15T16:58:47.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-241 td/osd-backfill-space/datafile 2021-01-15T16:58:47.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-242 td/osd-backfill-space/datafile 2021-01-15T16:58:47.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-243 td/osd-backfill-space/datafile 2021-01-15T16:58:47.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-244 td/osd-backfill-space/datafile 2021-01-15T16:58:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:47.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-245 td/osd-backfill-space/datafile 2021-01-15T16:58:48.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-246 td/osd-backfill-space/datafile 2021-01-15T16:58:48.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-247 td/osd-backfill-space/datafile 2021-01-15T16:58:48.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-248 td/osd-backfill-space/datafile 2021-01-15T16:58:48.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-249 td/osd-backfill-space/datafile 2021-01-15T16:58:48.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-250 td/osd-backfill-space/datafile 2021-01-15T16:58:48.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-251 td/osd-backfill-space/datafile 2021-01-15T16:58:48.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-252 td/osd-backfill-space/datafile 2021-01-15T16:58:48.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:48.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-253 td/osd-backfill-space/datafile 2021-01-15T16:58:49.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-254 td/osd-backfill-space/datafile 2021-01-15T16:58:49.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-255 td/osd-backfill-space/datafile 2021-01-15T16:58:49.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-256 td/osd-backfill-space/datafile 2021-01-15T16:58:49.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-257 td/osd-backfill-space/datafile 2021-01-15T16:58:49.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-258 td/osd-backfill-space/datafile 2021-01-15T16:58:49.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-259 td/osd-backfill-space/datafile 2021-01-15T16:58:49.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-260 td/osd-backfill-space/datafile 2021-01-15T16:58:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-261 td/osd-backfill-space/datafile 2021-01-15T16:58:50.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-262 td/osd-backfill-space/datafile 2021-01-15T16:58:50.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-263 td/osd-backfill-space/datafile 2021-01-15T16:58:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-264 td/osd-backfill-space/datafile 2021-01-15T16:58:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-265 td/osd-backfill-space/datafile 2021-01-15T16:58:50.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-266 td/osd-backfill-space/datafile 2021-01-15T16:58:50.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-267 td/osd-backfill-space/datafile 2021-01-15T16:58:50.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:50.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-268 td/osd-backfill-space/datafile 2021-01-15T16:58:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-269 td/osd-backfill-space/datafile 2021-01-15T16:58:51.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-270 td/osd-backfill-space/datafile 2021-01-15T16:58:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-271 td/osd-backfill-space/datafile 2021-01-15T16:58:51.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-272 td/osd-backfill-space/datafile 2021-01-15T16:58:51.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-273 td/osd-backfill-space/datafile 2021-01-15T16:58:51.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-274 td/osd-backfill-space/datafile 2021-01-15T16:58:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-275 td/osd-backfill-space/datafile 2021-01-15T16:58:52.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-276 td/osd-backfill-space/datafile 2021-01-15T16:58:52.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-277 td/osd-backfill-space/datafile 2021-01-15T16:58:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-278 td/osd-backfill-space/datafile 2021-01-15T16:58:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-279 td/osd-backfill-space/datafile 2021-01-15T16:58:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-280 td/osd-backfill-space/datafile 2021-01-15T16:58:52.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-281 td/osd-backfill-space/datafile 2021-01-15T16:58:52.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:52.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-282 td/osd-backfill-space/datafile 2021-01-15T16:58:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-283 td/osd-backfill-space/datafile 2021-01-15T16:58:53.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-284 td/osd-backfill-space/datafile 2021-01-15T16:58:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-285 td/osd-backfill-space/datafile 2021-01-15T16:58:53.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-286 td/osd-backfill-space/datafile 2021-01-15T16:58:53.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-287 td/osd-backfill-space/datafile 2021-01-15T16:58:53.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-288 td/osd-backfill-space/datafile 2021-01-15T16:58:53.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-289 td/osd-backfill-space/datafile 2021-01-15T16:58:53.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:53.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-290 td/osd-backfill-space/datafile 2021-01-15T16:58:54.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-291 td/osd-backfill-space/datafile 2021-01-15T16:58:54.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-292 td/osd-backfill-space/datafile 2021-01-15T16:58:54.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-293 td/osd-backfill-space/datafile 2021-01-15T16:58:54.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-294 td/osd-backfill-space/datafile 2021-01-15T16:58:54.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-295 td/osd-backfill-space/datafile 2021-01-15T16:58:54.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-296 td/osd-backfill-space/datafile 2021-01-15T16:58:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-297 td/osd-backfill-space/datafile 2021-01-15T16:58:55.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-298 td/osd-backfill-space/datafile 2021-01-15T16:58:55.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-299 td/osd-backfill-space/datafile 2021-01-15T16:58:55.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-300 td/osd-backfill-space/datafile 2021-01-15T16:58:55.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-301 td/osd-backfill-space/datafile 2021-01-15T16:58:55.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-302 td/osd-backfill-space/datafile 2021-01-15T16:58:55.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-303 td/osd-backfill-space/datafile 2021-01-15T16:58:55.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-304 td/osd-backfill-space/datafile 2021-01-15T16:58:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:55.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-305 td/osd-backfill-space/datafile 2021-01-15T16:58:56.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-306 td/osd-backfill-space/datafile 2021-01-15T16:58:56.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-307 td/osd-backfill-space/datafile 2021-01-15T16:58:56.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-308 td/osd-backfill-space/datafile 2021-01-15T16:58:56.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-309 td/osd-backfill-space/datafile 2021-01-15T16:58:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-310 td/osd-backfill-space/datafile 2021-01-15T16:58:56.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-311 td/osd-backfill-space/datafile 2021-01-15T16:58:56.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:56.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-312 td/osd-backfill-space/datafile 2021-01-15T16:58:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-313 td/osd-backfill-space/datafile 2021-01-15T16:58:57.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-314 td/osd-backfill-space/datafile 2021-01-15T16:58:57.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-315 td/osd-backfill-space/datafile 2021-01-15T16:58:57.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-316 td/osd-backfill-space/datafile 2021-01-15T16:58:57.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-317 td/osd-backfill-space/datafile 2021-01-15T16:58:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-318 td/osd-backfill-space/datafile 2021-01-15T16:58:57.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-319 td/osd-backfill-space/datafile 2021-01-15T16:58:57.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:57.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-320 td/osd-backfill-space/datafile 2021-01-15T16:58:58.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-321 td/osd-backfill-space/datafile 2021-01-15T16:58:58.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-322 td/osd-backfill-space/datafile 2021-01-15T16:58:58.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-323 td/osd-backfill-space/datafile 2021-01-15T16:58:58.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-324 td/osd-backfill-space/datafile 2021-01-15T16:58:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-325 td/osd-backfill-space/datafile 2021-01-15T16:58:58.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-326 td/osd-backfill-space/datafile 2021-01-15T16:58:58.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-327 td/osd-backfill-space/datafile 2021-01-15T16:58:58.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-328 td/osd-backfill-space/datafile 2021-01-15T16:58:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-329 td/osd-backfill-space/datafile 2021-01-15T16:58:59.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-330 td/osd-backfill-space/datafile 2021-01-15T16:58:59.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-331 td/osd-backfill-space/datafile 2021-01-15T16:58:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-332 td/osd-backfill-space/datafile 2021-01-15T16:58:59.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-333 td/osd-backfill-space/datafile 2021-01-15T16:58:59.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-334 td/osd-backfill-space/datafile 2021-01-15T16:58:59.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:58:59.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-335 td/osd-backfill-space/datafile 2021-01-15T16:59:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-336 td/osd-backfill-space/datafile 2021-01-15T16:59:00.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-337 td/osd-backfill-space/datafile 2021-01-15T16:59:00.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-338 td/osd-backfill-space/datafile 2021-01-15T16:59:00.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-339 td/osd-backfill-space/datafile 2021-01-15T16:59:00.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-340 td/osd-backfill-space/datafile 2021-01-15T16:59:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-341 td/osd-backfill-space/datafile 2021-01-15T16:59:00.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-342 td/osd-backfill-space/datafile 2021-01-15T16:59:00.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:00.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-343 td/osd-backfill-space/datafile 2021-01-15T16:59:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-344 td/osd-backfill-space/datafile 2021-01-15T16:59:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-345 td/osd-backfill-space/datafile 2021-01-15T16:59:01.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-346 td/osd-backfill-space/datafile 2021-01-15T16:59:01.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-347 td/osd-backfill-space/datafile 2021-01-15T16:59:01.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-348 td/osd-backfill-space/datafile 2021-01-15T16:59:01.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-349 td/osd-backfill-space/datafile 2021-01-15T16:59:01.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:01.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-350 td/osd-backfill-space/datafile 2021-01-15T16:59:02.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-351 td/osd-backfill-space/datafile 2021-01-15T16:59:02.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-352 td/osd-backfill-space/datafile 2021-01-15T16:59:02.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-353 td/osd-backfill-space/datafile 2021-01-15T16:59:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-354 td/osd-backfill-space/datafile 2021-01-15T16:59:02.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-355 td/osd-backfill-space/datafile 2021-01-15T16:59:02.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-356 td/osd-backfill-space/datafile 2021-01-15T16:59:02.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-357 td/osd-backfill-space/datafile 2021-01-15T16:59:02.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:02.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-358 td/osd-backfill-space/datafile 2021-01-15T16:59:03.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-359 td/osd-backfill-space/datafile 2021-01-15T16:59:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-360 td/osd-backfill-space/datafile 2021-01-15T16:59:03.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-361 td/osd-backfill-space/datafile 2021-01-15T16:59:03.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-362 td/osd-backfill-space/datafile 2021-01-15T16:59:03.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-363 td/osd-backfill-space/datafile 2021-01-15T16:59:03.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-364 td/osd-backfill-space/datafile 2021-01-15T16:59:03.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:03.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-365 td/osd-backfill-space/datafile 2021-01-15T16:59:04.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-366 td/osd-backfill-space/datafile 2021-01-15T16:59:04.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-367 td/osd-backfill-space/datafile 2021-01-15T16:59:04.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-368 td/osd-backfill-space/datafile 2021-01-15T16:59:04.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-369 td/osd-backfill-space/datafile 2021-01-15T16:59:04.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-370 td/osd-backfill-space/datafile 2021-01-15T16:59:04.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-371 td/osd-backfill-space/datafile 2021-01-15T16:59:04.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-372 td/osd-backfill-space/datafile 2021-01-15T16:59:04.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:04.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-373 td/osd-backfill-space/datafile 2021-01-15T16:59:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-374 td/osd-backfill-space/datafile 2021-01-15T16:59:05.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-375 td/osd-backfill-space/datafile 2021-01-15T16:59:05.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-376 td/osd-backfill-space/datafile 2021-01-15T16:59:05.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-377 td/osd-backfill-space/datafile 2021-01-15T16:59:05.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-378 td/osd-backfill-space/datafile 2021-01-15T16:59:05.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-379 td/osd-backfill-space/datafile 2021-01-15T16:59:05.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-380 td/osd-backfill-space/datafile 2021-01-15T16:59:05.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:05.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-381 td/osd-backfill-space/datafile 2021-01-15T16:59:06.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-382 td/osd-backfill-space/datafile 2021-01-15T16:59:06.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-383 td/osd-backfill-space/datafile 2021-01-15T16:59:06.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-384 td/osd-backfill-space/datafile 2021-01-15T16:59:06.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-385 td/osd-backfill-space/datafile 2021-01-15T16:59:06.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-386 td/osd-backfill-space/datafile 2021-01-15T16:59:06.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-387 td/osd-backfill-space/datafile 2021-01-15T16:59:06.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:06.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-388 td/osd-backfill-space/datafile 2021-01-15T16:59:07.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-389 td/osd-backfill-space/datafile 2021-01-15T16:59:07.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-390 td/osd-backfill-space/datafile 2021-01-15T16:59:07.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-391 td/osd-backfill-space/datafile 2021-01-15T16:59:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-392 td/osd-backfill-space/datafile 2021-01-15T16:59:07.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-393 td/osd-backfill-space/datafile 2021-01-15T16:59:07.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-394 td/osd-backfill-space/datafile 2021-01-15T16:59:07.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-395 td/osd-backfill-space/datafile 2021-01-15T16:59:07.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-396 td/osd-backfill-space/datafile 2021-01-15T16:59:07.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:07.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-397 td/osd-backfill-space/datafile 2021-01-15T16:59:08.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-398 td/osd-backfill-space/datafile 2021-01-15T16:59:08.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-399 td/osd-backfill-space/datafile 2021-01-15T16:59:08.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-400 td/osd-backfill-space/datafile 2021-01-15T16:59:08.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-401 td/osd-backfill-space/datafile 2021-01-15T16:59:08.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-402 td/osd-backfill-space/datafile 2021-01-15T16:59:08.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-403 td/osd-backfill-space/datafile 2021-01-15T16:59:08.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:08.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-404 td/osd-backfill-space/datafile 2021-01-15T16:59:09.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-405 td/osd-backfill-space/datafile 2021-01-15T16:59:09.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-406 td/osd-backfill-space/datafile 2021-01-15T16:59:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-407 td/osd-backfill-space/datafile 2021-01-15T16:59:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-408 td/osd-backfill-space/datafile 2021-01-15T16:59:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-409 td/osd-backfill-space/datafile 2021-01-15T16:59:09.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-410 td/osd-backfill-space/datafile 2021-01-15T16:59:09.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-411 td/osd-backfill-space/datafile 2021-01-15T16:59:09.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:09.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-412 td/osd-backfill-space/datafile 2021-01-15T16:59:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-413 td/osd-backfill-space/datafile 2021-01-15T16:59:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-414 td/osd-backfill-space/datafile 2021-01-15T16:59:10.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-415 td/osd-backfill-space/datafile 2021-01-15T16:59:10.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-416 td/osd-backfill-space/datafile 2021-01-15T16:59:10.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-417 td/osd-backfill-space/datafile 2021-01-15T16:59:10.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-418 td/osd-backfill-space/datafile 2021-01-15T16:59:10.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-419 td/osd-backfill-space/datafile 2021-01-15T16:59:10.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:10.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-420 td/osd-backfill-space/datafile 2021-01-15T16:59:11.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-421 td/osd-backfill-space/datafile 2021-01-15T16:59:11.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-422 td/osd-backfill-space/datafile 2021-01-15T16:59:11.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-423 td/osd-backfill-space/datafile 2021-01-15T16:59:11.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-424 td/osd-backfill-space/datafile 2021-01-15T16:59:11.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-425 td/osd-backfill-space/datafile 2021-01-15T16:59:11.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-426 td/osd-backfill-space/datafile 2021-01-15T16:59:11.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-427 td/osd-backfill-space/datafile 2021-01-15T16:59:11.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:11.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-428 td/osd-backfill-space/datafile 2021-01-15T16:59:12.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:12.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-429 td/osd-backfill-space/datafile 2021-01-15T16:59:12.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:12.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-430 td/osd-backfill-space/datafile 2021-01-15T16:59:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-431 td/osd-backfill-space/datafile 2021-01-15T16:59:12.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:12.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-432 td/osd-backfill-space/datafile 2021-01-15T16:59:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-433 td/osd-backfill-space/datafile 2021-01-15T16:59:13.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-434 td/osd-backfill-space/datafile 2021-01-15T16:59:13.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-435 td/osd-backfill-space/datafile 2021-01-15T16:59:13.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-436 td/osd-backfill-space/datafile 2021-01-15T16:59:13.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-437 td/osd-backfill-space/datafile 2021-01-15T16:59:13.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-438 td/osd-backfill-space/datafile 2021-01-15T16:59:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-439 td/osd-backfill-space/datafile 2021-01-15T16:59:13.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:13.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-440 td/osd-backfill-space/datafile 2021-01-15T16:59:14.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-441 td/osd-backfill-space/datafile 2021-01-15T16:59:14.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-442 td/osd-backfill-space/datafile 2021-01-15T16:59:14.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-443 td/osd-backfill-space/datafile 2021-01-15T16:59:14.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-444 td/osd-backfill-space/datafile 2021-01-15T16:59:14.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-445 td/osd-backfill-space/datafile 2021-01-15T16:59:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-446 td/osd-backfill-space/datafile 2021-01-15T16:59:14.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-447 td/osd-backfill-space/datafile 2021-01-15T16:59:14.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:14.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-448 td/osd-backfill-space/datafile 2021-01-15T16:59:15.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-449 td/osd-backfill-space/datafile 2021-01-15T16:59:15.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-450 td/osd-backfill-space/datafile 2021-01-15T16:59:15.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-451 td/osd-backfill-space/datafile 2021-01-15T16:59:15.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-452 td/osd-backfill-space/datafile 2021-01-15T16:59:15.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-453 td/osd-backfill-space/datafile 2021-01-15T16:59:15.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-454 td/osd-backfill-space/datafile 2021-01-15T16:59:15.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-455 td/osd-backfill-space/datafile 2021-01-15T16:59:15.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:15.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-456 td/osd-backfill-space/datafile 2021-01-15T16:59:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-457 td/osd-backfill-space/datafile 2021-01-15T16:59:16.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-458 td/osd-backfill-space/datafile 2021-01-15T16:59:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-459 td/osd-backfill-space/datafile 2021-01-15T16:59:16.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-460 td/osd-backfill-space/datafile 2021-01-15T16:59:16.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-461 td/osd-backfill-space/datafile 2021-01-15T16:59:16.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-462 td/osd-backfill-space/datafile 2021-01-15T16:59:16.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:16.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-463 td/osd-backfill-space/datafile 2021-01-15T16:59:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-464 td/osd-backfill-space/datafile 2021-01-15T16:59:17.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-465 td/osd-backfill-space/datafile 2021-01-15T16:59:17.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-466 td/osd-backfill-space/datafile 2021-01-15T16:59:17.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-467 td/osd-backfill-space/datafile 2021-01-15T16:59:17.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-468 td/osd-backfill-space/datafile 2021-01-15T16:59:17.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-469 td/osd-backfill-space/datafile 2021-01-15T16:59:17.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-470 td/osd-backfill-space/datafile 2021-01-15T16:59:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-471 td/osd-backfill-space/datafile 2021-01-15T16:59:18.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-472 td/osd-backfill-space/datafile 2021-01-15T16:59:18.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-473 td/osd-backfill-space/datafile 2021-01-15T16:59:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-474 td/osd-backfill-space/datafile 2021-01-15T16:59:18.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-475 td/osd-backfill-space/datafile 2021-01-15T16:59:18.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-476 td/osd-backfill-space/datafile 2021-01-15T16:59:18.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-477 td/osd-backfill-space/datafile 2021-01-15T16:59:18.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:18.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-478 td/osd-backfill-space/datafile 2021-01-15T16:59:19.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-479 td/osd-backfill-space/datafile 2021-01-15T16:59:19.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-480 td/osd-backfill-space/datafile 2021-01-15T16:59:19.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-481 td/osd-backfill-space/datafile 2021-01-15T16:59:19.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-482 td/osd-backfill-space/datafile 2021-01-15T16:59:19.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-483 td/osd-backfill-space/datafile 2021-01-15T16:59:19.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-484 td/osd-backfill-space/datafile 2021-01-15T16:59:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-485 td/osd-backfill-space/datafile 2021-01-15T16:59:19.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:19.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-486 td/osd-backfill-space/datafile 2021-01-15T16:59:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-487 td/osd-backfill-space/datafile 2021-01-15T16:59:20.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-488 td/osd-backfill-space/datafile 2021-01-15T16:59:20.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-489 td/osd-backfill-space/datafile 2021-01-15T16:59:20.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-490 td/osd-backfill-space/datafile 2021-01-15T16:59:20.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-491 td/osd-backfill-space/datafile 2021-01-15T16:59:20.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-492 td/osd-backfill-space/datafile 2021-01-15T16:59:20.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-493 td/osd-backfill-space/datafile 2021-01-15T16:59:20.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:20.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-494 td/osd-backfill-space/datafile 2021-01-15T16:59:21.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-495 td/osd-backfill-space/datafile 2021-01-15T16:59:21.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-496 td/osd-backfill-space/datafile 2021-01-15T16:59:21.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-497 td/osd-backfill-space/datafile 2021-01-15T16:59:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-498 td/osd-backfill-space/datafile 2021-01-15T16:59:21.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-499 td/osd-backfill-space/datafile 2021-01-15T16:59:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-500 td/osd-backfill-space/datafile 2021-01-15T16:59:21.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:21.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-501 td/osd-backfill-space/datafile 2021-01-15T16:59:22.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-502 td/osd-backfill-space/datafile 2021-01-15T16:59:22.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-503 td/osd-backfill-space/datafile 2021-01-15T16:59:22.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-504 td/osd-backfill-space/datafile 2021-01-15T16:59:22.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-505 td/osd-backfill-space/datafile 2021-01-15T16:59:22.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-506 td/osd-backfill-space/datafile 2021-01-15T16:59:22.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-507 td/osd-backfill-space/datafile 2021-01-15T16:59:22.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-508 td/osd-backfill-space/datafile 2021-01-15T16:59:22.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:22.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-509 td/osd-backfill-space/datafile 2021-01-15T16:59:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-510 td/osd-backfill-space/datafile 2021-01-15T16:59:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-511 td/osd-backfill-space/datafile 2021-01-15T16:59:23.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-512 td/osd-backfill-space/datafile 2021-01-15T16:59:23.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-513 td/osd-backfill-space/datafile 2021-01-15T16:59:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-514 td/osd-backfill-space/datafile 2021-01-15T16:59:23.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-515 td/osd-backfill-space/datafile 2021-01-15T16:59:23.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:23.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-516 td/osd-backfill-space/datafile 2021-01-15T16:59:24.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-517 td/osd-backfill-space/datafile 2021-01-15T16:59:24.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-518 td/osd-backfill-space/datafile 2021-01-15T16:59:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-519 td/osd-backfill-space/datafile 2021-01-15T16:59:24.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-520 td/osd-backfill-space/datafile 2021-01-15T16:59:24.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-521 td/osd-backfill-space/datafile 2021-01-15T16:59:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-522 td/osd-backfill-space/datafile 2021-01-15T16:59:24.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-523 td/osd-backfill-space/datafile 2021-01-15T16:59:24.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:24.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-524 td/osd-backfill-space/datafile 2021-01-15T16:59:25.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-525 td/osd-backfill-space/datafile 2021-01-15T16:59:25.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-526 td/osd-backfill-space/datafile 2021-01-15T16:59:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-527 td/osd-backfill-space/datafile 2021-01-15T16:59:25.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-528 td/osd-backfill-space/datafile 2021-01-15T16:59:25.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-529 td/osd-backfill-space/datafile 2021-01-15T16:59:25.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-530 td/osd-backfill-space/datafile 2021-01-15T16:59:25.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-531 td/osd-backfill-space/datafile 2021-01-15T16:59:25.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:25.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-532 td/osd-backfill-space/datafile 2021-01-15T16:59:26.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-533 td/osd-backfill-space/datafile 2021-01-15T16:59:26.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-534 td/osd-backfill-space/datafile 2021-01-15T16:59:26.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-535 td/osd-backfill-space/datafile 2021-01-15T16:59:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-536 td/osd-backfill-space/datafile 2021-01-15T16:59:26.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-537 td/osd-backfill-space/datafile 2021-01-15T16:59:26.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-538 td/osd-backfill-space/datafile 2021-01-15T16:59:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-539 td/osd-backfill-space/datafile 2021-01-15T16:59:27.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-540 td/osd-backfill-space/datafile 2021-01-15T16:59:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-541 td/osd-backfill-space/datafile 2021-01-15T16:59:27.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-542 td/osd-backfill-space/datafile 2021-01-15T16:59:27.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-543 td/osd-backfill-space/datafile 2021-01-15T16:59:27.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-544 td/osd-backfill-space/datafile 2021-01-15T16:59:27.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-545 td/osd-backfill-space/datafile 2021-01-15T16:59:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-546 td/osd-backfill-space/datafile 2021-01-15T16:59:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-547 td/osd-backfill-space/datafile 2021-01-15T16:59:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-548 td/osd-backfill-space/datafile 2021-01-15T16:59:28.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-549 td/osd-backfill-space/datafile 2021-01-15T16:59:28.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-550 td/osd-backfill-space/datafile 2021-01-15T16:59:28.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-551 td/osd-backfill-space/datafile 2021-01-15T16:59:28.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-552 td/osd-backfill-space/datafile 2021-01-15T16:59:28.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-553 td/osd-backfill-space/datafile 2021-01-15T16:59:28.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:28.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-554 td/osd-backfill-space/datafile 2021-01-15T16:59:29.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-555 td/osd-backfill-space/datafile 2021-01-15T16:59:29.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-556 td/osd-backfill-space/datafile 2021-01-15T16:59:29.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-557 td/osd-backfill-space/datafile 2021-01-15T16:59:29.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-558 td/osd-backfill-space/datafile 2021-01-15T16:59:29.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-559 td/osd-backfill-space/datafile 2021-01-15T16:59:29.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-560 td/osd-backfill-space/datafile 2021-01-15T16:59:29.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-561 td/osd-backfill-space/datafile 2021-01-15T16:59:29.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:29.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-562 td/osd-backfill-space/datafile 2021-01-15T16:59:30.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-563 td/osd-backfill-space/datafile 2021-01-15T16:59:30.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-564 td/osd-backfill-space/datafile 2021-01-15T16:59:30.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-565 td/osd-backfill-space/datafile 2021-01-15T16:59:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-566 td/osd-backfill-space/datafile 2021-01-15T16:59:30.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-567 td/osd-backfill-space/datafile 2021-01-15T16:59:30.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-568 td/osd-backfill-space/datafile 2021-01-15T16:59:30.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-569 td/osd-backfill-space/datafile 2021-01-15T16:59:30.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:30.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-570 td/osd-backfill-space/datafile 2021-01-15T16:59:31.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-571 td/osd-backfill-space/datafile 2021-01-15T16:59:31.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-572 td/osd-backfill-space/datafile 2021-01-15T16:59:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-573 td/osd-backfill-space/datafile 2021-01-15T16:59:31.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-574 td/osd-backfill-space/datafile 2021-01-15T16:59:31.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-575 td/osd-backfill-space/datafile 2021-01-15T16:59:31.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-576 td/osd-backfill-space/datafile 2021-01-15T16:59:31.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-577 td/osd-backfill-space/datafile 2021-01-15T16:59:31.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:31.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-578 td/osd-backfill-space/datafile 2021-01-15T16:59:32.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-579 td/osd-backfill-space/datafile 2021-01-15T16:59:32.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-580 td/osd-backfill-space/datafile 2021-01-15T16:59:32.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-581 td/osd-backfill-space/datafile 2021-01-15T16:59:32.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-582 td/osd-backfill-space/datafile 2021-01-15T16:59:32.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-583 td/osd-backfill-space/datafile 2021-01-15T16:59:32.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-584 td/osd-backfill-space/datafile 2021-01-15T16:59:32.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:32.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-585 td/osd-backfill-space/datafile 2021-01-15T16:59:33.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-586 td/osd-backfill-space/datafile 2021-01-15T16:59:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-587 td/osd-backfill-space/datafile 2021-01-15T16:59:33.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-588 td/osd-backfill-space/datafile 2021-01-15T16:59:33.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-589 td/osd-backfill-space/datafile 2021-01-15T16:59:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-590 td/osd-backfill-space/datafile 2021-01-15T16:59:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-591 td/osd-backfill-space/datafile 2021-01-15T16:59:33.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-592 td/osd-backfill-space/datafile 2021-01-15T16:59:33.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:33.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-593 td/osd-backfill-space/datafile 2021-01-15T16:59:34.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-594 td/osd-backfill-space/datafile 2021-01-15T16:59:34.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-595 td/osd-backfill-space/datafile 2021-01-15T16:59:34.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-596 td/osd-backfill-space/datafile 2021-01-15T16:59:34.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-597 td/osd-backfill-space/datafile 2021-01-15T16:59:34.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-598 td/osd-backfill-space/datafile 2021-01-15T16:59:34.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-599 td/osd-backfill-space/datafile 2021-01-15T16:59:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:482: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T16:59:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:484: TEST_backfill_multi_partial: rados -p test1 put obj-1-600 td/osd-backfill-space/datafile 2021-01-15T16:59:34.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:487: TEST_backfill_multi_partial: ceph pg dump pgs 2021-01-15T16:59:35.337 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T16:59:35.338 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:58:09.112776+0000 0'0 39:23 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:07.000190+0000 0'0 2021-01-15T16:57:07.000190+0000 0 2021-01-15T16:59:35.338 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 554 0 0 0 0 1446494 0 0 54 54 stale+active+clean 2021-01-15T16:57:04.288669+0000 35'554 35:582 [1] 1 [1] 1 0'0 2021-01-15T16:57:01.996196+0000 0'0 2021-01-15T16:57:01.996196+0000 0 2021-01-15T16:59:35.338 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 566 0 0 0 0 579584 0 0 66 66 active+clean 2021-01-15T16:58:09.112982+0000 40'566 40:609 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:04.485874+0000 0'0 2021-01-15T16:57:04.485874+0000 0 2021-01-15T16:59:35.338 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T16:59:35.339 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T16:59:35.339 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T16:59:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:489: TEST_backfill_multi_partial: _objectstore_tool_nowait td/osd-backfill-space 2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2021-01-15T16:59:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/osd-backfill-space 2021-01-15T16:59:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2021-01-15T16:59:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2021-01-15T16:59:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2021-01-15T16:59:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/osd-backfill-space TERM osd.2 2021-01-15T16:59:35.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T16:59:35.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T16:59:35.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T16:59:35.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T16:59:35.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T16:59:35.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T16:59:35.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-backfill-space 2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2021-01-15T16:59:35.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2021-01-15T16:59:35.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:59:35.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2021-01-15T16:59:35.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:59:35.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/2 2021-01-15T16:59:35.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/2 --op export --pgid 2.0 --file td/osd-backfill-space/export.out 2021-01-15T16:59:37.118 INFO:tasks.workunit.client.0.smithi145.stderr:Exporting 2.0 info 2.0( v 40'600 (40'500,40'600] local-lis/les=38/39 n=600 ec=26/26 lis/c=38/38 les/c/f=39/39/0 sis=38) 2021-01-15T16:59:37.127 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:00f101c6:::obj-1-121:head# 2021-01-15T16:59:37.127 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0168051e:::obj-1-571:head# 2021-01-15T16:59:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:01b67738:::obj-1-182:head# 2021-01-15T16:59:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:040dbc59:::obj-1-23:head# 2021-01-15T16:59:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0415ce6e:::obj-1-20:head# 2021-01-15T16:59:37.129 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:058b28d7:::obj-1-572:head# 2021-01-15T16:59:37.130 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:059d2e5c:::obj-1-87:head# 2021-01-15T16:59:37.130 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:05a823ca:::obj-1-52:head# 2021-01-15T16:59:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:05b8b7fe:::obj-1-119:head# 2021-01-15T16:59:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:05c9492b:::obj-1-78:head# 2021-01-15T16:59:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:05e91e96:::obj-1-393:head# 2021-01-15T16:59:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:07bdb780:::obj-1-204:head# 2021-01-15T16:59:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:07cfbd49:::obj-1-318:head# 2021-01-15T16:59:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:084a829f:::obj-1-112:head# 2021-01-15T16:59:37.134 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:08cedd34:::obj-1-438:head# 2021-01-15T16:59:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0941af69:::obj-1-117:head# 2021-01-15T16:59:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:09607265:::obj-1-429:head# 2021-01-15T16:59:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0be86f7e:::obj-1-568:head# 2021-01-15T16:59:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0ca5819d:::obj-1-327:head# 2021-01-15T16:59:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0cd02051:::obj-1-208:head# 2021-01-15T16:59:37.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0d0264b5:::obj-1-171:head# 2021-01-15T16:59:37.137 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0d195060:::obj-1-420:head# 2021-01-15T16:59:37.138 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0d362b19:::obj-1-219:head# 2021-01-15T16:59:37.138 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0dea6f4d:::obj-1-61:head# 2021-01-15T16:59:37.139 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0ecd8390:::obj-1-132:head# 2021-01-15T16:59:37.139 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0f49e2fa:::obj-1-200:head# 2021-01-15T16:59:37.140 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0f7bb95f:::obj-1-268:head# 2021-01-15T16:59:37.141 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0fabf252:::obj-1-441:head# 2021-01-15T16:59:37.142 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:0fb1697a:::obj-1-443:head# 2021-01-15T16:59:37.142 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:105d3efe:::obj-1-490:head# 2021-01-15T16:59:37.142 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:10f52c01:::obj-1-449:head# 2021-01-15T16:59:37.143 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:11965425:::obj-1-335:head# 2021-01-15T16:59:37.143 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:11c9f5f8:::obj-1-565:head# 2021-01-15T16:59:37.144 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:11cfcfb6:::obj-1-588:head# 2021-01-15T16:59:37.174 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:11d46f0a:::obj-1-387:head# 2021-01-15T16:59:37.175 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:11ebb0d6:::obj-1-214:head# 2021-01-15T16:59:37.175 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1295e402:::obj-1-178:head# 2021-01-15T16:59:37.176 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:12c996c1:::obj-1-517:head# 2021-01-15T16:59:37.176 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:133dec15:::obj-1-475:head# 2021-01-15T16:59:37.177 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1475c981:::obj-1-505:head# 2021-01-15T16:59:37.178 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:148e92a4:::obj-1-509:head# 2021-01-15T16:59:37.178 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1490b121:::obj-1-172:head# 2021-01-15T16:59:37.178 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:14ce082c:::obj-1-99:head# 2021-01-15T16:59:37.179 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1552ad99:::obj-1-41:head# 2021-01-15T16:59:37.179 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:15544270:::obj-1-271:head# 2021-01-15T16:59:37.180 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1589d957:::obj-1-86:head# 2021-01-15T16:59:37.180 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:16db3947:::obj-1-477:head# 2021-01-15T16:59:37.181 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:16dc98bd:::obj-1-375:head# 2021-01-15T16:59:37.182 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:17e54db3:::obj-1-196:head# 2021-01-15T16:59:37.183 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:17f2ad3f:::obj-1-10:head# 2021-01-15T16:59:37.183 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:18075e41:::obj-1-55:head# 2021-01-15T16:59:37.184 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:18086253:::obj-1-71:head# 2021-01-15T16:59:37.187 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:191cf8d8:::obj-1-372:head# 2021-01-15T16:59:37.187 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:19311f70:::obj-1-454:head# 2021-01-15T16:59:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:198318e4:::obj-1-464:head# 2021-01-15T16:59:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1999d806:::obj-1-163:head# 2021-01-15T16:59:37.189 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1a92cab7:::obj-1-412:head# 2021-01-15T16:59:37.189 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1b16c0f6:::obj-1-210:head# 2021-01-15T16:59:37.189 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1b53ed46:::obj-1-577:head# 2021-01-15T16:59:37.192 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1b5c5a8d:::obj-1-556:head# 2021-01-15T16:59:37.192 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1c12c05e:::obj-1-79:head# 2021-01-15T16:59:37.193 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1c571ba5:::obj-1-262:head# 2021-01-15T16:59:37.193 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1c5f42b7:::obj-1-92:head# 2021-01-15T16:59:37.193 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1c6ad881:::obj-1-312:head# 2021-01-15T16:59:37.194 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1c89877b:::obj-1-142:head# 2021-01-15T16:59:37.194 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1e4168ca:::obj-1-282:head# 2021-01-15T16:59:37.195 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1e424445:::obj-1-484:head# 2021-01-15T16:59:37.195 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1e467959:::obj-1-22:head# 2021-01-15T16:59:37.195 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1eaf7e7c:::obj-1-392:head# 2021-01-15T16:59:37.197 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1f06c86d:::obj-1-307:head# 2021-01-15T16:59:37.198 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1f2ee1f8:::obj-1-306:head# 2021-01-15T16:59:37.198 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:1f4ef040:::obj-1-349:head# 2021-01-15T16:59:37.199 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2039481f:::obj-1-545:head# 2021-01-15T16:59:37.199 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:22233df0:::obj-1-543:head# 2021-01-15T16:59:37.199 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2225184e:::obj-1-266:head# 2021-01-15T16:59:37.200 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:234766bb:::obj-1-281:head# 2021-01-15T16:59:37.200 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2372e022:::obj-1-205:head# 2021-01-15T16:59:37.200 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2379547b:::obj-1-113:head# 2021-01-15T16:59:37.201 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:23ae14bf:::obj-1-531:head# 2021-01-15T16:59:37.201 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:23bc5f90:::obj-1-452:head# 2021-01-15T16:59:37.202 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:23fbef3d:::obj-1-82:head# 2021-01-15T16:59:37.202 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:24106c2b:::obj-1-232:head# 2021-01-15T16:59:37.203 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2529b4e4:::obj-1-75:head# 2021-01-15T16:59:37.204 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:25a24ae7:::obj-1-459:head# 2021-01-15T16:59:37.205 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:26627aa9:::obj-1-7:head# 2021-01-15T16:59:37.205 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:26b77221:::obj-1-227:head# 2021-01-15T16:59:37.207 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:27450826:::obj-1-344:head# 2021-01-15T16:59:37.207 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:278905af:::obj-1-592:head# 2021-01-15T16:59:37.208 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:27ad0d2d:::obj-1-136:head# 2021-01-15T16:59:37.208 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:28a8c213:::obj-1-118:head# 2021-01-15T16:59:37.208 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:28c0c009:::obj-1-74:head# 2021-01-15T16:59:37.209 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2a214708:::obj-1-260:head# 2021-01-15T16:59:37.209 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2a67e195:::obj-1-474:head# 2021-01-15T16:59:37.210 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2aa852ff:::obj-1-5:head# 2021-01-15T16:59:37.210 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2ba1fb52:::obj-1-563:head# 2021-01-15T16:59:37.211 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2bae8cb9:::obj-1-320:head# 2021-01-15T16:59:37.211 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2c569d02:::obj-1-139:head# 2021-01-15T16:59:37.212 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2c71d691:::obj-1-201:head# 2021-01-15T16:59:37.212 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2c88e92c:::obj-1-389:head# 2021-01-15T16:59:37.213 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2d659109:::obj-1-580:head# 2021-01-15T16:59:37.213 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2d74ea60:::obj-1-230:head# 2021-01-15T16:59:37.214 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2de96edf:::obj-1-233:head# 2021-01-15T16:59:37.214 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2e43da40:::obj-1-382:head# 2021-01-15T16:59:37.215 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2e530c9f:::obj-1-358:head# 2021-01-15T16:59:37.215 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:2ed02239:::obj-1-81:head# 2021-01-15T16:59:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:30264a0c:::obj-1-373:head# 2021-01-15T16:59:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:302ca5e7:::obj-1-206:head# 2021-01-15T16:59:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:309878e8:::obj-1-508:head# 2021-01-15T16:59:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:309a0ac3:::obj-1-188:head# 2021-01-15T16:59:37.217 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:30f030e9:::obj-1-53:head# 2021-01-15T16:59:37.218 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:31242a73:::obj-1-528:head# 2021-01-15T16:59:37.218 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:31463825:::obj-1-34:head# 2021-01-15T16:59:37.219 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3166dc86:::obj-1-355:head# 2021-01-15T16:59:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:31baa2fa:::obj-1-252:head# 2021-01-15T16:59:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:31d8935e:::obj-1-193:head# 2021-01-15T16:59:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:32670b14:::obj-1-305:head# 2021-01-15T16:59:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:33da02fa:::obj-1-114:head# 2021-01-15T16:59:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:34373133:::obj-1-446:head# 2021-01-15T16:59:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:347b7083:::obj-1-89:head# 2021-01-15T16:59:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:349125c1:::obj-1-148:head# 2021-01-15T16:59:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:34b80eee:::obj-1-138:head# 2021-01-15T16:59:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:34db48f8:::obj-1-49:head# 2021-01-15T16:59:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3669ff0c:::obj-1-313:head# 2021-01-15T16:59:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:36c1b610:::obj-1-249:head# 2021-01-15T16:59:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:374c250b:::obj-1-324:head# 2021-01-15T16:59:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3761a149:::obj-1-342:head# 2021-01-15T16:59:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:379ff95a:::obj-1-341:head# 2021-01-15T16:59:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:37dc15a6:::obj-1-26:head# 2021-01-15T16:59:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:38130e33:::obj-1-552:head# 2021-01-15T16:59:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:39c44880:::obj-1-538:head# 2021-01-15T16:59:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3a08dec9:::obj-1-368:head# 2021-01-15T16:59:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3a2b46a6:::obj-1-407:head# 2021-01-15T16:59:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3a7dfe0a:::obj-1-285:head# 2021-01-15T16:59:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3aebb144:::obj-1-248:head# 2021-01-15T16:59:37.229 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3b08b36f:::obj-1-310:head# 2021-01-15T16:59:37.229 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3b28ffa3:::obj-1-295:head# 2021-01-15T16:59:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3ba79bd1:::obj-1-444:head# 2021-01-15T16:59:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3c05df8d:::obj-1-319:head# 2021-01-15T16:59:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3c57e6a9:::obj-1-436:head# 2021-01-15T16:59:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3c73d8ee:::obj-1-298:head# 2021-01-15T16:59:37.237 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3cb6ae2a:::obj-1-400:head# 2021-01-15T16:59:37.237 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3cb71491:::obj-1-328:head# 2021-01-15T16:59:37.238 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3cb842fe:::obj-1-541:head# 2021-01-15T16:59:37.238 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3cf6835e:::obj-1-496:head# 2021-01-15T16:59:37.239 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3cf78507:::obj-1-573:head# 2021-01-15T16:59:37.239 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3d0dab84:::obj-1-554:head# 2021-01-15T16:59:37.240 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3d1a5f9d:::obj-1-323:head# 2021-01-15T16:59:37.240 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3da5f3a2:::obj-1-414:head# 2021-01-15T16:59:37.241 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3dbb65f0:::obj-1-440:head# 2021-01-15T16:59:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3dc94618:::obj-1-223:head# 2021-01-15T16:59:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3df3d5ab:::obj-1-25:head# 2021-01-15T16:59:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3e167988:::obj-1-336:head# 2021-01-15T16:59:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3e4ef3bd:::obj-1-104:head# 2021-01-15T16:59:37.252 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3e9e1efd:::obj-1-357:head# 2021-01-15T16:59:37.252 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3ebbe3a7:::obj-1-42:head# 2021-01-15T16:59:37.253 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3ec80373:::obj-1-403:head# 2021-01-15T16:59:37.253 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3ed086e4:::obj-1-13:head# 2021-01-15T16:59:37.254 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:3fe8e157:::obj-1-164:head# 2021-01-15T16:59:37.254 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:401ddfd4:::obj-1-379:head# 2021-01-15T16:59:37.255 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4092f6cd:::obj-1-430:head# 2021-01-15T16:59:37.256 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:414bd7cc:::obj-1-562:head# 2021-01-15T16:59:37.256 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:422bd9d0:::obj-1-287:head# 2021-01-15T16:59:37.259 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4255002d:::obj-1-578:head# 2021-01-15T16:59:37.260 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4270725c:::obj-1-109:head# 2021-01-15T16:59:37.260 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:42836669:::obj-1-31:head# 2021-01-15T16:59:37.261 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:428d93ef:::obj-1-32:head# 2021-01-15T16:59:37.261 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4327ef5b:::obj-1-453:head# 2021-01-15T16:59:37.262 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:440eb07b:::obj-1-236:head# 2021-01-15T16:59:37.262 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:442a028a:::obj-1-365:head# 2021-01-15T16:59:37.263 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:449ca54a:::obj-1-161:head# 2021-01-15T16:59:37.263 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:45128c3b:::obj-1-343:head# 2021-01-15T16:59:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:45801bf6:::obj-1-264:head# 2021-01-15T16:59:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:45896f61:::obj-1-48:head# 2021-01-15T16:59:37.265 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:459b7f1c:::obj-1-190:head# 2021-01-15T16:59:37.265 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:466af96a:::obj-1-486:head# 2021-01-15T16:59:37.266 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:47387ce4:::obj-1-575:head# 2021-01-15T16:59:37.267 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:47c94cc8:::obj-1-564:head# 2021-01-15T16:59:37.267 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:47d7f8ed:::obj-1-544:head# 2021-01-15T16:59:37.268 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:47fc0ec3:::obj-1-558:head# 2021-01-15T16:59:37.269 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:48ba42d8:::obj-1-156:head# 2021-01-15T16:59:37.269 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:494e269e:::obj-1-600:head# 2021-01-15T16:59:37.270 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:49646095:::obj-1-73:head# 2021-01-15T16:59:37.270 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:49eb65ab:::obj-1-374:head# 2021-01-15T16:59:37.270 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4a0075a4:::obj-1-432:head# 2021-01-15T16:59:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4af1ffc2:::obj-1-72:head# 2021-01-15T16:59:37.272 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4bc1c46f:::obj-1-346:head# 2021-01-15T16:59:37.272 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4bd55e68:::obj-1-93:head# 2021-01-15T16:59:37.273 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4c6d2760:::obj-1-198:head# 2021-01-15T16:59:37.273 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4cb9be3b:::obj-1-451:head# 2021-01-15T16:59:37.274 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4d260a64:::obj-1-514:head# 2021-01-15T16:59:37.278 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4d7d6284:::obj-1-583:head# 2021-01-15T16:59:37.282 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4e79ac02:::obj-1-437:head# 2021-01-15T16:59:37.285 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4e980099:::obj-1-507:head# 2021-01-15T16:59:37.286 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4ea8f99c:::obj-1-211:head# 2021-01-15T16:59:37.286 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4eabdc2a:::obj-1-590:head# 2021-01-15T16:59:37.286 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4f97fd73:::obj-1-243:head# 2021-01-15T16:59:37.287 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4fe00224:::obj-1-229:head# 2021-01-15T16:59:37.287 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4fead9ab:::obj-1-476:head# 2021-01-15T16:59:37.287 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:4ffddfd7:::obj-1-462:head# 2021-01-15T16:59:37.288 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5085c04d:::obj-1-100:head# 2021-01-15T16:59:37.288 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:50a8b465:::obj-1-574:head# 2021-01-15T16:59:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:518a151f:::obj-1-189:head# 2021-01-15T16:59:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:519afc95:::obj-1-314:head# 2021-01-15T16:59:37.290 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:52199781:::obj-1-202:head# 2021-01-15T16:59:37.290 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:52313062:::obj-1-337:head# 2021-01-15T16:59:37.291 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:526a5196:::obj-1-225:head# 2021-01-15T16:59:37.291 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:53128d32:::obj-1-483:head# 2021-01-15T16:59:37.292 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:53c2bb21:::obj-1-532:head# 2021-01-15T16:59:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:53fa4f3c:::obj-1-423:head# 2021-01-15T16:59:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:541c1d30:::obj-1-143:head# 2021-01-15T16:59:37.294 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:548d20c3:::obj-1-352:head# 2021-01-15T16:59:37.294 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:54e4c6f9:::obj-1-101:head# 2021-01-15T16:59:37.295 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:55016e06:::obj-1-276:head# 2021-01-15T16:59:37.295 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5510c784:::obj-1-491:head# 2021-01-15T16:59:37.296 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:551275ef:::obj-1-239:head# 2021-01-15T16:59:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:556df001:::obj-1-46:head# 2021-01-15T16:59:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:558fb5b1:::obj-1-536:head# 2021-01-15T16:59:37.298 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:56390d98:::obj-1-64:head# 2021-01-15T16:59:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:564f962a:::obj-1-125:head# 2021-01-15T16:59:37.299 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:56936e94:::obj-1-251:head# 2021-01-15T16:59:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:57585d3d:::obj-1-294:head# 2021-01-15T16:59:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5764cf7b:::obj-1-479:head# 2021-01-15T16:59:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5782341a:::obj-1-218:head# 2021-01-15T16:59:37.301 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:57d07a1a:::obj-1-317:head# 2021-01-15T16:59:37.302 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:58700205:::obj-1-176:head# 2021-01-15T16:59:37.302 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:58aee718:::obj-1-434:head# 2021-01-15T16:59:37.303 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:594c9d52:::obj-1-458:head# 2021-01-15T16:59:37.303 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5a70b555:::obj-1-279:head# 2021-01-15T16:59:37.304 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5b02568b:::obj-1-586:head# 2021-01-15T16:59:37.304 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5b69cdcf:::obj-1-391:head# 2021-01-15T16:59:37.305 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5cf151b5:::obj-1-510:head# 2021-01-15T16:59:37.306 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5d7fd07f:::obj-1-152:head# 2021-01-15T16:59:37.306 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5de288d7:::obj-1-235:head# 2021-01-15T16:59:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5ec497cd:::obj-1-44:head# 2021-01-15T16:59:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5fb3a09f:::obj-1-518:head# 2021-01-15T16:59:37.308 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:5ff3c2ef:::obj-1-470:head# 2021-01-15T16:59:37.309 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:601efcb1:::obj-1-549:head# 2021-01-15T16:59:37.309 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:60831338:::obj-1-263:head# 2021-01-15T16:59:37.310 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6092777b:::obj-1-145:head# 2021-01-15T16:59:37.310 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:60b17206:::obj-1-411:head# 2021-01-15T16:59:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:610c957c:::obj-1-278:head# 2021-01-15T16:59:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:613b5ef4:::obj-1-504:head# 2021-01-15T16:59:37.312 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:61ed7d60:::obj-1-274:head# 2021-01-15T16:59:37.312 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:621b6751:::obj-1-212:head# 2021-01-15T16:59:37.313 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:62b0aba9:::obj-1-253:head# 2021-01-15T16:59:37.314 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:64017240:::obj-1-102:head# 2021-01-15T16:59:37.314 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:645f4f49:::obj-1-332:head# 2021-01-15T16:59:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:65db82b9:::obj-1-334:head# 2021-01-15T16:59:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:667422e5:::obj-1-465:head# 2021-01-15T16:59:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:66c50bc2:::obj-1-322:head# 2021-01-15T16:59:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:681a4577:::obj-1-511:head# 2021-01-15T16:59:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6874e007:::obj-1-398:head# 2021-01-15T16:59:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:693746c1:::obj-1-582:head# 2021-01-15T16:59:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:69638b0c:::obj-1-8:head# 2021-01-15T16:59:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6967de6e:::obj-1-331:head# 2021-01-15T16:59:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:69da97aa:::obj-1-340:head# 2021-01-15T16:59:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6b3a50a6:::obj-1-144:head# 2021-01-15T16:59:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6be4ef26:::obj-1-579:head# 2021-01-15T16:59:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6d735549:::obj-1-599:head# 2021-01-15T16:59:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6e22c25e:::obj-1-9:head# 2021-01-15T16:59:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6e2ced4e:::obj-1-370:head# 2021-01-15T16:59:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6e961169:::obj-1-428:head# 2021-01-15T16:59:37.323 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6efcbbee:::obj-1-168:head# 2021-01-15T16:59:37.323 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6f20741c:::obj-1-33:head# 2021-01-15T16:59:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6fe4c885:::obj-1-273:head# 2021-01-15T16:59:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:6fffdea2:::obj-1-425:head# 2021-01-15T16:59:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:70716852:::obj-1-593:head# 2021-01-15T16:59:37.325 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:70919606:::obj-1-515:head# 2021-01-15T16:59:37.326 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:70c39d76:::obj-1-467:head# 2021-01-15T16:59:37.327 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:70f57ad3:::obj-1-361:head# 2021-01-15T16:59:37.327 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7177c7b5:::obj-1-503:head# 2021-01-15T16:59:37.328 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:719b69bd:::obj-1-497:head# 2021-01-15T16:59:37.329 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:724f9347:::obj-1-6:head# 2021-01-15T16:59:37.330 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:726d9f6b:::obj-1-111:head# 2021-01-15T16:59:37.330 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:72a91baf:::obj-1-439:head# 2021-01-15T16:59:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:72aa43ee:::obj-1-534:head# 2021-01-15T16:59:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:72c40e11:::obj-1-288:head# 2021-01-15T16:59:37.332 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:74196d45:::obj-1-246:head# 2021-01-15T16:59:37.332 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:743f42b1:::obj-1-326:head# 2021-01-15T16:59:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7523b538:::obj-1-494:head# 2021-01-15T16:59:37.334 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:75310ed6:::obj-1-364:head# 2021-01-15T16:59:37.334 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7568e0bc:::obj-1-585:head# 2021-01-15T16:59:37.335 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:760219a3:::obj-1-353:head# 2021-01-15T16:59:37.335 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:763338c9:::obj-1-555:head# 2021-01-15T16:59:37.336 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:77bc2d65:::obj-1-386:head# 2021-01-15T16:59:37.336 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:77ed0c7a:::obj-1-492:head# 2021-01-15T16:59:37.337 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:785a4052:::obj-1-456:head# 2021-01-15T16:59:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:796799c3:::obj-1-195:head# 2021-01-15T16:59:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:79caf0c1:::obj-1-445:head# 2021-01-15T16:59:37.339 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7af5b3e4:::obj-1-217:head# 2021-01-15T16:59:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7b3a9063:::obj-1-169:head# 2021-01-15T16:59:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7b509a92:::obj-1-561:head# 2021-01-15T16:59:37.341 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7bc646b1:::obj-1-415:head# 2021-01-15T16:59:37.341 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7c2f9188:::obj-1-482:head# 2021-01-15T16:59:37.342 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7d0da936:::obj-1-11:head# 2021-01-15T16:59:37.343 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7d25ca2d:::obj-1-242:head# 2021-01-15T16:59:37.343 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7d64f81e:::obj-1-473:head# 2021-01-15T16:59:37.344 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7ef2ea1b:::obj-1-417:head# 2021-01-15T16:59:37.345 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7f52b238:::obj-1-308:head# 2021-01-15T16:59:37.350 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:7f7a313c:::obj-1-551:head# 2021-01-15T16:59:37.350 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:806827a7:::obj-1-472:head# 2021-01-15T16:59:37.351 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8112c3ab:::obj-1-45:head# 2021-01-15T16:59:37.352 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:81365b55:::obj-1-315:head# 2021-01-15T16:59:37.352 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:818ffdc3:::obj-1-345:head# 2021-01-15T16:59:37.353 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:81babab8:::obj-1-535:head# 2021-01-15T16:59:37.353 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8288f652:::obj-1-347:head# 2021-01-15T16:59:37.353 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8312eadd:::obj-1-14:head# 2021-01-15T16:59:37.354 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:83b89bc7:::obj-1-137:head# 2021-01-15T16:59:37.354 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:84a12321:::obj-1-134:head# 2021-01-15T16:59:37.355 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:84b26d62:::obj-1-408:head# 2021-01-15T16:59:37.356 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:84bbb26e:::obj-1-43:head# 2021-01-15T16:59:37.356 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:86fc0014:::obj-1-222:head# 2021-01-15T16:59:37.356 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:87488bf4:::obj-1-174:head# 2021-01-15T16:59:37.357 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8754389a:::obj-1-65:head# 2021-01-15T16:59:37.357 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:87a6ad67:::obj-1-366:head# 2021-01-15T16:59:37.358 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:87d32b83:::obj-1-513:head# 2021-01-15T16:59:37.359 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8813a346:::obj-1-540:head# 2021-01-15T16:59:37.359 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:88146951:::obj-1-216:head# 2021-01-15T16:59:37.360 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8820800d:::obj-1-83:head# 2021-01-15T16:59:37.360 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:88319745:::obj-1-98:head# 2021-01-15T16:59:37.361 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:88c969b8:::obj-1-54:head# 2021-01-15T16:59:37.361 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:895a2d53:::obj-1-150:head# 2021-01-15T16:59:37.362 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8ae006a0:::obj-1-241:head# 2021-01-15T16:59:37.362 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8bd44e64:::obj-1-300:head# 2021-01-15T16:59:37.363 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8beb257f:::obj-1-3:head# 2021-01-15T16:59:37.364 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8cd8c743:::obj-1-527:head# 2021-01-15T16:59:37.364 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8daa99c7:::obj-1-237:head# 2021-01-15T16:59:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8f5bccbf:::obj-1-59:head# 2021-01-15T16:59:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8f6d9945:::obj-1-151:head# 2021-01-15T16:59:37.366 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8f7c568c:::obj-1-261:head# 2021-01-15T16:59:37.366 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:8fbc59a9:::obj-1-589:head# 2021-01-15T16:59:37.367 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:90225d94:::obj-1-194:head# 2021-01-15T16:59:37.367 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:904c50d3:::obj-1-63:head# 2021-01-15T16:59:37.368 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:90e1bd79:::obj-1-450:head# 2021-01-15T16:59:37.368 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9274b06d:::obj-1-460:head# 2021-01-15T16:59:37.369 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:92af41f5:::obj-1-409:head# 2021-01-15T16:59:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:935a08d6:::obj-1-192:head# 2021-01-15T16:59:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:935f8a13:::obj-1-24:head# 2021-01-15T16:59:37.371 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:95230148:::obj-1-280:head# 2021-01-15T16:59:37.372 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:95342eb0:::obj-1-390:head# 2021-01-15T16:59:37.372 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9659d09c:::obj-1-292:head# 2021-01-15T16:59:37.373 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:970e8cb4:::obj-1-18:head# 2021-01-15T16:59:37.373 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:97d21cf1:::obj-1-80:head# 2021-01-15T16:59:37.374 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:982bdb12:::obj-1-302:head# 2021-01-15T16:59:37.374 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:985dfd75:::obj-1-91:head# 2021-01-15T16:59:37.375 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:986d245a:::obj-1-154:head# 2021-01-15T16:59:37.375 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:98a03e99:::obj-1-529:head# 2021-01-15T16:59:37.376 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:98b7103e:::obj-1-275:head# 2021-01-15T16:59:37.376 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:990f0ecf:::obj-1-427:head# 2021-01-15T16:59:37.376 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:996c1f26:::obj-1-96:head# 2021-01-15T16:59:37.377 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:99bd2018:::obj-1-17:head# 2021-01-15T16:59:37.377 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9a7434c9:::obj-1-469:head# 2021-01-15T16:59:37.378 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9a8e36cc:::obj-1-598:head# 2021-01-15T16:59:37.378 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9abcc369:::obj-1-442:head# 2021-01-15T16:59:37.379 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9b0ea06e:::obj-1-231:head# 2021-01-15T16:59:37.379 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9b34e80d:::obj-1-378:head# 2021-01-15T16:59:37.380 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9b55a2a9:::obj-1-60:head# 2021-01-15T16:59:37.380 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9be3f2da:::obj-1-597:head# 2021-01-15T16:59:37.381 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9bf62c88:::obj-1-447:head# 2021-01-15T16:59:37.381 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9c31857b:::obj-1-272:head# 2021-01-15T16:59:37.382 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9c533e91:::obj-1-19:head# 2021-01-15T16:59:37.383 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9ca70688:::obj-1-570:head# 2021-01-15T16:59:37.383 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9ccce337:::obj-1-147:head# 2021-01-15T16:59:37.384 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9d4b2dfc:::obj-1-267:head# 2021-01-15T16:59:37.384 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9d4df969:::obj-1-286:head# 2021-01-15T16:59:37.385 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9d714651:::obj-1-495:head# 2021-01-15T16:59:37.385 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9da65518:::obj-1-498:head# 2021-01-15T16:59:37.386 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9e94f27b:::obj-1-209:head# 2021-01-15T16:59:37.386 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9f39e910:::obj-1-354:head# 2021-01-15T16:59:37.387 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9f8f9ef4:::obj-1-226:head# 2021-01-15T16:59:37.387 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:9fcd3f36:::obj-1-47:head# 2021-01-15T16:59:37.388 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a0326d7d:::obj-1-338:head# 2021-01-15T16:59:37.388 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a05e7475:::obj-1-62:head# 2021-01-15T16:59:37.389 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a0955052:::obj-1-533:head# 2021-01-15T16:59:37.389 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a0aa6848:::obj-1-38:head# 2021-01-15T16:59:37.389 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a0fdc285:::obj-1-133:head# 2021-01-15T16:59:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a10290b8:::obj-1-245:head# 2021-01-15T16:59:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a118a8a8:::obj-1-500:head# 2021-01-15T16:59:37.391 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a11a4079:::obj-1-199:head# 2021-01-15T16:59:37.391 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a1b3ec36:::obj-1-15:head# 2021-01-15T16:59:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a1f5079b:::obj-1-468:head# 2021-01-15T16:59:37.393 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a1ff04b5:::obj-1-167:head# 2021-01-15T16:59:37.394 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a21cc437:::obj-1-128:head# 2021-01-15T16:59:37.394 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a2966c77:::obj-1-140:head# 2021-01-15T16:59:37.395 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a321e537:::obj-1-184:head# 2021-01-15T16:59:37.395 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a33418d8:::obj-1-488:head# 2021-01-15T16:59:37.396 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a38b1c1a:::obj-1-584:head# 2021-01-15T16:59:37.396 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a3f83221:::obj-1-158:head# 2021-01-15T16:59:37.397 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a46e33eb:::obj-1-106:head# 2021-01-15T16:59:37.397 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a491460d:::obj-1-421:head# 2021-01-15T16:59:37.398 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a4dd813b:::obj-1-530:head# 2021-01-15T16:59:37.398 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a4e0d96d:::obj-1-351:head# 2021-01-15T16:59:37.399 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a52657b4:::obj-1-557:head# 2021-01-15T16:59:37.399 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a5b3bca6:::obj-1-277:head# 2021-01-15T16:59:37.400 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a5f5a9fb:::obj-1-170:head# 2021-01-15T16:59:37.400 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a6aeed57:::obj-1-141:head# 2021-01-15T16:59:37.401 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a6dda70f:::obj-1-471:head# 2021-01-15T16:59:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a71186c0:::obj-1-406:head# 2021-01-15T16:59:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a764ad8c:::obj-1-221:head# 2021-01-15T16:59:37.403 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a7b63473:::obj-1-131:head# 2021-01-15T16:59:37.403 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a88042fa:::obj-1-296:head# 2021-01-15T16:59:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:a8805fe0:::obj-1-213:head# 2021-01-15T16:59:37.404 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:aa4f12b1:::obj-1-270:head# 2021-01-15T16:59:37.405 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ab27f701:::obj-1-399:head# 2021-01-15T16:59:37.405 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ab8027c1:::obj-1-360:head# 2021-01-15T16:59:37.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ac3ef8ab:::obj-1-371:head# 2021-01-15T16:59:37.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ac4db76d:::obj-1-405:head# 2021-01-15T16:59:37.406 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ad53cb29:::obj-1-256:head# 2021-01-15T16:59:37.407 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ad704c16:::obj-1-485:head# 2021-01-15T16:59:37.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:adec4920:::obj-1-107:head# 2021-01-15T16:59:37.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ae3a94b3:::obj-1-95:head# 2021-01-15T16:59:37.408 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ae92616e:::obj-1-448:head# 2021-01-15T16:59:37.409 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ae975168:::obj-1-426:head# 2021-01-15T16:59:37.410 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:aee8fd62:::obj-1-27:head# 2021-01-15T16:59:37.410 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:af38aa87:::obj-1-50:head# 2021-01-15T16:59:37.411 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b0327eb3:::obj-1-380:head# 2021-01-15T16:59:37.411 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b03cd166:::obj-1-203:head# 2021-01-15T16:59:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b06f06e0:::obj-1-126:head# 2021-01-15T16:59:37.412 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b0972538:::obj-1-506:head# 2021-01-15T16:59:37.413 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b0cfb86a:::obj-1-359:head# 2021-01-15T16:59:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b10adb1c:::obj-1-333:head# 2021-01-15T16:59:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b20b9004:::obj-1-401:head# 2021-01-15T16:59:37.415 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b2113757:::obj-1-4:head# 2021-01-15T16:59:37.415 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b22441e9:::obj-1-29:head# 2021-01-15T16:59:37.416 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b2472384:::obj-1-207:head# 2021-01-15T16:59:37.416 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b2604227:::obj-1-502:head# 2021-01-15T16:59:37.417 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b3277f89:::obj-1-175:head# 2021-01-15T16:59:37.417 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b3d09e0d:::obj-1-258:head# 2021-01-15T16:59:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b3e9969a:::obj-1-499:head# 2021-01-15T16:59:37.419 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b4892e7d:::obj-1-155:head# 2021-01-15T16:59:37.419 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b4b76084:::obj-1-581:head# 2021-01-15T16:59:37.420 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b6e47ae5:::obj-1-215:head# 2021-01-15T16:59:37.420 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b75af5ff:::obj-1-228:head# 2021-01-15T16:59:37.421 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b8e2e20f:::obj-1-220:head# 2021-01-15T16:59:37.421 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:b8e80c4e:::obj-1-576:head# 2021-01-15T16:59:37.422 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ba502aee:::obj-1-255:head# 2021-01-15T16:59:37.422 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ba9c3e6e:::obj-1-57:head# 2021-01-15T16:59:37.423 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bb068062:::obj-1-461:head# 2021-01-15T16:59:37.424 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bb16c115:::obj-1-157:head# 2021-01-15T16:59:37.424 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bb447914:::obj-1-481:head# 2021-01-15T16:59:37.425 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bb6c8e0f:::obj-1-348:head# 2021-01-15T16:59:37.425 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bb77d6d0:::obj-1-301:head# 2021-01-15T16:59:37.426 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bc51c5f6:::obj-1-422:head# 2021-01-15T16:59:37.426 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bc838d5c:::obj-1-35:head# 2021-01-15T16:59:37.427 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bceae5eb:::obj-1-56:head# 2021-01-15T16:59:37.427 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bea17ef6:::obj-1-84:head# 2021-01-15T16:59:37.428 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:beb88be5:::obj-1-396:head# 2021-01-15T16:59:37.428 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:beed7b18:::obj-1-569:head# 2021-01-15T16:59:37.429 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bf0bff49:::obj-1-385:head# 2021-01-15T16:59:37.430 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bf545d5a:::obj-1-69:head# 2021-01-15T16:59:37.430 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bf560a7f:::obj-1-40:head# 2021-01-15T16:59:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:bff6999c:::obj-1-240:head# 2021-01-15T16:59:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c0a33d38:::obj-1-321:head# 2021-01-15T16:59:37.432 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c1e4c9b4:::obj-1-299:head# 2021-01-15T16:59:37.432 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c1ec515a:::obj-1-16:head# 2021-01-15T16:59:37.433 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c22de3c8:::obj-1-110:head# 2021-01-15T16:59:37.434 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c231cd7c:::obj-1-58:head# 2021-01-15T16:59:37.434 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c2ee6ace:::obj-1-186:head# 2021-01-15T16:59:37.435 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c32bfef2:::obj-1-181:head# 2021-01-15T16:59:37.435 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c575a298:::obj-1-466:head# 2021-01-15T16:59:37.436 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c58c3f80:::obj-1-493:head# 2021-01-15T16:59:37.437 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c6442cf2:::obj-1-455:head# 2021-01-15T16:59:37.437 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c6619413:::obj-1-37:head# 2021-01-15T16:59:37.438 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c6c2f8e5:::obj-1-519:head# 2021-01-15T16:59:37.439 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c6f811a2:::obj-1-180:head# 2021-01-15T16:59:37.439 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c72af975:::obj-1-21:head# 2021-01-15T16:59:37.440 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c73eb350:::obj-1-356:head# 2021-01-15T16:59:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c7a683da:::obj-1-410:head# 2021-01-15T16:59:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c7eb9bcc:::obj-1-553:head# 2021-01-15T16:59:37.442 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c886df99:::obj-1-269:head# 2021-01-15T16:59:37.442 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c932e3ec:::obj-1-135:head# 2021-01-15T16:59:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c9b6507a:::obj-1-520:head# 2021-01-15T16:59:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:c9e3b064:::obj-1-244:head# 2021-01-15T16:59:37.444 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cb705d66:::obj-1-377:head# 2021-01-15T16:59:37.445 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cc80e25c:::obj-1-115:head# 2021-01-15T16:59:37.445 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cd1ec88e:::obj-1-297:head# 2021-01-15T16:59:37.446 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cd7e6936:::obj-1-234:head# 2021-01-15T16:59:37.446 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ce32b69c:::obj-1-30:head# 2021-01-15T16:59:37.446 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf2b123c:::obj-1-293:head# 2021-01-15T16:59:37.447 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf2e80a9:::obj-1-542:head# 2021-01-15T16:59:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf51da6a:::obj-1-284:head# 2021-01-15T16:59:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf590f42:::obj-1-381:head# 2021-01-15T16:59:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf68a2b1:::obj-1-68:head# 2021-01-15T16:59:37.449 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cf6aefb6:::obj-1-28:head# 2021-01-15T16:59:37.449 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:cfd8e6af:::obj-1-105:head# 2021-01-15T16:59:37.450 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d0640bad:::obj-1-254:head# 2021-01-15T16:59:37.450 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d101cda5:::obj-1-259:head# 2021-01-15T16:59:37.450 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d1be9386:::obj-1-416:head# 2021-01-15T16:59:37.451 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d243f036:::obj-1-265:head# 2021-01-15T16:59:37.451 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d2808c34:::obj-1-159:head# 2021-01-15T16:59:37.452 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d3415824:::obj-1-39:head# 2021-01-15T16:59:37.452 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d424f07a:::obj-1-547:head# 2021-01-15T16:59:37.453 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d44139f2:::obj-1-289:head# 2021-01-15T16:59:37.453 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d46015ab:::obj-1-546:head# 2021-01-15T16:59:37.454 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d4e503db:::obj-1-525:head# 2021-01-15T16:59:37.454 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d4e97d18:::obj-1-85:head# 2021-01-15T16:59:37.455 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d511492e:::obj-1-388:head# 2021-01-15T16:59:37.455 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d526a7fa:::obj-1-12:head# 2021-01-15T16:59:37.456 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d5667b59:::obj-1-369:head# 2021-01-15T16:59:37.456 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d57f2261:::obj-1-419:head# 2021-01-15T16:59:37.457 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d57f281b:::obj-1-250:head# 2021-01-15T16:59:37.457 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d6e05894:::obj-1-166:head# 2021-01-15T16:59:37.458 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d7689481:::obj-1-424:head# 2021-01-15T16:59:37.458 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d7d27743:::obj-1-129:head# 2021-01-15T16:59:37.459 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d7e4d3e9:::obj-1-127:head# 2021-01-15T16:59:37.460 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d7f1422a:::obj-1-594:head# 2021-01-15T16:59:37.460 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d8469fd2:::obj-1-153:head# 2021-01-15T16:59:37.461 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d84efd9e:::obj-1-435:head# 2021-01-15T16:59:37.461 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d883004b:::obj-1-162:head# 2021-01-15T16:59:37.461 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d8b60854:::obj-1-257:head# 2021-01-15T16:59:37.462 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d91aaae6:::obj-1-1:head# 2021-01-15T16:59:37.462 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d98695ca:::obj-1-524:head# 2021-01-15T16:59:37.463 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:d9ae97c8:::obj-1-177:head# 2021-01-15T16:59:37.463 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:da1713bd:::obj-1-309:head# 2021-01-15T16:59:37.464 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:da37824c:::obj-1-2:head# 2021-01-15T16:59:37.464 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:dab221b0:::obj-1-526:head# 2021-01-15T16:59:37.464 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:db27d885:::obj-1-363:head# 2021-01-15T16:59:37.465 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:dc001d76:::obj-1-329:head# 2021-01-15T16:59:37.465 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:dc4f4119:::obj-1-122:head# 2021-01-15T16:59:37.466 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:dcc36eb7:::obj-1-283:head# 2021-01-15T16:59:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:df7da06c:::obj-1-124:head# 2021-01-15T16:59:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:df7fcacb:::obj-1-330:head# 2021-01-15T16:59:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e05cbe31:::obj-1-413:head# 2021-01-15T16:59:37.468 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e1060a54:::obj-1-384:head# 2021-01-15T16:59:37.468 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e15b839b:::obj-1-316:head# 2021-01-15T16:59:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e161b6e5:::obj-1-103:head# 2021-01-15T16:59:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e1d1d79c:::obj-1-512:head# 2021-01-15T16:59:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e224b482:::obj-1-501:head# 2021-01-15T16:59:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e22d6f76:::obj-1-70:head# 2021-01-15T16:59:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e2bd801c:::obj-1-523:head# 2021-01-15T16:59:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e3d9f395:::obj-1-397:head# 2021-01-15T16:59:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e474ea25:::obj-1-457:head# 2021-01-15T16:59:37.473 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e49e1ff3:::obj-1-130:head# 2021-01-15T16:59:37.473 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e56c67a0:::obj-1-191:head# 2021-01-15T16:59:37.474 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e576e921:::obj-1-376:head# 2021-01-15T16:59:37.474 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e619fd26:::obj-1-395:head# 2021-01-15T16:59:37.475 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e625bdc5:::obj-1-522:head# 2021-01-15T16:59:37.475 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e68a0fa6:::obj-1-394:head# 2021-01-15T16:59:37.476 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e6cf260f:::obj-1-77:head# 2021-01-15T16:59:37.476 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e6d6255a:::obj-1-108:head# 2021-01-15T16:59:37.477 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e7a35d2f:::obj-1-116:head# 2021-01-15T16:59:37.478 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e7b07de5:::obj-1-238:head# 2021-01-15T16:59:37.478 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e8037aba:::obj-1-404:head# 2021-01-15T16:59:37.478 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e8e8b34f:::obj-1-291:head# 2021-01-15T16:59:37.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:e987cd7a:::obj-1-123:head# 2021-01-15T16:59:37.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ea534db1:::obj-1-311:head# 2021-01-15T16:59:37.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eabcc188:::obj-1-325:head# 2021-01-15T16:59:37.480 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eb086660:::obj-1-595:head# 2021-01-15T16:59:37.481 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eb2235d3:::obj-1-596:head# 2021-01-15T16:59:37.481 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eb35ae40:::obj-1-516:head# 2021-01-15T16:59:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eb360a13:::obj-1-224:head# 2021-01-15T16:59:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eb3a3ea1:::obj-1-120:head# 2021-01-15T16:59:37.483 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ebca9f6e:::obj-1-290:head# 2021-01-15T16:59:37.483 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ec81db71:::obj-1-478:head# 2021-01-15T16:59:37.484 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:edeafa4c:::obj-1-362:head# 2021-01-15T16:59:37.484 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eeb5261b:::obj-1-539:head# 2021-01-15T16:59:37.485 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:eec00eee:::obj-1-567:head# 2021-01-15T16:59:37.485 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ef075e10:::obj-1-463:head# 2021-01-15T16:59:37.486 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ef8e37d9:::obj-1-94:head# 2021-01-15T16:59:37.487 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:efad7f67:::obj-1-197:head# 2021-01-15T16:59:37.487 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:efe09ea7:::obj-1-537:head# 2021-01-15T16:59:37.488 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f0445019:::obj-1-480:head# 2021-01-15T16:59:37.488 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f08a432c:::obj-1-489:head# 2021-01-15T16:59:37.489 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f0bd9006:::obj-1-487:head# 2021-01-15T16:59:37.489 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f11a115d:::obj-1-591:head# 2021-01-15T16:59:37.490 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f232e0af:::obj-1-146:head# 2021-01-15T16:59:37.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f258db54:::obj-1-521:head# 2021-01-15T16:59:37.491 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f29cc204:::obj-1-566:head# 2021-01-15T16:59:37.492 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f2ae2451:::obj-1-66:head# 2021-01-15T16:59:37.492 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f318810a:::obj-1-160:head# 2021-01-15T16:59:37.493 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f3c35a0d:::obj-1-67:head# 2021-01-15T16:59:37.493 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f3ce9c90:::obj-1-88:head# 2021-01-15T16:59:37.494 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f3e3e81f:::obj-1-304:head# 2021-01-15T16:59:37.494 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f3f65b0a:::obj-1-367:head# 2021-01-15T16:59:37.495 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f4ea014b:::obj-1-149:head# 2021-01-15T16:59:37.495 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f52d115b:::obj-1-383:head# 2021-01-15T16:59:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f590311e:::obj-1-76:head# 2021-01-15T16:59:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f5a37734:::obj-1-51:head# 2021-01-15T16:59:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f5e2e77c:::obj-1-179:head# 2021-01-15T16:59:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f5e381bc:::obj-1-173:head# 2021-01-15T16:59:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f6c0ae0b:::obj-1-247:head# 2021-01-15T16:59:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f6db8a82:::obj-1-548:head# 2021-01-15T16:59:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f6e2dc46:::obj-1-36:head# 2021-01-15T16:59:37.499 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f76ee569:::obj-1-587:head# 2021-01-15T16:59:37.499 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f773b9ed:::obj-1-550:head# 2021-01-15T16:59:37.500 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f7e1adad:::obj-1-350:head# 2021-01-15T16:59:37.500 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f7eebf7a:::obj-1-559:head# 2021-01-15T16:59:37.501 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f877f025:::obj-1-187:head# 2021-01-15T16:59:37.501 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f992f9e9:::obj-1-402:head# 2021-01-15T16:59:37.502 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:f9fe960a:::obj-1-418:head# 2021-01-15T16:59:37.502 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fa9b882f:::obj-1-165:head# 2021-01-15T16:59:37.503 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fbd6461c:::obj-1-339:head# 2021-01-15T16:59:37.503 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fbeab28d:::obj-1-431:head# 2021-01-15T16:59:37.504 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fdea4138:::obj-1-185:head# 2021-01-15T16:59:37.504 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fdfa7c9b:::obj-1-183:head# 2021-01-15T16:59:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:fe1ba57e:::obj-1-90:head# 2021-01-15T16:59:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:feb7adea:::obj-1-560:head# 2021-01-15T16:59:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ffab306e:::obj-1-97:head# 2021-01-15T16:59:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ffdfd397:::obj-1-433:head# 2021-01-15T16:59:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:Read #2:ffe8cf04:::obj-1-303:head# 2021-01-15T16:59:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:Export successful 2021-01-15T16:59:37.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/osd-backfill-space 2 2021-01-15T16:59:37.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T16:59:37.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T16:59:37.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T16:59:37.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T16:59:37.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T16:59:37.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T16:59:37.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T16:59:37.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T16:59:37.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T16:59:37.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T16:59:37.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T16:59:37.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T16:59:37.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T16:59:37.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T16:59:37.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T16:59:37.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T16:59:37.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T16:59:37.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T16:59:37.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T16:59:37.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T16:59:37.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:59:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T16:59:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T16:59:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T16:59:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T16:59:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T16:59:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T16:59:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:59:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T16:59:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T16:59:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T16:59:37.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T16:59:37.786 INFO:tasks.workunit.client.0.smithi145.stderr:start osd.2 2021-01-15T16:59:37.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T16:59:37.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/2/whoami 2021-01-15T16:59:37.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T16:59:37.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T16:59:37.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T16:59:37.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T16:59:37.807 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:59:37.807+0000 7f46d93d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:59:37.809 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:59:37.809+0000 7f46d93d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:59:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:59:37.811+0000 7f46d93d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T16:59:38.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T16:59:38.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T16:59:38.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T16:59:38.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T16:59:38.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T16:59:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:59:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T16:59:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T16:59:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:59:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:59:38.328 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:59:38.328+0000 7f46d93d0080 -1 Falling back to public interface 2021-01-15T16:59:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T16:59:39.253 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T16:59:39.253+0000 7f46d93d0080 -1 osd.2 40 log_to_monitors {default=true} 2021-01-15T16:59:39.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T16:59:39.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T16:59:39.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T16:59:39.603 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T16:59:39.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T16:59:39.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T16:59:40.007 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 up in weight 1 up_from 43 up_thru 43 down_at 41 last_clean_interval [15,40) [v2:127.0.0.1:6818/237063,v1:127.0.0.1:6819/237063] [v2:127.0.0.1:6820/237063,v1:127.0.0.1:6821/237063] exists,up da24b4ca-22e9-4a54-afb6-b97afd81ecc4 2021-01-15T16:59:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T16:59:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T16:59:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T16:59:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:490: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2021-01-15T16:59:40.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2021-01-15T16:59:40.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=224142 2021-01-15T16:59:40.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2021-01-15T16:59:40.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2021-01-15T16:59:40.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2021-01-15T16:59:40.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T16:59:40.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 224142 2021-01-15T16:59:40.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2021-01-15T16:59:40.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2021-01-15T16:59:40.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2021-01-15T16:59:40.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T16:59:40.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 224142 2021-01-15T16:59:40.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2021-01-15T16:59:40.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2021-01-15T16:59:40.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2021-01-15T16:59:40.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:491: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --force --op remove --pgid 2.0 2021-01-15T16:59:40.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2021-01-15T16:59:40.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:59:40.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:59:40.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:59:40.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2021-01-15T16:59:40.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --force --op remove --pgid 2.0 2021-01-15T16:59:41.303 INFO:tasks.workunit.client.0.smithi145.stderr:PG '2.0' not found 2021-01-15T16:59:41.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: _objectstore_tool_nodown: return 1 2021-01-15T16:59:41.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:492: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op import --pgid 2.0 --file td/osd-backfill-space/export.out 2021-01-15T16:59:41.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2021-01-15T16:59:41.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T16:59:41.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T16:59:41.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T16:59:41.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2021-01-15T16:59:41.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op import --pgid 2.0 --file td/osd-backfill-space/export.out 2021-01-15T16:59:42.241 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={21=1},2={26=1},3={31=1}} deleted_pools ) 2021-01-15T16:59:42.241 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 2.0 2021-01-15T16:59:42.241 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:00f101c6:::obj-1-121:head# 2021-01-15T16:59:42.241 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.321 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0168051e:::obj-1-571:head# 2021-01-15T16:59:42.322 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.380 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:01b67738:::obj-1-182:head# 2021-01-15T16:59:42.380 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.413 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:040dbc59:::obj-1-23:head# 2021-01-15T16:59:42.413 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.446 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0415ce6e:::obj-1-20:head# 2021-01-15T16:59:42.447 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.497 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:058b28d7:::obj-1-572:head# 2021-01-15T16:59:42.497 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.530 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:059d2e5c:::obj-1-87:head# 2021-01-15T16:59:42.531 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.563 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:05a823ca:::obj-1-52:head# 2021-01-15T16:59:42.564 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.597 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:05b8b7fe:::obj-1-119:head# 2021-01-15T16:59:42.597 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.631 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:05c9492b:::obj-1-78:head# 2021-01-15T16:59:42.631 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.664 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:05e91e96:::obj-1-393:head# 2021-01-15T16:59:42.664 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.697 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:07bdb780:::obj-1-204:head# 2021-01-15T16:59:42.698 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.772 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:07cfbd49:::obj-1-318:head# 2021-01-15T16:59:42.772 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.814 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:084a829f:::obj-1-112:head# 2021-01-15T16:59:42.815 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.848 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:08cedd34:::obj-1-438:head# 2021-01-15T16:59:42.848 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.881 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0941af69:::obj-1-117:head# 2021-01-15T16:59:42.882 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.915 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:09607265:::obj-1-429:head# 2021-01-15T16:59:42.915 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.948 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0be86f7e:::obj-1-568:head# 2021-01-15T16:59:42.948 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:42.982 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0ca5819d:::obj-1-327:head# 2021-01-15T16:59:42.982 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.023 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0cd02051:::obj-1-208:head# 2021-01-15T16:59:43.024 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.057 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0d0264b5:::obj-1-171:head# 2021-01-15T16:59:43.058 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.091 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0d195060:::obj-1-420:head# 2021-01-15T16:59:43.091 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.124 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0d362b19:::obj-1-219:head# 2021-01-15T16:59:43.124 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.157 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0dea6f4d:::obj-1-61:head# 2021-01-15T16:59:43.158 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.191 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0ecd8390:::obj-1-132:head# 2021-01-15T16:59:43.191 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.224 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0f49e2fa:::obj-1-200:head# 2021-01-15T16:59:43.225 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.493 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0f7bb95f:::obj-1-268:head# 2021-01-15T16:59:43.493 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.592 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0fabf252:::obj-1-441:head# 2021-01-15T16:59:43.592 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.617 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:0fb1697a:::obj-1-443:head# 2021-01-15T16:59:43.617 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.642 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:105d3efe:::obj-1-490:head# 2021-01-15T16:59:43.642 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.667 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:10f52c01:::obj-1-449:head# 2021-01-15T16:59:43.667 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.692 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:11965425:::obj-1-335:head# 2021-01-15T16:59:43.692 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.717 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:11c9f5f8:::obj-1-565:head# 2021-01-15T16:59:43.717 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.743 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:11cfcfb6:::obj-1-588:head# 2021-01-15T16:59:43.743 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.767 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:11d46f0a:::obj-1-387:head# 2021-01-15T16:59:43.768 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.801 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:11ebb0d6:::obj-1-214:head# 2021-01-15T16:59:43.801 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.826 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1295e402:::obj-1-178:head# 2021-01-15T16:59:43.827 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.860 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:12c996c1:::obj-1-517:head# 2021-01-15T16:59:43.860 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.893 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:133dec15:::obj-1-475:head# 2021-01-15T16:59:43.893 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.926 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1475c981:::obj-1-505:head# 2021-01-15T16:59:43.927 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.960 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:148e92a4:::obj-1-509:head# 2021-01-15T16:59:43.960 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:43.994 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1490b121:::obj-1-172:head# 2021-01-15T16:59:43.994 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.035 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:14ce082c:::obj-1-99:head# 2021-01-15T16:59:44.036 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.119 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1552ad99:::obj-1-41:head# 2021-01-15T16:59:44.119 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.194 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:15544270:::obj-1-271:head# 2021-01-15T16:59:44.195 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.227 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1589d957:::obj-1-86:head# 2021-01-15T16:59:44.228 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.261 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:16db3947:::obj-1-477:head# 2021-01-15T16:59:44.261 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.294 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:16dc98bd:::obj-1-375:head# 2021-01-15T16:59:44.295 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.370 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:17e54db3:::obj-1-196:head# 2021-01-15T16:59:44.370 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.420 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:17f2ad3f:::obj-1-10:head# 2021-01-15T16:59:44.421 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.454 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:18075e41:::obj-1-55:head# 2021-01-15T16:59:44.454 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.488 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:18086253:::obj-1-71:head# 2021-01-15T16:59:44.488 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.522 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:191cf8d8:::obj-1-372:head# 2021-01-15T16:59:44.523 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.554 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:19311f70:::obj-1-454:head# 2021-01-15T16:59:44.555 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.587 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:198318e4:::obj-1-464:head# 2021-01-15T16:59:44.588 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.621 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1999d806:::obj-1-163:head# 2021-01-15T16:59:44.621 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.655 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1a92cab7:::obj-1-412:head# 2021-01-15T16:59:44.655 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.688 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1b16c0f6:::obj-1-210:head# 2021-01-15T16:59:44.689 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.722 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1b53ed46:::obj-1-577:head# 2021-01-15T16:59:44.722 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.755 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1b5c5a8d:::obj-1-556:head# 2021-01-15T16:59:44.756 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.822 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1c12c05e:::obj-1-79:head# 2021-01-15T16:59:44.822 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.855 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1c571ba5:::obj-1-262:head# 2021-01-15T16:59:44.856 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.888 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1c5f42b7:::obj-1-92:head# 2021-01-15T16:59:44.889 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.922 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1c6ad881:::obj-1-312:head# 2021-01-15T16:59:44.922 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:44.956 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1c89877b:::obj-1-142:head# 2021-01-15T16:59:44.956 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.023 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1e4168ca:::obj-1-282:head# 2021-01-15T16:59:45.023 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.048 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1e424445:::obj-1-484:head# 2021-01-15T16:59:45.048 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.073 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1e467959:::obj-1-22:head# 2021-01-15T16:59:45.073 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.098 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1eaf7e7c:::obj-1-392:head# 2021-01-15T16:59:45.098 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.123 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1f06c86d:::obj-1-307:head# 2021-01-15T16:59:45.123 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.148 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1f2ee1f8:::obj-1-306:head# 2021-01-15T16:59:45.148 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.173 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:1f4ef040:::obj-1-349:head# 2021-01-15T16:59:45.173 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.199 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2039481f:::obj-1-545:head# 2021-01-15T16:59:45.199 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.232 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:22233df0:::obj-1-543:head# 2021-01-15T16:59:45.232 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.257 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2225184e:::obj-1-266:head# 2021-01-15T16:59:45.258 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.282 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:234766bb:::obj-1-281:head# 2021-01-15T16:59:45.282 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.307 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2372e022:::obj-1-205:head# 2021-01-15T16:59:45.308 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.332 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2379547b:::obj-1-113:head# 2021-01-15T16:59:45.333 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.358 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:23ae14bf:::obj-1-531:head# 2021-01-15T16:59:45.358 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.383 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:23bc5f90:::obj-1-452:head# 2021-01-15T16:59:45.383 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.439 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:23fbef3d:::obj-1-82:head# 2021-01-15T16:59:45.439 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.475 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:24106c2b:::obj-1-232:head# 2021-01-15T16:59:45.475 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.500 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2529b4e4:::obj-1-75:head# 2021-01-15T16:59:45.500 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.525 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:25a24ae7:::obj-1-459:head# 2021-01-15T16:59:45.525 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.550 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:26627aa9:::obj-1-7:head# 2021-01-15T16:59:45.550 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.575 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:26b77221:::obj-1-227:head# 2021-01-15T16:59:45.575 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.600 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:27450826:::obj-1-344:head# 2021-01-15T16:59:45.601 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.625 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:278905af:::obj-1-592:head# 2021-01-15T16:59:45.625 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.650 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:27ad0d2d:::obj-1-136:head# 2021-01-15T16:59:45.651 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.676 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:28a8c213:::obj-1-118:head# 2021-01-15T16:59:45.676 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.701 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:28c0c009:::obj-1-74:head# 2021-01-15T16:59:45.701 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.734 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2a214708:::obj-1-260:head# 2021-01-15T16:59:45.734 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.768 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2a67e195:::obj-1-474:head# 2021-01-15T16:59:45.768 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.801 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2aa852ff:::obj-1-5:head# 2021-01-15T16:59:45.801 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.835 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2ba1fb52:::obj-1-563:head# 2021-01-15T16:59:45.835 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.868 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2bae8cb9:::obj-1-320:head# 2021-01-15T16:59:45.868 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.901 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2c569d02:::obj-1-139:head# 2021-01-15T16:59:45.902 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.935 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2c71d691:::obj-1-201:head# 2021-01-15T16:59:45.935 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:45.968 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2c88e92c:::obj-1-389:head# 2021-01-15T16:59:45.969 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.003 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2d659109:::obj-1-580:head# 2021-01-15T16:59:46.003 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.110 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2d74ea60:::obj-1-230:head# 2021-01-15T16:59:46.111 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.169 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2de96edf:::obj-1-233:head# 2021-01-15T16:59:46.169 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.227 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2e43da40:::obj-1-382:head# 2021-01-15T16:59:46.228 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.261 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2e530c9f:::obj-1-358:head# 2021-01-15T16:59:46.261 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.303 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:2ed02239:::obj-1-81:head# 2021-01-15T16:59:46.304 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.345 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:30264a0c:::obj-1-373:head# 2021-01-15T16:59:46.345 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.403 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:302ca5e7:::obj-1-206:head# 2021-01-15T16:59:46.404 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.437 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:309878e8:::obj-1-508:head# 2021-01-15T16:59:46.437 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.470 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:309a0ac3:::obj-1-188:head# 2021-01-15T16:59:46.471 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.504 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:30f030e9:::obj-1-53:head# 2021-01-15T16:59:46.504 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.537 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:31242a73:::obj-1-528:head# 2021-01-15T16:59:46.537 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.571 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:31463825:::obj-1-34:head# 2021-01-15T16:59:46.571 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.604 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3166dc86:::obj-1-355:head# 2021-01-15T16:59:46.604 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.638 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:31baa2fa:::obj-1-252:head# 2021-01-15T16:59:46.638 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.671 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:31d8935e:::obj-1-193:head# 2021-01-15T16:59:46.671 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.780 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:32670b14:::obj-1-305:head# 2021-01-15T16:59:46.780 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.830 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:33da02fa:::obj-1-114:head# 2021-01-15T16:59:46.830 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.863 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:34373133:::obj-1-446:head# 2021-01-15T16:59:46.864 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.897 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:347b7083:::obj-1-89:head# 2021-01-15T16:59:46.897 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.930 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:349125c1:::obj-1-148:head# 2021-01-15T16:59:46.931 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.964 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:34b80eee:::obj-1-138:head# 2021-01-15T16:59:46.964 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:46.997 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:34db48f8:::obj-1-49:head# 2021-01-15T16:59:46.998 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.031 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3669ff0c:::obj-1-313:head# 2021-01-15T16:59:47.031 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.064 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:36c1b610:::obj-1-249:head# 2021-01-15T16:59:47.064 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.098 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:374c250b:::obj-1-324:head# 2021-01-15T16:59:47.098 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.131 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3761a149:::obj-1-342:head# 2021-01-15T16:59:47.132 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.165 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:379ff95a:::obj-1-341:head# 2021-01-15T16:59:47.165 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.198 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:37dc15a6:::obj-1-26:head# 2021-01-15T16:59:47.198 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.232 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:38130e33:::obj-1-552:head# 2021-01-15T16:59:47.232 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.273 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:39c44880:::obj-1-538:head# 2021-01-15T16:59:47.274 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.298 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3a08dec9:::obj-1-368:head# 2021-01-15T16:59:47.299 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.349 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3a2b46a6:::obj-1-407:head# 2021-01-15T16:59:47.350 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.382 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3a7dfe0a:::obj-1-285:head# 2021-01-15T16:59:47.383 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.407 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3aebb144:::obj-1-248:head# 2021-01-15T16:59:47.408 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.432 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3b08b36f:::obj-1-310:head# 2021-01-15T16:59:47.433 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.458 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3b28ffa3:::obj-1-295:head# 2021-01-15T16:59:47.458 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.482 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3ba79bd1:::obj-1-444:head# 2021-01-15T16:59:47.483 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.508 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3c05df8d:::obj-1-319:head# 2021-01-15T16:59:47.508 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.533 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3c57e6a9:::obj-1-436:head# 2021-01-15T16:59:47.533 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.558 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3c73d8ee:::obj-1-298:head# 2021-01-15T16:59:47.558 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.583 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3cb6ae2a:::obj-1-400:head# 2021-01-15T16:59:47.583 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.608 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3cb71491:::obj-1-328:head# 2021-01-15T16:59:47.609 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.633 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3cb842fe:::obj-1-541:head# 2021-01-15T16:59:47.634 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.658 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3cf6835e:::obj-1-496:head# 2021-01-15T16:59:47.659 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.684 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3cf78507:::obj-1-573:head# 2021-01-15T16:59:47.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.709 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3d0dab84:::obj-1-554:head# 2021-01-15T16:59:47.709 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.734 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3d1a5f9d:::obj-1-323:head# 2021-01-15T16:59:47.734 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.759 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3da5f3a2:::obj-1-414:head# 2021-01-15T16:59:47.759 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.784 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3dbb65f0:::obj-1-440:head# 2021-01-15T16:59:47.785 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.809 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3dc94618:::obj-1-223:head# 2021-01-15T16:59:47.810 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.834 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3df3d5ab:::obj-1-25:head# 2021-01-15T16:59:47.835 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.860 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3e167988:::obj-1-336:head# 2021-01-15T16:59:47.860 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.893 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3e4ef3bd:::obj-1-104:head# 2021-01-15T16:59:47.893 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.927 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3e9e1efd:::obj-1-357:head# 2021-01-15T16:59:47.927 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:47.960 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3ebbe3a7:::obj-1-42:head# 2021-01-15T16:59:47.960 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.052 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3ec80373:::obj-1-403:head# 2021-01-15T16:59:48.052 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.111 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3ed086e4:::obj-1-13:head# 2021-01-15T16:59:48.111 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.161 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:3fe8e157:::obj-1-164:head# 2021-01-15T16:59:48.161 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.194 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:401ddfd4:::obj-1-379:head# 2021-01-15T16:59:48.194 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.227 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4092f6cd:::obj-1-430:head# 2021-01-15T16:59:48.228 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.261 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:414bd7cc:::obj-1-562:head# 2021-01-15T16:59:48.261 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.303 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:422bd9d0:::obj-1-287:head# 2021-01-15T16:59:48.303 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.361 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4255002d:::obj-1-578:head# 2021-01-15T16:59:48.362 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.395 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4270725c:::obj-1-109:head# 2021-01-15T16:59:48.395 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.428 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:42836669:::obj-1-31:head# 2021-01-15T16:59:48.429 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.470 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:428d93ef:::obj-1-32:head# 2021-01-15T16:59:48.470 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.512 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4327ef5b:::obj-1-453:head# 2021-01-15T16:59:48.512 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.546 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:440eb07b:::obj-1-236:head# 2021-01-15T16:59:48.546 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.596 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:442a028a:::obj-1-365:head# 2021-01-15T16:59:48.596 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.689 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:449ca54a:::obj-1-161:head# 2021-01-15T16:59:48.689 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.738 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:45128c3b:::obj-1-343:head# 2021-01-15T16:59:48.739 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.788 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:45801bf6:::obj-1-264:head# 2021-01-15T16:59:48.789 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.855 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:45896f61:::obj-1-48:head# 2021-01-15T16:59:48.856 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.914 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:459b7f1c:::obj-1-190:head# 2021-01-15T16:59:48.914 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:48.965 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:466af96a:::obj-1-486:head# 2021-01-15T16:59:48.965 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.014 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:47387ce4:::obj-1-575:head# 2021-01-15T16:59:49.015 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.065 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:47c94cc8:::obj-1-564:head# 2021-01-15T16:59:49.065 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.115 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:47d7f8ed:::obj-1-544:head# 2021-01-15T16:59:49.115 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.165 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:47fc0ec3:::obj-1-558:head# 2021-01-15T16:59:49.166 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.215 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:48ba42d8:::obj-1-156:head# 2021-01-15T16:59:49.216 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.286 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:494e269e:::obj-1-600:head# 2021-01-15T16:59:49.287 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.324 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:49646095:::obj-1-73:head# 2021-01-15T16:59:49.324 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.357 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:49eb65ab:::obj-1-374:head# 2021-01-15T16:59:49.358 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.391 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4a0075a4:::obj-1-432:head# 2021-01-15T16:59:49.391 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.424 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4af1ffc2:::obj-1-72:head# 2021-01-15T16:59:49.425 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.458 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4bc1c46f:::obj-1-346:head# 2021-01-15T16:59:49.458 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.491 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4bd55e68:::obj-1-93:head# 2021-01-15T16:59:49.492 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.525 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4c6d2760:::obj-1-198:head# 2021-01-15T16:59:49.526 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.558 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4cb9be3b:::obj-1-451:head# 2021-01-15T16:59:49.559 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.592 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4d260a64:::obj-1-514:head# 2021-01-15T16:59:49.593 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.617 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4d7d6284:::obj-1-583:head# 2021-01-15T16:59:49.618 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.651 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4e79ac02:::obj-1-437:head# 2021-01-15T16:59:49.651 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.684 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4e980099:::obj-1-507:head# 2021-01-15T16:59:49.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.717 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4ea8f99c:::obj-1-211:head# 2021-01-15T16:59:49.718 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.760 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4eabdc2a:::obj-1-590:head# 2021-01-15T16:59:49.760 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.793 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4f97fd73:::obj-1-243:head# 2021-01-15T16:59:49.793 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.826 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4fe00224:::obj-1-229:head# 2021-01-15T16:59:49.826 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.859 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4fead9ab:::obj-1-476:head# 2021-01-15T16:59:49.860 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.926 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:4ffddfd7:::obj-1-462:head# 2021-01-15T16:59:49.926 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:49.968 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5085c04d:::obj-1-100:head# 2021-01-15T16:59:49.968 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.001 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:50a8b465:::obj-1-574:head# 2021-01-15T16:59:50.002 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.043 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:518a151f:::obj-1-189:head# 2021-01-15T16:59:50.044 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.093 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:519afc95:::obj-1-314:head# 2021-01-15T16:59:50.094 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.127 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:52199781:::obj-1-202:head# 2021-01-15T16:59:50.127 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.161 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:52313062:::obj-1-337:head# 2021-01-15T16:59:50.161 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.194 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:526a5196:::obj-1-225:head# 2021-01-15T16:59:50.195 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.228 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:53128d32:::obj-1-483:head# 2021-01-15T16:59:50.228 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.270 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:53c2bb21:::obj-1-532:head# 2021-01-15T16:59:50.270 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.328 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:53fa4f3c:::obj-1-423:head# 2021-01-15T16:59:50.328 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.421 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:541c1d30:::obj-1-143:head# 2021-01-15T16:59:50.421 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.471 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:548d20c3:::obj-1-352:head# 2021-01-15T16:59:50.471 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.521 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:54e4c6f9:::obj-1-101:head# 2021-01-15T16:59:50.522 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.604 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:55016e06:::obj-1-276:head# 2021-01-15T16:59:50.605 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.655 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5510c784:::obj-1-491:head# 2021-01-15T16:59:50.656 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.705 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:551275ef:::obj-1-239:head# 2021-01-15T16:59:50.705 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.756 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:556df001:::obj-1-46:head# 2021-01-15T16:59:50.756 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.805 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:558fb5b1:::obj-1-536:head# 2021-01-15T16:59:50.806 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.856 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:56390d98:::obj-1-64:head# 2021-01-15T16:59:50.856 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.906 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:564f962a:::obj-1-125:head# 2021-01-15T16:59:50.906 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:50.958 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:56936e94:::obj-1-251:head# 2021-01-15T16:59:50.958 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.007 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:57585d3d:::obj-1-294:head# 2021-01-15T16:59:51.007 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.065 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5764cf7b:::obj-1-479:head# 2021-01-15T16:59:51.065 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.098 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5782341a:::obj-1-218:head# 2021-01-15T16:59:51.099 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.132 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:57d07a1a:::obj-1-317:head# 2021-01-15T16:59:51.132 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.196 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:58700205:::obj-1-176:head# 2021-01-15T16:59:51.197 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.241 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:58aee718:::obj-1-434:head# 2021-01-15T16:59:51.241 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.274 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:594c9d52:::obj-1-458:head# 2021-01-15T16:59:51.274 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.308 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5a70b555:::obj-1-279:head# 2021-01-15T16:59:51.308 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.341 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5b02568b:::obj-1-586:head# 2021-01-15T16:59:51.341 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.393 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5b69cdcf:::obj-1-391:head# 2021-01-15T16:59:51.393 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.442 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5cf151b5:::obj-1-510:head# 2021-01-15T16:59:51.443 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.492 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5d7fd07f:::obj-1-152:head# 2021-01-15T16:59:51.492 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.542 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5de288d7:::obj-1-235:head# 2021-01-15T16:59:51.543 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.593 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5ec497cd:::obj-1-44:head# 2021-01-15T16:59:51.593 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.643 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5fb3a09f:::obj-1-518:head# 2021-01-15T16:59:51.643 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.676 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:5ff3c2ef:::obj-1-470:head# 2021-01-15T16:59:51.677 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.701 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:601efcb1:::obj-1-549:head# 2021-01-15T16:59:51.701 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.735 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:60831338:::obj-1-263:head# 2021-01-15T16:59:51.735 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.852 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6092777b:::obj-1-145:head# 2021-01-15T16:59:51.853 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.927 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:60b17206:::obj-1-411:head# 2021-01-15T16:59:51.927 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:51.977 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:610c957c:::obj-1-278:head# 2021-01-15T16:59:51.977 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.069 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:613b5ef4:::obj-1-504:head# 2021-01-15T16:59:52.070 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.136 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:61ed7d60:::obj-1-274:head# 2021-01-15T16:59:52.137 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.186 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:621b6751:::obj-1-212:head# 2021-01-15T16:59:52.187 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.237 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:62b0aba9:::obj-1-253:head# 2021-01-15T16:59:52.237 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.295 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:64017240:::obj-1-102:head# 2021-01-15T16:59:52.296 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.387 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:645f4f49:::obj-1-332:head# 2021-01-15T16:59:52.388 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.484 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:65db82b9:::obj-1-334:head# 2021-01-15T16:59:52.484 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.524 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:667422e5:::obj-1-465:head# 2021-01-15T16:59:52.524 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.556 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:66c50bc2:::obj-1-322:head# 2021-01-15T16:59:52.556 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.589 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:681a4577:::obj-1-511:head# 2021-01-15T16:59:52.590 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.623 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6874e007:::obj-1-398:head# 2021-01-15T16:59:52.623 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.656 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:693746c1:::obj-1-582:head# 2021-01-15T16:59:52.656 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.689 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:69638b0c:::obj-1-8:head# 2021-01-15T16:59:52.690 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.723 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6967de6e:::obj-1-331:head# 2021-01-15T16:59:52.723 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.748 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:69da97aa:::obj-1-340:head# 2021-01-15T16:59:52.748 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.784 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6b3a50a6:::obj-1-144:head# 2021-01-15T16:59:52.784 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.831 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6be4ef26:::obj-1-579:head# 2021-01-15T16:59:52.832 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.881 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6d735549:::obj-1-599:head# 2021-01-15T16:59:52.882 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.915 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6e22c25e:::obj-1-9:head# 2021-01-15T16:59:52.915 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.948 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6e2ced4e:::obj-1-370:head# 2021-01-15T16:59:52.948 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:52.981 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6e961169:::obj-1-428:head# 2021-01-15T16:59:52.982 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.015 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6efcbbee:::obj-1-168:head# 2021-01-15T16:59:53.015 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.048 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6f20741c:::obj-1-33:head# 2021-01-15T16:59:53.049 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.081 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6fe4c885:::obj-1-273:head# 2021-01-15T16:59:53.082 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.157 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:6fffdea2:::obj-1-425:head# 2021-01-15T16:59:53.157 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.190 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:70716852:::obj-1-593:head# 2021-01-15T16:59:53.190 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.223 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:70919606:::obj-1-515:head# 2021-01-15T16:59:53.224 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.257 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:70c39d76:::obj-1-467:head# 2021-01-15T16:59:53.257 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.290 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:70f57ad3:::obj-1-361:head# 2021-01-15T16:59:53.290 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.323 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7177c7b5:::obj-1-503:head# 2021-01-15T16:59:53.324 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.357 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:719b69bd:::obj-1-497:head# 2021-01-15T16:59:53.357 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.407 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:724f9347:::obj-1-6:head# 2021-01-15T16:59:53.407 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.440 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:726d9f6b:::obj-1-111:head# 2021-01-15T16:59:53.440 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.473 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:72a91baf:::obj-1-439:head# 2021-01-15T16:59:53.473 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.507 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:72aa43ee:::obj-1-534:head# 2021-01-15T16:59:53.507 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.540 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:72c40e11:::obj-1-288:head# 2021-01-15T16:59:53.540 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.574 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:74196d45:::obj-1-246:head# 2021-01-15T16:59:53.574 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.606 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:743f42b1:::obj-1-326:head# 2021-01-15T16:59:53.607 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.640 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7523b538:::obj-1-494:head# 2021-01-15T16:59:53.640 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.673 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:75310ed6:::obj-1-364:head# 2021-01-15T16:59:53.673 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.698 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7568e0bc:::obj-1-585:head# 2021-01-15T16:59:53.698 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.765 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:760219a3:::obj-1-353:head# 2021-01-15T16:59:53.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.790 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:763338c9:::obj-1-555:head# 2021-01-15T16:59:53.790 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.815 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:77bc2d65:::obj-1-386:head# 2021-01-15T16:59:53.816 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.848 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:77ed0c7a:::obj-1-492:head# 2021-01-15T16:59:53.849 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.873 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:785a4052:::obj-1-456:head# 2021-01-15T16:59:53.874 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.916 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:796799c3:::obj-1-195:head# 2021-01-15T16:59:53.917 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.948 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:79caf0c1:::obj-1-445:head# 2021-01-15T16:59:53.949 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:53.973 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7af5b3e4:::obj-1-217:head# 2021-01-15T16:59:53.973 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.007 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7b3a9063:::obj-1-169:head# 2021-01-15T16:59:54.007 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.069 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7b509a92:::obj-1-561:head# 2021-01-15T16:59:54.069 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.140 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7bc646b1:::obj-1-415:head# 2021-01-15T16:59:54.140 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.173 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7c2f9188:::obj-1-482:head# 2021-01-15T16:59:54.173 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.207 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7d0da936:::obj-1-11:head# 2021-01-15T16:59:54.207 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.240 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7d25ca2d:::obj-1-242:head# 2021-01-15T16:59:54.241 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.282 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7d64f81e:::obj-1-473:head# 2021-01-15T16:59:54.282 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.345 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7ef2ea1b:::obj-1-417:head# 2021-01-15T16:59:54.345 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.465 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7f52b238:::obj-1-308:head# 2021-01-15T16:59:54.466 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.507 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:7f7a313c:::obj-1-551:head# 2021-01-15T16:59:54.545 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.546 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:806827a7:::obj-1-472:head# 2021-01-15T16:59:54.546 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.582 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8112c3ab:::obj-1-45:head# 2021-01-15T16:59:54.582 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.623 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:81365b55:::obj-1-315:head# 2021-01-15T16:59:54.624 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.657 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:818ffdc3:::obj-1-345:head# 2021-01-15T16:59:54.657 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.698 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:81babab8:::obj-1-535:head# 2021-01-15T16:59:54.699 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.732 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8288f652:::obj-1-347:head# 2021-01-15T16:59:54.732 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.765 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8312eadd:::obj-1-14:head# 2021-01-15T16:59:54.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.798 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:83b89bc7:::obj-1-137:head# 2021-01-15T16:59:54.799 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.832 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:84a12321:::obj-1-134:head# 2021-01-15T16:59:54.833 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.865 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:84b26d62:::obj-1-408:head# 2021-01-15T16:59:54.865 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.899 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:84bbb26e:::obj-1-43:head# 2021-01-15T16:59:54.899 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.932 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:86fc0014:::obj-1-222:head# 2021-01-15T16:59:54.932 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:54.974 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:87488bf4:::obj-1-174:head# 2021-01-15T16:59:54.974 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.049 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8754389a:::obj-1-65:head# 2021-01-15T16:59:55.049 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.082 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:87a6ad67:::obj-1-366:head# 2021-01-15T16:59:55.083 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.115 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:87d32b83:::obj-1-513:head# 2021-01-15T16:59:55.116 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.149 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8813a346:::obj-1-540:head# 2021-01-15T16:59:55.149 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.183 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:88146951:::obj-1-216:head# 2021-01-15T16:59:55.183 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.216 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8820800d:::obj-1-83:head# 2021-01-15T16:59:55.216 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.249 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:88319745:::obj-1-98:head# 2021-01-15T16:59:55.249 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.282 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:88c969b8:::obj-1-54:head# 2021-01-15T16:59:55.283 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.324 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:895a2d53:::obj-1-150:head# 2021-01-15T16:59:55.324 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.357 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8ae006a0:::obj-1-241:head# 2021-01-15T16:59:55.358 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.391 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8bd44e64:::obj-1-300:head# 2021-01-15T16:59:55.391 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.426 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8beb257f:::obj-1-3:head# 2021-01-15T16:59:55.426 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.457 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8cd8c743:::obj-1-527:head# 2021-01-15T16:59:55.457 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.491 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8daa99c7:::obj-1-237:head# 2021-01-15T16:59:55.491 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.524 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8f5bccbf:::obj-1-59:head# 2021-01-15T16:59:55.524 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.549 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8f6d9945:::obj-1-151:head# 2021-01-15T16:59:55.549 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.574 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8f7c568c:::obj-1-261:head# 2021-01-15T16:59:55.574 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.599 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:8fbc59a9:::obj-1-589:head# 2021-01-15T16:59:55.599 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.624 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:90225d94:::obj-1-194:head# 2021-01-15T16:59:55.624 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.691 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:904c50d3:::obj-1-63:head# 2021-01-15T16:59:55.692 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.716 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:90e1bd79:::obj-1-450:head# 2021-01-15T16:59:55.716 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.741 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9274b06d:::obj-1-460:head# 2021-01-15T16:59:55.741 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.765 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:92af41f5:::obj-1-409:head# 2021-01-15T16:59:55.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.791 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:935a08d6:::obj-1-192:head# 2021-01-15T16:59:55.791 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.817 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:935f8a13:::obj-1-24:head# 2021-01-15T16:59:55.817 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.841 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:95230148:::obj-1-280:head# 2021-01-15T16:59:55.841 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.866 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:95342eb0:::obj-1-390:head# 2021-01-15T16:59:55.866 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.891 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9659d09c:::obj-1-292:head# 2021-01-15T16:59:55.891 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.916 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:970e8cb4:::obj-1-18:head# 2021-01-15T16:59:55.916 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:97d21cf1:::obj-1-80:head# 2021-01-15T16:59:55.941 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.966 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:982bdb12:::obj-1-302:head# 2021-01-15T16:59:55.967 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:55.991 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:985dfd75:::obj-1-91:head# 2021-01-15T16:59:55.991 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.016 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:986d245a:::obj-1-154:head# 2021-01-15T16:59:56.017 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.094 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:98a03e99:::obj-1-529:head# 2021-01-15T16:59:56.094 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.124 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:98b7103e:::obj-1-275:head# 2021-01-15T16:59:56.125 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.149 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:990f0ecf:::obj-1-427:head# 2021-01-15T16:59:56.150 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.183 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:996c1f26:::obj-1-96:head# 2021-01-15T16:59:56.183 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.208 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:99bd2018:::obj-1-17:head# 2021-01-15T16:59:56.208 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.250 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9a7434c9:::obj-1-469:head# 2021-01-15T16:59:56.250 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.283 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9a8e36cc:::obj-1-598:head# 2021-01-15T16:59:56.283 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.386 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9abcc369:::obj-1-442:head# 2021-01-15T16:59:56.387 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.441 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9b0ea06e:::obj-1-231:head# 2021-01-15T16:59:56.441 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.474 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9b34e80d:::obj-1-378:head# 2021-01-15T16:59:56.474 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.507 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9b55a2a9:::obj-1-60:head# 2021-01-15T16:59:56.508 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.541 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9be3f2da:::obj-1-597:head# 2021-01-15T16:59:56.541 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.583 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9bf62c88:::obj-1-447:head# 2021-01-15T16:59:56.583 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.616 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9c31857b:::obj-1-272:head# 2021-01-15T16:59:56.617 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.650 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9c533e91:::obj-1-19:head# 2021-01-15T16:59:56.651 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.683 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9ca70688:::obj-1-570:head# 2021-01-15T16:59:56.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.716 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9ccce337:::obj-1-147:head# 2021-01-15T16:59:56.717 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.750 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9d4b2dfc:::obj-1-267:head# 2021-01-15T16:59:56.750 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.800 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9d4df969:::obj-1-286:head# 2021-01-15T16:59:56.800 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.888 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9d714651:::obj-1-495:head# 2021-01-15T16:59:56.889 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:56.925 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9da65518:::obj-1-498:head# 2021-01-15T16:59:56.925 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.008 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9e94f27b:::obj-1-209:head# 2021-01-15T16:59:57.009 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.041 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9f39e910:::obj-1-354:head# 2021-01-15T16:59:57.042 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.075 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9f8f9ef4:::obj-1-226:head# 2021-01-15T16:59:57.075 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.108 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:9fcd3f36:::obj-1-47:head# 2021-01-15T16:59:57.109 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.142 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a0326d7d:::obj-1-338:head# 2021-01-15T16:59:57.142 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.175 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a05e7475:::obj-1-62:head# 2021-01-15T16:59:57.176 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.209 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a0955052:::obj-1-533:head# 2021-01-15T16:59:57.209 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.243 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a0aa6848:::obj-1-38:head# 2021-01-15T16:59:57.243 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.275 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a0fdc285:::obj-1-133:head# 2021-01-15T16:59:57.276 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.308 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a10290b8:::obj-1-245:head# 2021-01-15T16:59:57.309 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.358 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a118a8a8:::obj-1-500:head# 2021-01-15T16:59:57.359 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.392 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a11a4079:::obj-1-199:head# 2021-01-15T16:59:57.392 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.425 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a1b3ec36:::obj-1-15:head# 2021-01-15T16:59:57.425 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.459 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a1f5079b:::obj-1-468:head# 2021-01-15T16:59:57.460 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.492 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a1ff04b5:::obj-1-167:head# 2021-01-15T16:59:57.492 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.525 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a21cc437:::obj-1-128:head# 2021-01-15T16:59:57.526 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.559 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a2966c77:::obj-1-140:head# 2021-01-15T16:59:57.559 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.642 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a321e537:::obj-1-184:head# 2021-01-15T16:59:57.643 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.684 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a33418d8:::obj-1-488:head# 2021-01-15T16:59:57.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.717 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a38b1c1a:::obj-1-584:head# 2021-01-15T16:59:57.717 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.750 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a3f83221:::obj-1-158:head# 2021-01-15T16:59:57.751 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.784 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a46e33eb:::obj-1-106:head# 2021-01-15T16:59:57.784 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.817 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a491460d:::obj-1-421:head# 2021-01-15T16:59:57.817 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.850 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a4dd813b:::obj-1-530:head# 2021-01-15T16:59:57.851 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.883 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a4e0d96d:::obj-1-351:head# 2021-01-15T16:59:57.884 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.917 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a52657b4:::obj-1-557:head# 2021-01-15T16:59:57.917 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.950 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a5b3bca6:::obj-1-277:head# 2021-01-15T16:59:57.951 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:57.984 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a5f5a9fb:::obj-1-170:head# 2021-01-15T16:59:57.984 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.017 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a6aeed57:::obj-1-141:head# 2021-01-15T16:59:58.017 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.093 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a6dda70f:::obj-1-471:head# 2021-01-15T16:59:58.093 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.125 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a71186c0:::obj-1-406:head# 2021-01-15T16:59:58.125 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.159 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a764ad8c:::obj-1-221:head# 2021-01-15T16:59:58.159 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.250 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a7b63473:::obj-1-131:head# 2021-01-15T16:59:58.251 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.292 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a88042fa:::obj-1-296:head# 2021-01-15T16:59:58.292 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.352 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:a8805fe0:::obj-1-213:head# 2021-01-15T16:59:58.352 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.417 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:aa4f12b1:::obj-1-270:head# 2021-01-15T16:59:58.418 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.442 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ab27f701:::obj-1-399:head# 2021-01-15T16:59:58.443 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.467 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ab8027c1:::obj-1-360:head# 2021-01-15T16:59:58.467 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.492 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ac3ef8ab:::obj-1-371:head# 2021-01-15T16:59:58.492 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.518 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ac4db76d:::obj-1-405:head# 2021-01-15T16:59:58.518 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.542 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ad53cb29:::obj-1-256:head# 2021-01-15T16:59:58.543 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.568 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ad704c16:::obj-1-485:head# 2021-01-15T16:59:58.569 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.592 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:adec4920:::obj-1-107:head# 2021-01-15T16:59:58.592 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.617 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ae3a94b3:::obj-1-95:head# 2021-01-15T16:59:58.617 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.642 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ae92616e:::obj-1-448:head# 2021-01-15T16:59:58.642 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.667 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ae975168:::obj-1-426:head# 2021-01-15T16:59:58.667 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.696 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:aee8fd62:::obj-1-27:head# 2021-01-15T16:59:58.696 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.717 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:af38aa87:::obj-1-50:head# 2021-01-15T16:59:58.718 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.750 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b0327eb3:::obj-1-380:head# 2021-01-15T16:59:58.751 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.784 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b03cd166:::obj-1-203:head# 2021-01-15T16:59:58.784 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.817 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b06f06e0:::obj-1-126:head# 2021-01-15T16:59:58.818 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.892 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b0972538:::obj-1-506:head# 2021-01-15T16:59:58.892 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.925 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b0cfb86a:::obj-1-359:head# 2021-01-15T16:59:58.926 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.959 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b10adb1c:::obj-1-333:head# 2021-01-15T16:59:58.960 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:58.993 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b20b9004:::obj-1-401:head# 2021-01-15T16:59:58.993 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.026 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b2113757:::obj-1-4:head# 2021-01-15T16:59:59.026 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.059 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b22441e9:::obj-1-29:head# 2021-01-15T16:59:59.059 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.092 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b2472384:::obj-1-207:head# 2021-01-15T16:59:59.093 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.142 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b2604227:::obj-1-502:head# 2021-01-15T16:59:59.143 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.175 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b3277f89:::obj-1-175:head# 2021-01-15T16:59:59.176 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.209 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b3d09e0d:::obj-1-258:head# 2021-01-15T16:59:59.209 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.242 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b3e9969a:::obj-1-499:head# 2021-01-15T16:59:59.243 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.277 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b4892e7d:::obj-1-155:head# 2021-01-15T16:59:59.277 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.309 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b4b76084:::obj-1-581:head# 2021-01-15T16:59:59.309 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.342 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b6e47ae5:::obj-1-215:head# 2021-01-15T16:59:59.343 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.376 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b75af5ff:::obj-1-228:head# 2021-01-15T16:59:59.376 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.409 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b8e2e20f:::obj-1-220:head# 2021-01-15T16:59:59.409 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.442 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:b8e80c4e:::obj-1-576:head# 2021-01-15T16:59:59.443 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.476 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ba502aee:::obj-1-255:head# 2021-01-15T16:59:59.476 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.559 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ba9c3e6e:::obj-1-57:head# 2021-01-15T16:59:59.559 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.592 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bb068062:::obj-1-461:head# 2021-01-15T16:59:59.593 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.626 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bb16c115:::obj-1-157:head# 2021-01-15T16:59:59.627 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.659 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bb447914:::obj-1-481:head# 2021-01-15T16:59:59.659 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.692 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bb6c8e0f:::obj-1-348:head# 2021-01-15T16:59:59.693 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.726 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bb77d6d0:::obj-1-301:head# 2021-01-15T16:59:59.726 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.759 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bc51c5f6:::obj-1-422:head# 2021-01-15T16:59:59.759 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.792 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bc838d5c:::obj-1-35:head# 2021-01-15T16:59:59.793 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.826 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bceae5eb:::obj-1-56:head# 2021-01-15T16:59:59.826 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.859 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bea17ef6:::obj-1-84:head# 2021-01-15T16:59:59.859 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.892 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:beb88be5:::obj-1-396:head# 2021-01-15T16:59:59.893 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.926 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:beed7b18:::obj-1-569:head# 2021-01-15T16:59:59.926 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T16:59:59.959 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bf0bff49:::obj-1-385:head# 2021-01-15T16:59:59.959 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.001 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bf545d5a:::obj-1-69:head# 2021-01-15T17:00:00.001 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.059 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bf560a7f:::obj-1-40:head# 2021-01-15T17:00:00.059 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.118 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:bff6999c:::obj-1-240:head# 2021-01-15T17:00:00.118 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.193 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c0a33d38:::obj-1-321:head# 2021-01-15T17:00:00.193 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.226 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c1e4c9b4:::obj-1-299:head# 2021-01-15T17:00:00.226 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.259 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c1ec515a:::obj-1-16:head# 2021-01-15T17:00:00.259 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.293 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c22de3c8:::obj-1-110:head# 2021-01-15T17:00:00.293 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.352 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c231cd7c:::obj-1-58:head# 2021-01-15T17:00:00.352 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.409 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c2ee6ace:::obj-1-186:head# 2021-01-15T17:00:00.410 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.442 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c32bfef2:::obj-1-181:head# 2021-01-15T17:00:00.443 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.476 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c575a298:::obj-1-466:head# 2021-01-15T17:00:00.477 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.509 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c58c3f80:::obj-1-493:head# 2021-01-15T17:00:00.510 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.543 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c6442cf2:::obj-1-455:head# 2021-01-15T17:00:00.544 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.576 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c6619413:::obj-1-37:head# 2021-01-15T17:00:00.576 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.609 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c6c2f8e5:::obj-1-519:head# 2021-01-15T17:00:00.610 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.643 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c6f811a2:::obj-1-180:head# 2021-01-15T17:00:00.643 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.676 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c72af975:::obj-1-21:head# 2021-01-15T17:00:00.676 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.709 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c73eb350:::obj-1-356:head# 2021-01-15T17:00:00.710 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.743 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c7a683da:::obj-1-410:head# 2021-01-15T17:00:00.743 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.818 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c7eb9bcc:::obj-1-553:head# 2021-01-15T17:00:00.818 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.851 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c886df99:::obj-1-269:head# 2021-01-15T17:00:00.852 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.885 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c932e3ec:::obj-1-135:head# 2021-01-15T17:00:00.885 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.918 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c9b6507a:::obj-1-520:head# 2021-01-15T17:00:00.918 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.952 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:c9e3b064:::obj-1-244:head# 2021-01-15T17:00:00.952 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:00.985 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cb705d66:::obj-1-377:head# 2021-01-15T17:00:00.985 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.018 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cc80e25c:::obj-1-115:head# 2021-01-15T17:00:01.019 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.068 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cd1ec88e:::obj-1-297:head# 2021-01-15T17:00:01.068 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.102 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cd7e6936:::obj-1-234:head# 2021-01-15T17:00:01.102 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.135 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ce32b69c:::obj-1-30:head# 2021-01-15T17:00:01.135 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.176 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf2b123c:::obj-1-293:head# 2021-01-15T17:00:01.177 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.218 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf2e80a9:::obj-1-542:head# 2021-01-15T17:00:01.218 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.261 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf51da6a:::obj-1-284:head# 2021-01-15T17:00:01.262 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.301 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf590f42:::obj-1-381:head# 2021-01-15T17:00:01.302 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.335 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf68a2b1:::obj-1-68:head# 2021-01-15T17:00:01.335 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.368 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cf6aefb6:::obj-1-28:head# 2021-01-15T17:00:01.368 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.401 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:cfd8e6af:::obj-1-105:head# 2021-01-15T17:00:01.402 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.468 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d0640bad:::obj-1-254:head# 2021-01-15T17:00:01.468 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.501 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d101cda5:::obj-1-259:head# 2021-01-15T17:00:01.502 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.535 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d1be9386:::obj-1-416:head# 2021-01-15T17:00:01.535 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.568 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d243f036:::obj-1-265:head# 2021-01-15T17:00:01.568 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.618 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d2808c34:::obj-1-159:head# 2021-01-15T17:00:01.619 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.651 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d3415824:::obj-1-39:head# 2021-01-15T17:00:01.652 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.685 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d424f07a:::obj-1-547:head# 2021-01-15T17:00:01.686 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.718 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d44139f2:::obj-1-289:head# 2021-01-15T17:00:01.719 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.752 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d46015ab:::obj-1-546:head# 2021-01-15T17:00:01.752 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.785 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d4e503db:::obj-1-525:head# 2021-01-15T17:00:01.785 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.818 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d4e97d18:::obj-1-85:head# 2021-01-15T17:00:01.819 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.866 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d511492e:::obj-1-388:head# 2021-01-15T17:00:01.866 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.932 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d526a7fa:::obj-1-12:head# 2021-01-15T17:00:01.933 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:01.968 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d5667b59:::obj-1-369:head# 2021-01-15T17:00:01.968 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.002 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d57f2261:::obj-1-419:head# 2021-01-15T17:00:02.002 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.035 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d57f281b:::obj-1-250:head# 2021-01-15T17:00:02.035 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.150 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d6e05894:::obj-1-166:head# 2021-01-15T17:00:02.150 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.185 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d7689481:::obj-1-424:head# 2021-01-15T17:00:02.185 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.218 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d7d27743:::obj-1-129:head# 2021-01-15T17:00:02.219 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.251 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d7e4d3e9:::obj-1-127:head# 2021-01-15T17:00:02.252 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.286 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d7f1422a:::obj-1-594:head# 2021-01-15T17:00:02.286 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.342 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d8469fd2:::obj-1-153:head# 2021-01-15T17:00:02.343 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.394 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d84efd9e:::obj-1-435:head# 2021-01-15T17:00:02.394 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.427 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d883004b:::obj-1-162:head# 2021-01-15T17:00:02.427 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.460 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d8b60854:::obj-1-257:head# 2021-01-15T17:00:02.460 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.493 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d91aaae6:::obj-1-1:head# 2021-01-15T17:00:02.494 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.527 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d98695ca:::obj-1-524:head# 2021-01-15T17:00:02.527 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.560 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:d9ae97c8:::obj-1-177:head# 2021-01-15T17:00:02.560 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.593 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:da1713bd:::obj-1-309:head# 2021-01-15T17:00:02.594 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.635 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:da37824c:::obj-1-2:head# 2021-01-15T17:00:02.636 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.669 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:dab221b0:::obj-1-526:head# 2021-01-15T17:00:02.669 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.744 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:db27d885:::obj-1-363:head# 2021-01-15T17:00:02.744 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.777 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:dc001d76:::obj-1-329:head# 2021-01-15T17:00:02.777 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.810 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:dc4f4119:::obj-1-122:head# 2021-01-15T17:00:02.811 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.844 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:dcc36eb7:::obj-1-283:head# 2021-01-15T17:00:02.844 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.877 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:df7da06c:::obj-1-124:head# 2021-01-15T17:00:02.877 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.911 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:df7fcacb:::obj-1-330:head# 2021-01-15T17:00:02.911 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.944 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e05cbe31:::obj-1-413:head# 2021-01-15T17:00:02.944 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:02.977 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e1060a54:::obj-1-384:head# 2021-01-15T17:00:02.978 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.010 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e15b839b:::obj-1-316:head# 2021-01-15T17:00:03.010 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.044 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e161b6e5:::obj-1-103:head# 2021-01-15T17:00:03.044 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.077 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e1d1d79c:::obj-1-512:head# 2021-01-15T17:00:03.078 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.110 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e224b482:::obj-1-501:head# 2021-01-15T17:00:03.111 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.144 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e22d6f76:::obj-1-70:head# 2021-01-15T17:00:03.145 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.177 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e2bd801c:::obj-1-523:head# 2021-01-15T17:00:03.177 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.210 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e3d9f395:::obj-1-397:head# 2021-01-15T17:00:03.211 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.244 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e474ea25:::obj-1-457:head# 2021-01-15T17:00:03.244 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.278 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e49e1ff3:::obj-1-130:head# 2021-01-15T17:00:03.278 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.310 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e56c67a0:::obj-1-191:head# 2021-01-15T17:00:03.311 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.394 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e576e921:::obj-1-376:head# 2021-01-15T17:00:03.395 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.427 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e619fd26:::obj-1-395:head# 2021-01-15T17:00:03.428 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.461 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e625bdc5:::obj-1-522:head# 2021-01-15T17:00:03.461 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.494 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e68a0fa6:::obj-1-394:head# 2021-01-15T17:00:03.495 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.519 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e6cf260f:::obj-1-77:head# 2021-01-15T17:00:03.519 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.552 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e6d6255a:::obj-1-108:head# 2021-01-15T17:00:03.553 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.578 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e7a35d2f:::obj-1-116:head# 2021-01-15T17:00:03.579 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.602 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e7b07de5:::obj-1-238:head# 2021-01-15T17:00:03.603 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.627 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e8037aba:::obj-1-404:head# 2021-01-15T17:00:03.628 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.661 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e8e8b34f:::obj-1-291:head# 2021-01-15T17:00:03.661 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.695 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:e987cd7a:::obj-1-123:head# 2021-01-15T17:00:03.697 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.734 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ea534db1:::obj-1-311:head# 2021-01-15T17:00:03.734 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.761 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eabcc188:::obj-1-325:head# 2021-01-15T17:00:03.761 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.794 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eb086660:::obj-1-595:head# 2021-01-15T17:00:03.795 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.827 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eb2235d3:::obj-1-596:head# 2021-01-15T17:00:03.828 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.861 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eb35ae40:::obj-1-516:head# 2021-01-15T17:00:03.861 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.894 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eb360a13:::obj-1-224:head# 2021-01-15T17:00:03.894 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.927 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eb3a3ea1:::obj-1-120:head# 2021-01-15T17:00:03.928 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:03.961 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ebca9f6e:::obj-1-290:head# 2021-01-15T17:00:03.961 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.044 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ec81db71:::obj-1-478:head# 2021-01-15T17:00:04.045 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.085 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:edeafa4c:::obj-1-362:head# 2021-01-15T17:00:04.086 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.119 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eeb5261b:::obj-1-539:head# 2021-01-15T17:00:04.119 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.152 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:eec00eee:::obj-1-567:head# 2021-01-15T17:00:04.152 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.186 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ef075e10:::obj-1-463:head# 2021-01-15T17:00:04.187 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.222 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ef8e37d9:::obj-1-94:head# 2021-01-15T17:00:04.223 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.325 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:efad7f67:::obj-1-197:head# 2021-01-15T17:00:04.325 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.392 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:efe09ea7:::obj-1-537:head# 2021-01-15T17:00:04.392 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.427 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f0445019:::obj-1-480:head# 2021-01-15T17:00:04.428 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.461 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f08a432c:::obj-1-489:head# 2021-01-15T17:00:04.461 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.494 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f0bd9006:::obj-1-487:head# 2021-01-15T17:00:04.495 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.527 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f11a115d:::obj-1-591:head# 2021-01-15T17:00:04.528 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.561 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f232e0af:::obj-1-146:head# 2021-01-15T17:00:04.561 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.595 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f258db54:::obj-1-521:head# 2021-01-15T17:00:04.595 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.669 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f29cc204:::obj-1-566:head# 2021-01-15T17:00:04.670 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.702 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f2ae2451:::obj-1-66:head# 2021-01-15T17:00:04.703 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.736 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f318810a:::obj-1-160:head# 2021-01-15T17:00:04.736 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.769 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f3c35a0d:::obj-1-67:head# 2021-01-15T17:00:04.770 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.811 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f3ce9c90:::obj-1-88:head# 2021-01-15T17:00:04.811 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.844 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f3e3e81f:::obj-1-304:head# 2021-01-15T17:00:04.845 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.878 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f3f65b0a:::obj-1-367:head# 2021-01-15T17:00:04.878 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.912 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f4ea014b:::obj-1-149:head# 2021-01-15T17:00:04.912 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.944 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f52d115b:::obj-1-383:head# 2021-01-15T17:00:04.945 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:04.978 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f590311e:::obj-1-76:head# 2021-01-15T17:00:04.978 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.020 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f5a37734:::obj-1-51:head# 2021-01-15T17:00:05.020 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.053 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f5e2e77c:::obj-1-179:head# 2021-01-15T17:00:05.053 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.078 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f5e381bc:::obj-1-173:head# 2021-01-15T17:00:05.078 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.103 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f6c0ae0b:::obj-1-247:head# 2021-01-15T17:00:05.103 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.128 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f6db8a82:::obj-1-548:head# 2021-01-15T17:00:05.128 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.153 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f6e2dc46:::obj-1-36:head# 2021-01-15T17:00:05.153 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.178 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f76ee569:::obj-1-587:head# 2021-01-15T17:00:05.179 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.203 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f773b9ed:::obj-1-550:head# 2021-01-15T17:00:05.203 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.228 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f7e1adad:::obj-1-350:head# 2021-01-15T17:00:05.229 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.294 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f7eebf7a:::obj-1-559:head# 2021-01-15T17:00:05.295 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.319 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f877f025:::obj-1-187:head# 2021-01-15T17:00:05.320 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.344 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f992f9e9:::obj-1-402:head# 2021-01-15T17:00:05.345 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.369 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:f9fe960a:::obj-1-418:head# 2021-01-15T17:00:05.370 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.395 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fa9b882f:::obj-1-165:head# 2021-01-15T17:00:05.395 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.419 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fbd6461c:::obj-1-339:head# 2021-01-15T17:00:05.420 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.445 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fbeab28d:::obj-1-431:head# 2021-01-15T17:00:05.445 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.470 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fdea4138:::obj-1-185:head# 2021-01-15T17:00:05.470 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.495 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fdfa7c9b:::obj-1-183:head# 2021-01-15T17:00:05.495 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.520 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:fe1ba57e:::obj-1-90:head# 2021-01-15T17:00:05.520 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.545 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:feb7adea:::obj-1-560:head# 2021-01-15T17:00:05.545 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.570 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ffab306e:::obj-1-97:head# 2021-01-15T17:00:05.570 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.595 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ffdfd397:::obj-1-433:head# 2021-01-15T17:00:05.595 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.620 INFO:tasks.workunit.client.0.smithi145.stdout:Write #2:ffe8cf04:::obj-1-303:head# 2021-01-15T17:00:05.620 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:05.645 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 40 info 2.0( v 40'600 (40'500,40'600] local-lis/les=38/39 n=600 ec=26/26 lis/c=38/38 les/c/f=39/39/0 sis=38) 2021-01-15T17:00:05.678 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2021-01-15T17:00:06.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:493: TEST_backfill_multi_partial: _objectstore_tool_nodown td/osd-backfill-space 1 --op import --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2021-01-15T17:00:06.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-backfill-space 2021-01-15T17:00:06.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T17:00:06.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T17:00:06.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T17:00:06.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-backfill-space/1 2021-01-15T17:00:06.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-space/1 --op import --pgid 1.0 --file td/osd-backfill-space/fillexport.out 2021-01-15T17:00:07.213 INFO:tasks.workunit.client.0.smithi145.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={21=1},2={26=1},3={31=1}} deleted_pools ) 2021-01-15T17:00:07.214 INFO:tasks.workunit.client.0.smithi145.stdout:Importing pgid 1.0 2021-01-15T17:00:07.214 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0002d87b:::obj-fill-318:head# 2021-01-15T17:00:07.214 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.303 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:00676868:::obj-fill-583:head# 2021-01-15T17:00:07.304 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.345 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:00f4514e:::obj-fill-547:head# 2021-01-15T17:00:07.346 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.387 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:01596a3c:::obj-fill-533:head# 2021-01-15T17:00:07.387 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.428 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0285e89c:::obj-fill-190:head# 2021-01-15T17:00:07.429 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.470 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:02bd591e:::obj-fill-463:head# 2021-01-15T17:00:07.470 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.512 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:02c01ffe:::obj-fill-28:head# 2021-01-15T17:00:07.512 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.553 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:03ada31e:::obj-fill-285:head# 2021-01-15T17:00:07.554 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.595 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:03b07c3f:::obj-fill-568:head# 2021-01-15T17:00:07.595 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.637 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:03b85770:::obj-fill-573:head# 2021-01-15T17:00:07.637 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.678 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:03dcbe89:::obj-fill-375:head# 2021-01-15T17:00:07.679 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.721 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04231826:::obj-fill-594:head# 2021-01-15T17:00:07.721 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.771 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:04d48684:::obj-fill-421:head# 2021-01-15T17:00:07.771 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.803 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:05385fcc:::obj-fill-299:head# 2021-01-15T17:00:07.804 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.879 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:05785d38:::obj-fill-159:head# 2021-01-15T17:00:07.879 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.912 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:057ec231:::obj-fill-138:head# 2021-01-15T17:00:07.913 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.946 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:06058b5e:::obj-fill-569:head# 2021-01-15T17:00:07.946 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:07.978 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:06d2a3d9:::obj-fill-284:head# 2021-01-15T17:00:07.979 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.012 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:07597b63:::obj-fill-459:head# 2021-01-15T17:00:08.012 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.070 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:075f4069:::obj-fill-550:head# 2021-01-15T17:00:08.071 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.120 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:08234052:::obj-fill-17:head# 2021-01-15T17:00:08.120 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.153 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:091ce2ec:::obj-fill-472:head# 2021-01-15T17:00:08.154 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.178 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:095fd509:::obj-fill-210:head# 2021-01-15T17:00:08.179 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.204 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0a52a594:::obj-fill-116:head# 2021-01-15T17:00:08.204 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.237 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0a63716b:::obj-fill-225:head# 2021-01-15T17:00:08.238 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.296 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0b0fa6fb:::obj-fill-541:head# 2021-01-15T17:00:08.296 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.337 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0b673f72:::obj-fill-232:head# 2021-01-15T17:00:08.338 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.387 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0bbcd1b2:::obj-fill-582:head# 2021-01-15T17:00:08.388 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.420 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0caf8ce3:::obj-fill-214:head# 2021-01-15T17:00:08.421 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.504 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0d94b9fb:::obj-fill-415:head# 2021-01-15T17:00:08.504 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.537 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0de4e6f7:::obj-fill-94:head# 2021-01-15T17:00:08.537 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.570 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0ef75768:::obj-fill-489:head# 2021-01-15T17:00:08.570 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.604 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0f67284f:::obj-fill-213:head# 2021-01-15T17:00:08.604 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.637 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:0f823df2:::obj-fill-234:head# 2021-01-15T17:00:08.637 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.670 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1005c08d:::obj-fill-490:head# 2021-01-15T17:00:08.671 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.703 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1039525f:::obj-fill-369:head# 2021-01-15T17:00:08.704 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.729 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:105fb36c:::obj-fill-226:head# 2021-01-15T17:00:08.729 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.762 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:111abdeb:::obj-fill-554:head# 2021-01-15T17:00:08.762 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.795 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:11392f3d:::obj-fill-453:head# 2021-01-15T17:00:08.796 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.820 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1169976c:::obj-fill-417:head# 2021-01-15T17:00:08.821 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.854 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:11840a2a:::obj-fill-477:head# 2021-01-15T17:00:08.854 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.880 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:11a4832e:::obj-fill-279:head# 2021-01-15T17:00:08.880 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.904 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:11a4d2f6:::obj-fill-256:head# 2021-01-15T17:00:08.905 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.937 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:12d52a31:::obj-fill-22:head# 2021-01-15T17:00:08.937 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.962 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:133c45db:::obj-fill-257:head# 2021-01-15T17:00:08.962 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:08.995 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13563256:::obj-fill-150:head# 2021-01-15T17:00:08.996 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.020 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13c09e00:::obj-fill-71:head# 2021-01-15T17:00:09.020 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.054 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13e9fbed:::obj-fill-242:head# 2021-01-15T17:00:09.054 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.130 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:13fae6c2:::obj-fill-163:head# 2021-01-15T17:00:09.130 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.162 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:145025e0:::obj-fill-543:head# 2021-01-15T17:00:09.162 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.187 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:14517c84:::obj-fill-13:head# 2021-01-15T17:00:09.188 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.221 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:14802e9c:::obj-fill-127:head# 2021-01-15T17:00:09.221 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.245 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:150f1c96:::obj-fill-62:head# 2021-01-15T17:00:09.246 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.270 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:155e90a2:::obj-fill-537:head# 2021-01-15T17:00:09.271 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.312 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1632c776:::obj-fill-262:head# 2021-01-15T17:00:09.313 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.362 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:163bde28:::obj-fill-523:head# 2021-01-15T17:00:09.363 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.395 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:169c4ad0:::obj-fill-85:head# 2021-01-15T17:00:09.396 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.429 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:16c4da1d:::obj-fill-436:head# 2021-01-15T17:00:09.429 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.462 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17b28c32:::obj-fill-345:head# 2021-01-15T17:00:09.463 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.496 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17b34255:::obj-fill-577:head# 2021-01-15T17:00:09.496 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.529 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17d2541f:::obj-fill-426:head# 2021-01-15T17:00:09.529 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.562 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:17ea19f2:::obj-fill-542:head# 2021-01-15T17:00:09.563 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.613 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:189155ea:::obj-fill-496:head# 2021-01-15T17:00:09.613 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.646 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:191a6d81:::obj-fill-548:head# 2021-01-15T17:00:09.646 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.679 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1a964993:::obj-fill-576:head# 2021-01-15T17:00:09.679 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.793 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1aa623dc:::obj-fill-356:head# 2021-01-15T17:00:09.793 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.829 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1abe5c38:::obj-fill-561:head# 2021-01-15T17:00:09.829 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.871 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1bddc06b:::obj-fill-7:head# 2021-01-15T17:00:09.871 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.904 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1be86b71:::obj-fill-295:head# 2021-01-15T17:00:09.905 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.946 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1c499bca:::obj-fill-240:head# 2021-01-15T17:00:09.946 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:09.982 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1c545e27:::obj-fill-509:head# 2021-01-15T17:00:09.983 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.012 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1c63c8f4:::obj-fill-4:head# 2021-01-15T17:00:10.013 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.069 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1d5a8f59:::obj-fill-377:head# 2021-01-15T17:00:10.070 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.129 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1d85b5df:::obj-fill-201:head# 2021-01-15T17:00:10.130 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.163 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1d9a0d5b:::obj-fill-581:head# 2021-01-15T17:00:10.163 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.196 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1dcf5667:::obj-fill-488:head# 2021-01-15T17:00:10.196 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.229 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1fa3090d:::obj-fill-458:head# 2021-01-15T17:00:10.230 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.263 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1fe937f6:::obj-fill-337:head# 2021-01-15T17:00:10.263 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.296 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:1ff54542:::obj-fill-349:head# 2021-01-15T17:00:10.296 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.345 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:20d9e0ca:::obj-fill-367:head# 2021-01-15T17:00:10.346 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.437 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:21b8e3bb:::obj-fill-538:head# 2021-01-15T17:00:10.438 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.462 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:227183d4:::obj-fill-235:head# 2021-01-15T17:00:10.463 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.487 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:229e5e5d:::obj-fill-522:head# 2021-01-15T17:00:10.488 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.512 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:234f8861:::obj-fill-476:head# 2021-01-15T17:00:10.513 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.538 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:23ac2c9d:::obj-fill-471:head# 2021-01-15T17:00:10.538 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.563 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:25db1742:::obj-fill-200:head# 2021-01-15T17:00:10.563 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.588 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:264f8545:::obj-fill-230:head# 2021-01-15T17:00:10.588 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.613 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:267ba97f:::obj-fill-493:head# 2021-01-15T17:00:10.614 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.638 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:26b067e8:::obj-fill-208:head# 2021-01-15T17:00:10.638 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.663 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:26f521cb:::obj-fill-300:head# 2021-01-15T17:00:10.663 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.688 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:27e70a11:::obj-fill-368:head# 2021-01-15T17:00:10.688 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.713 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:29357d4a:::obj-fill-456:head# 2021-01-15T17:00:10.713 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.738 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:29f2567e:::obj-fill-414:head# 2021-01-15T17:00:10.738 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.763 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2a10cbb6:::obj-fill-228:head# 2021-01-15T17:00:10.763 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.788 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2a1519a6:::obj-fill-148:head# 2021-01-15T17:00:10.788 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.813 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2a686ca6:::obj-fill-411:head# 2021-01-15T17:00:10.813 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.838 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2ab3f384:::obj-fill-77:head# 2021-01-15T17:00:10.838 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.863 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2ae6a1d0:::obj-fill-187:head# 2021-01-15T17:00:10.863 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.888 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2b890959:::obj-fill-222:head# 2021-01-15T17:00:10.888 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.913 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2bcc26c4:::obj-fill-114:head# 2021-01-15T17:00:10.913 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.938 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2bd3a3a2:::obj-fill-281:head# 2021-01-15T17:00:10.938 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.963 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2c1a21d7:::obj-fill-505:head# 2021-01-15T17:00:10.963 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:10.988 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2c255347:::obj-fill-95:head# 2021-01-15T17:00:10.988 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.055 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2c2a3f62:::obj-fill-131:head# 2021-01-15T17:00:11.055 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.080 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2c7d92ce:::obj-fill-442:head# 2021-01-15T17:00:11.080 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.105 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2d1ad117:::obj-fill-443:head# 2021-01-15T17:00:11.105 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.129 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2d57b2f7:::obj-fill-192:head# 2021-01-15T17:00:11.130 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.155 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2d7b53ea:::obj-fill-79:head# 2021-01-15T17:00:11.155 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.180 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2db25266:::obj-fill-121:head# 2021-01-15T17:00:11.182 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.204 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2e27853e:::obj-fill-473:head# 2021-01-15T17:00:11.205 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.230 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2f12cb21:::obj-fill-386:head# 2021-01-15T17:00:11.230 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.255 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2f957332:::obj-fill-422:head# 2021-01-15T17:00:11.255 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.280 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:2feae493:::obj-fill-301:head# 2021-01-15T17:00:11.280 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.305 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:302c95d2:::obj-fill-110:head# 2021-01-15T17:00:11.305 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.330 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:302d6b76:::obj-fill-374:head# 2021-01-15T17:00:11.331 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.355 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:30581f34:::obj-fill-313:head# 2021-01-15T17:00:11.355 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.380 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:307e86ac:::obj-fill-381:head# 2021-01-15T17:00:11.380 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.413 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:30ab77bf:::obj-fill-275:head# 2021-01-15T17:00:11.414 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.447 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:31127e6d:::obj-fill-250:head# 2021-01-15T17:00:11.447 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.480 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:32c93500:::obj-fill-155:head# 2021-01-15T17:00:11.481 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.513 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:339880b7:::obj-fill-227:head# 2021-01-15T17:00:11.514 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.547 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:34069e1e:::obj-fill-292:head# 2021-01-15T17:00:11.547 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.580 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:346dff31:::obj-fill-157:head# 2021-01-15T17:00:11.580 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.613 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:34ade0f1:::obj-fill-575:head# 2021-01-15T17:00:11.614 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.646 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:34bbae43:::obj-fill-90:head# 2021-01-15T17:00:11.647 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.730 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:34e97cf8:::obj-fill-334:head# 2021-01-15T17:00:11.730 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.772 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:35d7ccc2:::obj-fill-510:head# 2021-01-15T17:00:11.772 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.805 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:35f519b6:::obj-fill-513:head# 2021-01-15T17:00:11.805 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.838 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:364727e8:::obj-fill-130:head# 2021-01-15T17:00:11.838 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.888 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:364c0361:::obj-fill-586:head# 2021-01-15T17:00:11.889 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.923 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:368e7baf:::obj-fill-546:head# 2021-01-15T17:00:11.923 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:11.961 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:36ffb224:::obj-fill-82:head# 2021-01-15T17:00:11.961 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.027 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:371cb800:::obj-fill-330:head# 2021-01-15T17:00:12.028 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.086 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:37d5378d:::obj-fill-175:head# 2021-01-15T17:00:12.086 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:385593f3:::obj-fill-325:head# 2021-01-15T17:00:12.147 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:388d5073:::obj-fill-317:head# 2021-01-15T17:00:12.180 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:394af079:::obj-fill-59:head# 2021-01-15T17:00:12.222 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.256 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3a41ad37:::obj-fill-409:head# 2021-01-15T17:00:12.256 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.330 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3b3c63d2:::obj-fill-136:head# 2021-01-15T17:00:12.331 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.412 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3bb50131:::obj-fill-512:head# 2021-01-15T17:00:12.413 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.447 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3bb52bb5:::obj-fill-188:head# 2021-01-15T17:00:12.447 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.480 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3c0683d1:::obj-fill-372:head# 2021-01-15T17:00:12.481 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.514 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3c67d3f4:::obj-fill-249:head# 2021-01-15T17:00:12.514 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.548 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3c6a4a4c:::obj-fill-495:head# 2021-01-15T17:00:12.548 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.581 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3c7fc8c9:::obj-fill-324:head# 2021-01-15T17:00:12.581 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.606 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3d940c50:::obj-fill-591:head# 2021-01-15T17:00:12.606 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.631 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3ed5508e:::obj-fill-558:head# 2021-01-15T17:00:12.631 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.655 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:3f598316:::obj-fill-389:head# 2021-01-15T17:00:12.656 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.681 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:401cd0ce:::obj-fill-251:head# 2021-01-15T17:00:12.681 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.706 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:40fae006:::obj-fill-14:head# 2021-01-15T17:00:12.706 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.731 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4120b10f:::obj-fill-165:head# 2021-01-15T17:00:12.731 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.756 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:41b382c2:::obj-fill-289:head# 2021-01-15T17:00:12.756 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.780 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4277cdea:::obj-fill-319:head# 2021-01-15T17:00:12.781 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.806 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:42978c3c:::obj-fill-174:head# 2021-01-15T17:00:12.806 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.831 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:42b08dc2:::obj-fill-553:head# 2021-01-15T17:00:12.831 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.856 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:42ec4bf6:::obj-fill-209:head# 2021-01-15T17:00:12.856 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.881 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:430c129d:::obj-fill-448:head# 2021-01-15T17:00:12.881 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.906 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4335d553:::obj-fill-600:head# 2021-01-15T17:00:12.906 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.973 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:44804f62:::obj-fill-532:head# 2021-01-15T17:00:12.973 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:12.998 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:44954d54:::obj-fill-2:head# 2021-01-15T17:00:12.998 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4502f904:::obj-fill-429:head# 2021-01-15T17:00:13.023 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.047 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4568ddcb:::obj-fill-382:head# 2021-01-15T17:00:13.048 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.072 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:45939974:::obj-fill-563:head# 2021-01-15T17:00:13.073 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.097 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:462a5bbd:::obj-fill-567:head# 2021-01-15T17:00:13.098 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.122 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:47773ae4:::obj-fill-307:head# 2021-01-15T17:00:13.123 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.147 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4802b263:::obj-fill-387:head# 2021-01-15T17:00:13.148 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.172 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:483d938e:::obj-fill-128:head# 2021-01-15T17:00:13.173 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.198 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:48a7ad29:::obj-fill-147:head# 2021-01-15T17:00:13.199 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.222 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:493b61ce:::obj-fill-454:head# 2021-01-15T17:00:13.223 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.247 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4afb70fe:::obj-fill-186:head# 2021-01-15T17:00:13.248 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.273 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4bff08e3:::obj-fill-352:head# 2021-01-15T17:00:13.273 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.298 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4c01eabe:::obj-fill-370:head# 2021-01-15T17:00:13.298 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.323 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4c131560:::obj-fill-6:head# 2021-01-15T17:00:13.324 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.347 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4c4c5e14:::obj-fill-350:head# 2021-01-15T17:00:13.348 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.372 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4c80dc2c:::obj-fill-357:head# 2021-01-15T17:00:13.373 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.398 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4c9dee9b:::obj-fill-540:head# 2021-01-15T17:00:13.401 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.423 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4ce40c18:::obj-fill-280:head# 2021-01-15T17:00:13.423 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.448 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4d854fc3:::obj-fill-585:head# 2021-01-15T17:00:13.448 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.473 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:4f615b53:::obj-fill-101:head# 2021-01-15T17:00:13.474 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.499 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5042b30f:::obj-fill-84:head# 2021-01-15T17:00:13.499 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.523 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:507dc08b:::obj-fill-336:head# 2021-01-15T17:00:13.523 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.559 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:512fcf5a:::obj-fill-169:head# 2021-01-15T17:00:13.559 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.631 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:51a0c8f8:::obj-fill-578:head# 2021-01-15T17:00:13.631 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.665 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:51fd05a2:::obj-fill-425:head# 2021-01-15T17:00:13.665 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.698 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:52400ebc:::obj-fill-81:head# 2021-01-15T17:00:13.699 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.732 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5288d40c:::obj-fill-16:head# 2021-01-15T17:00:13.732 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.765 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5295b0ee:::obj-fill-224:head# 2021-01-15T17:00:13.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.799 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:531e4c60:::obj-fill-103:head# 2021-01-15T17:00:13.799 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.831 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:54259f28:::obj-fill-584:head# 2021-01-15T17:00:13.831 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.864 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:542d619c:::obj-fill-528:head# 2021-01-15T17:00:13.865 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.898 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5503ca3e:::obj-fill-494:head# 2021-01-15T17:00:13.899 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.931 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5555c48b:::obj-fill-403:head# 2021-01-15T17:00:13.932 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:13.965 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:558d1ec7:::obj-fill-447:head# 2021-01-15T17:00:13.966 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.023 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:55aa2233:::obj-fill-21:head# 2021-01-15T17:00:14.023 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.112 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:55aaf8c4:::obj-fill-29:head# 2021-01-15T17:00:14.112 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.148 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:55e246e1:::obj-fill-65:head# 2021-01-15T17:00:14.148 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.181 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:563b14a8:::obj-fill-120:head# 2021-01-15T17:00:14.181 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.256 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:56e1df25:::obj-fill-305:head# 2021-01-15T17:00:14.256 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.289 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5702979c:::obj-fill-73:head# 2021-01-15T17:00:14.290 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.337 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:570722ba:::obj-fill-219:head# 2021-01-15T17:00:14.338 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.398 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:57f7a09b:::obj-fill-571:head# 2021-01-15T17:00:14.398 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.431 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:585309ba:::obj-fill-144:head# 2021-01-15T17:00:14.431 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.495 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:585c7c9c:::obj-fill-437:head# 2021-01-15T17:00:14.495 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.532 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:587c63db:::obj-fill-182:head# 2021-01-15T17:00:14.532 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.565 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:58f34d6b:::obj-fill-598:head# 2021-01-15T17:00:14.567 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.598 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:58f40088:::obj-fill-298:head# 2021-01-15T17:00:14.599 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.631 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:591e150c:::obj-fill-32:head# 2021-01-15T17:00:14.632 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.664 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:59971ddb:::obj-fill-559:head# 2021-01-15T17:00:14.665 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.698 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5adafca4:::obj-fill-56:head# 2021-01-15T17:00:14.698 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.731 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5b32b04f:::obj-fill-484:head# 2021-01-15T17:00:14.732 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.765 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5bdaeb16:::obj-fill-99:head# 2021-01-15T17:00:14.765 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.798 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5c1d6f4b:::obj-fill-133:head# 2021-01-15T17:00:14.799 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.823 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5c92b8f1:::obj-fill-255:head# 2021-01-15T17:00:14.823 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.889 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5ca4850f:::obj-fill-238:head# 2021-01-15T17:00:14.890 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.932 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5d928a5b:::obj-fill-395:head# 2021-01-15T17:00:14.932 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.957 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5dd49320:::obj-fill-423:head# 2021-01-15T17:00:14.957 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:14.982 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5e7bc368:::obj-fill-506:head# 2021-01-15T17:00:14.983 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.007 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5ebd5ee3:::obj-fill-322:head# 2021-01-15T17:00:15.007 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.032 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5edeedf1:::obj-fill-557:head# 2021-01-15T17:00:15.032 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.057 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:5facf8e2:::obj-fill-15:head# 2021-01-15T17:00:15.057 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.084 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:603a2c76:::obj-fill-439:head# 2021-01-15T17:00:15.090 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.107 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:60b30714:::obj-fill-53:head# 2021-01-15T17:00:15.108 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.136 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:612b4a8a:::obj-fill-412:head# 2021-01-15T17:00:15.136 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.160 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:61a111b6:::obj-fill-87:head# 2021-01-15T17:00:15.163 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.183 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:63a0e8a2:::obj-fill-266:head# 2021-01-15T17:00:15.184 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.207 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:63aa4d60:::obj-fill-68:head# 2021-01-15T17:00:15.207 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.232 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:63ac5b95:::obj-fill-405:head# 2021-01-15T17:00:15.232 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.259 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6415a9e3:::obj-fill-400:head# 2021-01-15T17:00:15.259 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.283 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:64ba03ae:::obj-fill-20:head# 2021-01-15T17:00:15.283 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.307 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6512f015:::obj-fill-597:head# 2021-01-15T17:00:15.307 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.336 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:65433104:::obj-fill-504:head# 2021-01-15T17:00:15.337 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.357 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:65950c58:::obj-fill-86:head# 2021-01-15T17:00:15.357 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.382 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:65b6b2d7:::obj-fill-39:head# 2021-01-15T17:00:15.384 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.408 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:661c8afd:::obj-fill-267:head# 2021-01-15T17:00:15.414 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.432 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:66f87c94:::obj-fill-229:head# 2021-01-15T17:00:15.433 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.457 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:67292cef:::obj-fill-93:head# 2021-01-15T17:00:15.457 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.523 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:67f0e12c:::obj-fill-363:head# 2021-01-15T17:00:15.524 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.549 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:696eb83b:::obj-fill-185:head# 2021-01-15T17:00:15.549 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.582 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6b33981c:::obj-fill-259:head# 2021-01-15T17:00:15.582 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.607 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6b5a527c:::obj-fill-485:head# 2021-01-15T17:00:15.607 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.632 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6b6f1852:::obj-fill-390:head# 2021-01-15T17:00:15.632 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.657 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6ba5c5f8:::obj-fill-304:head# 2021-01-15T17:00:15.657 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.682 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6bce929d:::obj-fill-518:head# 2021-01-15T17:00:15.682 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.707 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6c8ffc04:::obj-fill-61:head# 2021-01-15T17:00:15.707 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.732 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6ca7e551:::obj-fill-40:head# 2021-01-15T17:00:15.732 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.766 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6caff2cd:::obj-fill-332:head# 2021-01-15T17:00:15.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.800 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6ce4adc7:::obj-fill-574:head# 2021-01-15T17:00:15.801 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.832 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6d09d770:::obj-fill-74:head# 2021-01-15T17:00:15.832 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.865 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6e039822:::obj-fill-278:head# 2021-01-15T17:00:15.865 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.899 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6f2cd109:::obj-fill-297:head# 2021-01-15T17:00:15.899 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.932 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6f3754c4:::obj-fill-88:head# 2021-01-15T17:00:15.932 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:15.966 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6f86cfd2:::obj-fill-511:head# 2021-01-15T17:00:15.967 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.001 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:6fe3f3f7:::obj-fill-193:head# 2021-01-15T17:00:16.001 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.036 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7002c57d:::obj-fill-50:head# 2021-01-15T17:00:16.037 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.176 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7030b2a3:::obj-fill-217:head# 2021-01-15T17:00:16.176 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.224 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:70665087:::obj-fill-51:head# 2021-01-15T17:00:16.224 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.257 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:70a0f5e0:::obj-fill-11:head# 2021-01-15T17:00:16.257 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.321 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:71f4643d:::obj-fill-236:head# 2021-01-15T17:00:16.321 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.382 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:722fbd0f:::obj-fill-302:head# 2021-01-15T17:00:16.382 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.415 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:72584da9:::obj-fill-445:head# 2021-01-15T17:00:16.416 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.449 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:72d91d24:::obj-fill-218:head# 2021-01-15T17:00:16.449 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.482 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:72f7268b:::obj-fill-106:head# 2021-01-15T17:00:16.483 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.515 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:74077abf:::obj-fill-435:head# 2021-01-15T17:00:16.516 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.549 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:740ca050:::obj-fill-112:head# 2021-01-15T17:00:16.549 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.590 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7435e881:::obj-fill-78:head# 2021-01-15T17:00:16.591 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.624 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:74ba0bb7:::obj-fill-478:head# 2021-01-15T17:00:16.624 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.657 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:74d82091:::obj-fill-96:head# 2021-01-15T17:00:16.657 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.690 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7515f56c:::obj-fill-483:head# 2021-01-15T17:00:16.691 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.724 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7550d6fe:::obj-fill-286:head# 2021-01-15T17:00:16.724 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.758 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:75807edc:::obj-fill-223:head# 2021-01-15T17:00:16.759 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.832 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:762ba324:::obj-fill-172:head# 2021-01-15T17:00:16.833 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.866 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:76369bda:::obj-fill-570:head# 2021-01-15T17:00:16.866 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.899 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7667a5b0:::obj-fill-196:head# 2021-01-15T17:00:16.899 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.932 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:77736c0c:::obj-fill-566:head# 2021-01-15T17:00:16.933 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:16.966 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:77fb47a4:::obj-fill-314:head# 2021-01-15T17:00:16.966 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.048 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:785ad894:::obj-fill-80:head# 2021-01-15T17:00:17.049 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.099 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:785e3d14:::obj-fill-530:head# 2021-01-15T17:00:17.100 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.132 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:788efef4:::obj-fill-595:head# 2021-01-15T17:00:17.133 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.157 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:789ec5ab:::obj-fill-44:head# 2021-01-15T17:00:17.158 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.182 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:798a3fa8:::obj-fill-517:head# 2021-01-15T17:00:17.183 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.207 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:79a105ef:::obj-fill-315:head# 2021-01-15T17:00:17.208 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.233 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:79d02f04:::obj-fill-243:head# 2021-01-15T17:00:17.233 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.257 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7a00dfcb:::obj-fill-402:head# 2021-01-15T17:00:17.258 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.282 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7b982684:::obj-fill-168:head# 2021-01-15T17:00:17.283 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.307 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7bb36851:::obj-fill-440:head# 2021-01-15T17:00:17.308 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.332 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7bb49d8c:::obj-fill-55:head# 2021-01-15T17:00:17.333 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.358 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7c77b783:::obj-fill-580:head# 2021-01-15T17:00:17.358 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.383 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7c89d75b:::obj-fill-555:head# 2021-01-15T17:00:17.383 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.458 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7d12fb67:::obj-fill-399:head# 2021-01-15T17:00:17.458 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.483 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7d139961:::obj-fill-92:head# 2021-01-15T17:00:17.483 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.508 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7d39ffd0:::obj-fill-135:head# 2021-01-15T17:00:17.509 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.533 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7e534cc4:::obj-fill-282:head# 2021-01-15T17:00:17.533 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.558 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7e7a08e0:::obj-fill-108:head# 2021-01-15T17:00:17.558 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.583 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7e8e2b67:::obj-fill-244:head# 2021-01-15T17:00:17.583 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.616 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7eea215e:::obj-fill-139:head# 2021-01-15T17:00:17.616 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.641 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7efa3c6e:::obj-fill-408:head# 2021-01-15T17:00:17.641 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.666 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7f03a24b:::obj-fill-45:head# 2021-01-15T17:00:17.666 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.691 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7f6a2e2b:::obj-fill-427:head# 2021-01-15T17:00:17.691 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.716 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7fb0abd6:::obj-fill-479:head# 2021-01-15T17:00:17.717 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.741 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:7fb77de1:::obj-fill-167:head# 2021-01-15T17:00:17.741 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.766 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:80012a03:::obj-fill-105:head# 2021-01-15T17:00:17.766 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.791 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:80075ae5:::obj-fill-376:head# 2021-01-15T17:00:17.791 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.816 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8089baa5:::obj-fill-97:head# 2021-01-15T17:00:17.817 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.849 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:80d0b4c5:::obj-fill-123:head# 2021-01-15T17:00:17.850 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.883 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:810f319f:::obj-fill-552:head# 2021-01-15T17:00:17.884 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.908 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:81427ef2:::obj-fill-474:head# 2021-01-15T17:00:17.908 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.933 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:818d780c:::obj-fill-33:head# 2021-01-15T17:00:17.933 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.958 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:819f31b9:::obj-fill-366:head# 2021-01-15T17:00:17.958 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:17.983 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8205549d:::obj-fill-125:head# 2021-01-15T17:00:17.983 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.008 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:821f56e5:::obj-fill-339:head# 2021-01-15T17:00:18.008 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.086 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:822807c6:::obj-fill-69:head# 2021-01-15T17:00:18.087 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.150 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:822fcf61:::obj-fill-560:head# 2021-01-15T17:00:18.150 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.183 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:82ad696b:::obj-fill-149:head# 2021-01-15T17:00:18.183 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.216 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:835cc771:::obj-fill-446:head# 2021-01-15T17:00:18.217 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.250 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8386bee1:::obj-fill-137:head# 2021-01-15T17:00:18.250 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.283 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:83c1303a:::obj-fill-311:head# 2021-01-15T17:00:18.285 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.329 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:83cf1c2a:::obj-fill-360:head# 2021-01-15T17:00:18.330 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.375 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:83d70321:::obj-fill-351:head# 2021-01-15T17:00:18.375 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.408 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:83ef162d:::obj-fill-331:head# 2021-01-15T17:00:18.408 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.441 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:840f8f20:::obj-fill-398:head# 2021-01-15T17:00:18.442 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.475 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:84d2e456:::obj-fill-207:head# 2021-01-15T17:00:18.475 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.517 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:856697ff:::obj-fill-333:head# 2021-01-15T17:00:18.517 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.550 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:858be937:::obj-fill-418:head# 2021-01-15T17:00:18.550 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.583 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:85a0b302:::obj-fill-438:head# 2021-01-15T17:00:18.583 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.616 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:85fa942e:::obj-fill-272:head# 2021-01-15T17:00:18.617 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.650 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:863eb6f8:::obj-fill-320:head# 2021-01-15T17:00:18.650 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.683 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:868d4588:::obj-fill-19:head# 2021-01-15T17:00:18.684 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.758 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8759d023:::obj-fill-466:head# 2021-01-15T17:00:18.759 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.791 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:88072111:::obj-fill-312:head# 2021-01-15T17:00:18.792 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.825 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:886edcef:::obj-fill-380:head# 2021-01-15T17:00:18.825 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.858 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:88ce753e:::obj-fill-43:head# 2021-01-15T17:00:18.858 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.891 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:890e1d83:::obj-fill-590:head# 2021-01-15T17:00:18.892 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.925 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:891566a3:::obj-fill-341:head# 2021-01-15T17:00:18.925 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.958 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:891ca235:::obj-fill-177:head# 2021-01-15T17:00:18.958 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:18.991 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:897fa9d0:::obj-fill-283:head# 2021-01-15T17:00:18.993 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.025 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:89cd304b:::obj-fill-592:head# 2021-01-15T17:00:19.025 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.058 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:89d04c4a:::obj-fill-388:head# 2021-01-15T17:00:19.059 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.092 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:89db45d9:::obj-fill-247:head# 2021-01-15T17:00:19.092 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.125 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8ad879c5:::obj-fill-492:head# 2021-01-15T17:00:19.126 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.158 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8af4b5de:::obj-fill-37:head# 2021-01-15T17:00:19.159 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.192 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8b132e86:::obj-fill-35:head# 2021-01-15T17:00:19.192 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.225 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8b80a60a:::obj-fill-100:head# 2021-01-15T17:00:19.225 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.258 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8bb64fa6:::obj-fill-599:head# 2021-01-15T17:00:19.259 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.292 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8bed03f5:::obj-fill-5:head# 2021-01-15T17:00:19.292 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.367 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8c60e135:::obj-fill-470:head# 2021-01-15T17:00:19.367 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.400 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8c774623:::obj-fill-38:head# 2021-01-15T17:00:19.401 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.433 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8cc528ae:::obj-fill-519:head# 2021-01-15T17:00:19.434 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.467 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8cf73826:::obj-fill-460:head# 2021-01-15T17:00:19.468 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.500 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8d3564b3:::obj-fill-452:head# 2021-01-15T17:00:19.501 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.525 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8dfe1ca2:::obj-fill-469:head# 2021-01-15T17:00:19.526 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.550 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8e527672:::obj-fill-246:head# 2021-01-15T17:00:19.551 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.575 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8e96103f:::obj-fill-444:head# 2021-01-15T17:00:19.576 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.600 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8ed6417f:::obj-fill-146:head# 2021-01-15T17:00:19.601 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.626 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8eda2ffc:::obj-fill-539:head# 2021-01-15T17:00:19.626 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.650 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8f08dd48:::obj-fill-26:head# 2021-01-15T17:00:19.651 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.675 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8f111b25:::obj-fill-18:head# 2021-01-15T17:00:19.675 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.700 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:8fde63f8:::obj-fill-338:head# 2021-01-15T17:00:19.701 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.725 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9030746d:::obj-fill-468:head# 2021-01-15T17:00:19.725 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.750 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9115d8a1:::obj-fill-260:head# 2021-01-15T17:00:19.751 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.775 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:91649909:::obj-fill-140:head# 2021-01-15T17:00:19.776 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.800 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:91c1081d:::obj-fill-404:head# 2021-01-15T17:00:19.801 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.826 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:91f42503:::obj-fill-216:head# 2021-01-15T17:00:19.826 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.850 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:927b34a0:::obj-fill-63:head# 2021-01-15T17:00:19.851 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.875 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:92a00a91:::obj-fill-467:head# 2021-01-15T17:00:19.876 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.900 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:930a7268:::obj-fill-450:head# 2021-01-15T17:00:19.901 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.925 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:93c3d0bf:::obj-fill-118:head# 2021-01-15T17:00:19.926 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:19.950 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:93ca7c9e:::obj-fill-132:head# 2021-01-15T17:00:19.951 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.009 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:942435d5:::obj-fill-378:head# 2021-01-15T17:00:20.010 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.053 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9445f8c5:::obj-fill-245:head# 2021-01-15T17:00:20.054 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.120 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9457c8d9:::obj-fill-535:head# 2021-01-15T17:00:20.121 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.150 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:950dc56e:::obj-fill-348:head# 2021-01-15T17:00:20.151 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.175 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9559ee1d:::obj-fill-545:head# 2021-01-15T17:00:20.176 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.201 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:957c5cc4:::obj-fill-212:head# 2021-01-15T17:00:20.201 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.234 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:95bdac02:::obj-fill-588:head# 2021-01-15T17:00:20.234 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.267 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:95f202af:::obj-fill-126:head# 2021-01-15T17:00:20.267 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.301 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:96789003:::obj-fill-83:head# 2021-01-15T17:00:20.301 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.346 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:96d494a8:::obj-fill-316:head# 2021-01-15T17:00:20.347 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.401 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:97adb85b:::obj-fill-233:head# 2021-01-15T17:00:20.401 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.459 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:980d998c:::obj-fill-197:head# 2021-01-15T17:00:20.459 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.492 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:98235c44:::obj-fill-396:head# 2021-01-15T17:00:20.493 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.525 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:987d3b87:::obj-fill-327:head# 2021-01-15T17:00:20.526 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.559 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:990a5e24:::obj-fill-515:head# 2021-01-15T17:00:20.559 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.592 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:995fec05:::obj-fill-191:head# 2021-01-15T17:00:20.593 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.659 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9a769b75:::obj-fill-587:head# 2021-01-15T17:00:20.660 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.692 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9b141990:::obj-fill-394:head# 2021-01-15T17:00:20.693 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.734 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9bb108fb:::obj-fill-579:head# 2021-01-15T17:00:20.735 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.767 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9ce19d92:::obj-fill-361:head# 2021-01-15T17:00:20.768 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.801 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9d8c3f95:::obj-fill-572:head# 2021-01-15T17:00:20.801 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.834 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9e283e00:::obj-fill-254:head# 2021-01-15T17:00:20.834 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.868 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9e3a1a61:::obj-fill-58:head# 2021-01-15T17:00:20.868 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.901 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9e8f6029:::obj-fill-41:head# 2021-01-15T17:00:20.902 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.934 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9ef117aa:::obj-fill-520:head# 2021-01-15T17:00:20.935 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:20.968 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9f3bdfc2:::obj-fill-184:head# 2021-01-15T17:00:20.968 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.001 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9fd89320:::obj-fill-364:head# 2021-01-15T17:00:21.002 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.034 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:9feb7062:::obj-fill-419:head# 2021-01-15T17:00:21.034 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.067 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a07504c4:::obj-fill-516:head# 2021-01-15T17:00:21.068 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.101 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a1897e34:::obj-fill-153:head# 2021-01-15T17:00:21.101 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.134 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a19a9d76:::obj-fill-30:head# 2021-01-15T17:00:21.134 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.168 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a1a49c98:::obj-fill-98:head# 2021-01-15T17:00:21.168 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.201 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a1ea911a:::obj-fill-36:head# 2021-01-15T17:00:21.201 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.234 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a24bad98:::obj-fill-335:head# 2021-01-15T17:00:21.235 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.301 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a263b16f:::obj-fill-326:head# 2021-01-15T17:00:21.301 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.334 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a27d661e:::obj-fill-152:head# 2021-01-15T17:00:21.334 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.367 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a282a129:::obj-fill-451:head# 2021-01-15T17:00:21.368 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.401 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a28f05ba:::obj-fill-241:head# 2021-01-15T17:00:21.401 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.434 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a2cdc5cc:::obj-fill-180:head# 2021-01-15T17:00:21.435 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.468 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a30777ea:::obj-fill-486:head# 2021-01-15T17:00:21.468 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.501 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a3a65661:::obj-fill-151:head# 2021-01-15T17:00:21.501 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.534 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a3cb2967:::obj-fill-195:head# 2021-01-15T17:00:21.535 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.576 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a3ffc098:::obj-fill-293:head# 2021-01-15T17:00:21.576 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.609 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a4a31ccc:::obj-fill-413:head# 2021-01-15T17:00:21.610 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.643 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a51c3677:::obj-fill-384:head# 2021-01-15T17:00:21.643 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.676 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a5e941a7:::obj-fill-141:head# 2021-01-15T17:00:21.676 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.709 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a6cdd8b2:::obj-fill-342:head# 2021-01-15T17:00:21.710 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.743 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a6e20b86:::obj-fill-593:head# 2021-01-15T17:00:21.743 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.776 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a7a9ccf9:::obj-fill-129:head# 2021-01-15T17:00:21.777 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.809 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a7b702d3:::obj-fill-49:head# 2021-01-15T17:00:21.810 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.843 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a8daf8bd:::obj-fill-391:head# 2021-01-15T17:00:21.843 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.876 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:a99c3cef:::obj-fill-171:head# 2021-01-15T17:00:21.877 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.943 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:aabb5718:::obj-fill-521:head# 2021-01-15T17:00:21.943 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:21.976 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ab0cb851:::obj-fill-183:head# 2021-01-15T17:00:21.976 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.009 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ab7a05e4:::obj-fill-203:head# 2021-01-15T17:00:22.010 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.069 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ab7dda83:::obj-fill-113:head# 2021-01-15T17:00:22.069 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.145 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:abb3502c:::obj-fill-461:head# 2021-01-15T17:00:22.145 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.193 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ac1c41be:::obj-fill-162:head# 2021-01-15T17:00:22.194 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.226 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ac2114ba:::obj-fill-239:head# 2021-01-15T17:00:22.227 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.259 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ad355fca:::obj-fill-271:head# 2021-01-15T17:00:22.260 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.321 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ad4634a0:::obj-fill-487:head# 2021-01-15T17:00:22.321 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.385 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:add2bb13:::obj-fill-31:head# 2021-01-15T17:00:22.385 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.426 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:aeb6258d:::obj-fill-124:head# 2021-01-15T17:00:22.427 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.460 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:af17f7d7:::obj-fill-464:head# 2021-01-15T17:00:22.460 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.494 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:af1e1cf0:::obj-fill-237:head# 2021-01-15T17:00:22.494 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.577 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:afc439ce:::obj-fill-248:head# 2021-01-15T17:00:22.577 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.610 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:afceab4d:::obj-fill-527:head# 2021-01-15T17:00:22.610 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.643 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b03af40c:::obj-fill-143:head# 2021-01-15T17:00:22.643 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.685 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b04cf0db:::obj-fill-507:head# 2021-01-15T17:00:22.685 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.726 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b07896f0:::obj-fill-269:head# 2021-01-15T17:00:22.727 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.768 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b0b954d5:::obj-fill-524:head# 2021-01-15T17:00:22.768 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.810 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b1cbfcb7:::obj-fill-134:head# 2021-01-15T17:00:22.810 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.852 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b2978fd3:::obj-fill-27:head# 2021-01-15T17:00:22.853 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.893 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b394798f:::obj-fill-156:head# 2021-01-15T17:00:22.893 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.935 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b3c6f36d:::obj-fill-401:head# 2021-01-15T17:00:22.935 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:22.976 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b3f8e5ac:::obj-fill-290:head# 2021-01-15T17:00:22.977 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.018 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b41ed5f8:::obj-fill-76:head# 2021-01-15T17:00:23.019 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.060 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b4878e76:::obj-fill-253:head# 2021-01-15T17:00:23.060 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.101 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b4e19741:::obj-fill-503:head# 2021-01-15T17:00:23.102 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.143 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b53a3690:::obj-fill-264:head# 2021-01-15T17:00:23.144 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.218 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b545ab4f:::obj-fill-24:head# 2021-01-15T17:00:23.219 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.260 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b55af5e6:::obj-fill-491:head# 2021-01-15T17:00:23.260 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.310 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b57edf82:::obj-fill-526:head# 2021-01-15T17:00:23.310 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.353 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b589d2be:::obj-fill-323:head# 2021-01-15T17:00:23.354 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.393 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b6207f32:::obj-fill-344:head# 2021-01-15T17:00:23.394 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.435 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b757ee2e:::obj-fill-433:head# 2021-01-15T17:00:23.436 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.477 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b75cc479:::obj-fill-34:head# 2021-01-15T17:00:23.477 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.518 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b7b03117:::obj-fill-42:head# 2021-01-15T17:00:23.519 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.560 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b7e6205d:::obj-fill-501:head# 2021-01-15T17:00:23.561 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.602 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b8d18b7a:::obj-fill-353:head# 2021-01-15T17:00:23.602 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.643 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b9189c29:::obj-fill-328:head# 2021-01-15T17:00:23.644 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.685 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b98e07aa:::obj-fill-54:head# 2021-01-15T17:00:23.686 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.727 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:b9cd1e9a:::obj-fill-371:head# 2021-01-15T17:00:23.727 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.760 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bc3322bb:::obj-fill-340:head# 2021-01-15T17:00:23.760 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.793 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bcf43161:::obj-fill-294:head# 2021-01-15T17:00:23.794 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.875 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bd07fc4f:::obj-fill-549:head# 2021-01-15T17:00:23.875 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.902 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bdf48e47:::obj-fill-430:head# 2021-01-15T17:00:23.902 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.943 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:be01c111:::obj-fill-1:head# 2021-01-15T17:00:23.944 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:23.977 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:be3a4bc7:::obj-fill-102:head# 2021-01-15T17:00:23.977 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.002 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:be8e1fcd:::obj-fill-321:head# 2021-01-15T17:00:24.002 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.027 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bf5fcbac:::obj-fill-287:head# 2021-01-15T17:00:24.027 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.102 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bf806d14:::obj-fill-115:head# 2021-01-15T17:00:24.102 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.135 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bfb28823:::obj-fill-72:head# 2021-01-15T17:00:24.136 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.168 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:bfef805e:::obj-fill-119:head# 2021-01-15T17:00:24.169 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.202 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c02f8e10:::obj-fill-497:head# 2021-01-15T17:00:24.203 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.235 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c050cd6e:::obj-fill-8:head# 2021-01-15T17:00:24.236 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.269 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c218e297:::obj-fill-173:head# 2021-01-15T17:00:24.269 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.302 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c2bc66fe:::obj-fill-252:head# 2021-01-15T17:00:24.302 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.361 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c3b55903:::obj-fill-428:head# 2021-01-15T17:00:24.361 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.410 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c51907b0:::obj-fill-457:head# 2021-01-15T17:00:24.411 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.485 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c561dc83:::obj-fill-75:head# 2021-01-15T17:00:24.485 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.518 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c5d1e1a9:::obj-fill-268:head# 2021-01-15T17:00:24.519 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.552 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c60ab0d0:::obj-fill-303:head# 2021-01-15T17:00:24.552 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.585 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c644ce52:::obj-fill-514:head# 2021-01-15T17:00:24.586 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.618 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c6ab3a11:::obj-fill-407:head# 2021-01-15T17:00:24.619 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.652 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c74377bf:::obj-fill-432:head# 2021-01-15T17:00:24.653 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.685 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c767fff6:::obj-fill-531:head# 2021-01-15T17:00:24.685 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.727 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c7a9adad:::obj-fill-475:head# 2021-01-15T17:00:24.727 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.760 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c7e3a9c2:::obj-fill-465:head# 2021-01-15T17:00:24.760 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.794 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c850aed7:::obj-fill-424:head# 2021-01-15T17:00:24.794 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.827 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c8a0088a:::obj-fill-329:head# 2021-01-15T17:00:24.828 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.860 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:c9d419f9:::obj-fill-176:head# 2021-01-15T17:00:24.861 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.894 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:caf25ec2:::obj-fill-274:head# 2021-01-15T17:00:24.894 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.927 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cb625830:::obj-fill-70:head# 2021-01-15T17:00:24.927 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.961 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cbb65125:::obj-fill-529:head# 2021-01-15T17:00:24.961 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:24.994 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cbe2544c:::obj-fill-3:head# 2021-01-15T17:00:24.995 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.027 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cbfce4ee:::obj-fill-455:head# 2021-01-15T17:00:25.028 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.061 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cc62419c:::obj-fill-498:head# 2021-01-15T17:00:25.061 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.135 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cce831fa:::obj-fill-160:head# 2021-01-15T17:00:25.136 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.170 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cda71af4:::obj-fill-122:head# 2021-01-15T17:00:25.170 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.211 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cea7a728:::obj-fill-265:head# 2021-01-15T17:00:25.211 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.253 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cee19f0a:::obj-fill-142:head# 2021-01-15T17:00:25.254 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.294 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cf22117e:::obj-fill-499:head# 2021-01-15T17:00:25.294 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.335 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cfbfa66d:::obj-fill-66:head# 2021-01-15T17:00:25.336 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.377 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cfc2552c:::obj-fill-211:head# 2021-01-15T17:00:25.377 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.419 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:cfc563e8:::obj-fill-392:head# 2021-01-15T17:00:25.518 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.519 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d02d8880:::obj-fill-362:head# 2021-01-15T17:00:25.519 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.519 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d054947e:::obj-fill-67:head# 2021-01-15T17:00:25.519 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.544 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d061c977:::obj-fill-164:head# 2021-01-15T17:00:25.544 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.586 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d06a5f57:::obj-fill-296:head# 2021-01-15T17:00:25.586 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.627 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d110062a:::obj-fill-343:head# 2021-01-15T17:00:25.627 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.671 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d1b8ad47:::obj-fill-383:head# 2021-01-15T17:00:25.672 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.711 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d1e80db6:::obj-fill-117:head# 2021-01-15T17:00:25.711 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.794 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d210027e:::obj-fill-231:head# 2021-01-15T17:00:25.794 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.836 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d24428ed:::obj-fill-199:head# 2021-01-15T17:00:25.836 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.877 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d27c3c4b:::obj-fill-154:head# 2021-01-15T17:00:25.878 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.927 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d28957e3:::obj-fill-263:head# 2021-01-15T17:00:25.928 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:25.969 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d2d44da7:::obj-fill-544:head# 2021-01-15T17:00:25.970 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.012 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d31004a4:::obj-fill-434:head# 2021-01-15T17:00:26.012 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.075 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d32487c3:::obj-fill-385:head# 2021-01-15T17:00:26.076 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.128 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d340891c:::obj-fill-12:head# 2021-01-15T17:00:26.128 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.169 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d4135728:::obj-fill-310:head# 2021-01-15T17:00:26.170 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.202 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d5a43972:::obj-fill-9:head# 2021-01-15T17:00:26.203 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.244 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d5efeb15:::obj-fill-347:head# 2021-01-15T17:00:26.245 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.286 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d6463f70:::obj-fill-536:head# 2021-01-15T17:00:26.286 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.327 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d666d9b7:::obj-fill-420:head# 2021-01-15T17:00:26.327 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.427 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d6e86688:::obj-fill-166:head# 2021-01-15T17:00:26.428 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.478 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d730370d:::obj-fill-206:head# 2021-01-15T17:00:26.478 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.511 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d783551c:::obj-fill-109:head# 2021-01-15T17:00:26.512 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.545 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d7f77e5f:::obj-fill-551:head# 2021-01-15T17:00:26.545 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.570 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d843056c:::obj-fill-291:head# 2021-01-15T17:00:26.570 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.595 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d84eb173:::obj-fill-221:head# 2021-01-15T17:00:26.595 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.628 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d8dd069c:::obj-fill-60:head# 2021-01-15T17:00:26.629 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.661 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d8f2f750:::obj-fill-562:head# 2021-01-15T17:00:26.662 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.695 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d8f80ac2:::obj-fill-179:head# 2021-01-15T17:00:26.695 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.728 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:d9f904cf:::obj-fill-308:head# 2021-01-15T17:00:26.728 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.761 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:db0a534b:::obj-fill-410:head# 2021-01-15T17:00:26.762 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.795 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dc4403f4:::obj-fill-416:head# 2021-01-15T17:00:26.795 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.828 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dc66c3ca:::obj-fill-220:head# 2021-01-15T17:00:26.828 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.861 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dc8198dc:::obj-fill-596:head# 2021-01-15T17:00:26.862 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.895 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dcbade2b:::obj-fill-10:head# 2021-01-15T17:00:26.895 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.928 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ddb787ea:::obj-fill-556:head# 2021-01-15T17:00:26.928 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.961 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:de58065b:::obj-fill-306:head# 2021-01-15T17:00:26.961 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:26.995 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:deaa6fab:::obj-fill-194:head# 2021-01-15T17:00:26.995 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.070 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dfee7714:::obj-fill-202:head# 2021-01-15T17:00:27.070 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.108 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:dff5df4a:::obj-fill-565:head# 2021-01-15T17:00:27.109 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.162 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e2053442:::obj-fill-107:head# 2021-01-15T17:00:27.162 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.195 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e32055b8:::obj-fill-57:head# 2021-01-15T17:00:27.195 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.228 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e37fac1e:::obj-fill-161:head# 2021-01-15T17:00:27.229 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.261 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e388f5aa:::obj-fill-273:head# 2021-01-15T17:00:27.262 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.295 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e38b1f4d:::obj-fill-393:head# 2021-01-15T17:00:27.295 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.328 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e4792474:::obj-fill-158:head# 2021-01-15T17:00:27.328 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.361 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e4899cb0:::obj-fill-270:head# 2021-01-15T17:00:27.362 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.395 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e62ae9f1:::obj-fill-288:head# 2021-01-15T17:00:27.395 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.428 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e64fd95a:::obj-fill-449:head# 2021-01-15T17:00:27.428 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.461 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e6682cfc:::obj-fill-373:head# 2021-01-15T17:00:27.462 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.495 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e716717b:::obj-fill-178:head# 2021-01-15T17:00:27.495 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.528 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e80226af:::obj-fill-354:head# 2021-01-15T17:00:27.528 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.561 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e83a5940:::obj-fill-564:head# 2021-01-15T17:00:27.562 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.595 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e854bfbe:::obj-fill-589:head# 2021-01-15T17:00:27.595 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.628 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e8b197bc:::obj-fill-89:head# 2021-01-15T17:00:27.628 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.712 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e92fca88:::obj-fill-397:head# 2021-01-15T17:00:27.712 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.745 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e94d8f7b:::obj-fill-261:head# 2021-01-15T17:00:27.745 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.778 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:e98cc0a4:::obj-fill-525:head# 2021-01-15T17:00:27.778 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.811 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ea5a9459:::obj-fill-431:head# 2021-01-15T17:00:27.812 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.846 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ebc54d47:::obj-fill-258:head# 2021-01-15T17:00:27.846 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.878 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ec0002dd:::obj-fill-205:head# 2021-01-15T17:00:27.879 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.920 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ec740b5c:::obj-fill-46:head# 2021-01-15T17:00:27.921 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:27.962 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ed35dda4:::obj-fill-170:head# 2021-01-15T17:00:27.962 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.003 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ed51b8f7:::obj-fill-91:head# 2021-01-15T17:00:28.004 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.057 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:edd96161:::obj-fill-48:head# 2021-01-15T17:00:28.058 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.120 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ee3f0d4b:::obj-fill-47:head# 2021-01-15T17:00:28.120 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.162 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ee4784dd:::obj-fill-480:head# 2021-01-15T17:00:28.162 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.204 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ee5a8271:::obj-fill-198:head# 2021-01-15T17:00:28.204 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.245 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ef745b66:::obj-fill-145:head# 2021-01-15T17:00:28.245 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.332 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:efa5309d:::obj-fill-215:head# 2021-01-15T17:00:28.333 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.434 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:efbdd3a1:::obj-fill-276:head# 2021-01-15T17:00:28.435 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.470 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f059d4c0:::obj-fill-482:head# 2021-01-15T17:00:28.471 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.512 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f0f9fb7a:::obj-fill-379:head# 2021-01-15T17:00:28.512 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.553 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f16a3656:::obj-fill-462:head# 2021-01-15T17:00:28.554 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.603 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f1b395ae:::obj-fill-25:head# 2021-01-15T17:00:28.604 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.645 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5876dad:::obj-fill-359:head# 2021-01-15T17:00:28.645 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.687 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5a283cd:::obj-fill-534:head# 2021-01-15T17:00:28.687 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.729 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5cd905a:::obj-fill-358:head# 2021-01-15T17:00:28.730 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.772 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5e48302:::obj-fill-406:head# 2021-01-15T17:00:28.773 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.812 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f5ea65fc:::obj-fill-52:head# 2021-01-15T17:00:28.813 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.854 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f642de6b:::obj-fill-204:head# 2021-01-15T17:00:28.855 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.887 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f716eafd:::obj-fill-502:head# 2021-01-15T17:00:28.888 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.920 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f752e9cb:::obj-fill-23:head# 2021-01-15T17:00:28.921 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:28.987 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f85fe42c:::obj-fill-346:head# 2021-01-15T17:00:28.987 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.021 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:f882fd3b:::obj-fill-355:head# 2021-01-15T17:00:29.021 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.054 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fa133332:::obj-fill-309:head# 2021-01-15T17:00:29.055 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.087 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fa875e52:::obj-fill-64:head# 2021-01-15T17:00:29.087 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.121 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fad91a22:::obj-fill-481:head# 2021-01-15T17:00:29.121 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.154 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fbbc3122:::obj-fill-111:head# 2021-01-15T17:00:29.155 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.188 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fc539ddb:::obj-fill-181:head# 2021-01-15T17:00:29.188 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.222 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fc54128d:::obj-fill-104:head# 2021-01-15T17:00:29.222 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.245 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fe00aded:::obj-fill-365:head# 2021-01-15T17:00:29.246 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.279 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fe1e9971:::obj-fill-189:head# 2021-01-15T17:00:29.280 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.312 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ff0e0563:::obj-fill-508:head# 2021-01-15T17:00:29.312 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.346 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ff4e156c:::obj-fill-441:head# 2021-01-15T17:00:29.346 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.379 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:ffa355f7:::obj-fill-500:head# 2021-01-15T17:00:29.379 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.412 INFO:tasks.workunit.client.0.smithi145.stdout:Write #1:fff6c93b:::obj-fill-277:head# 2021-01-15T17:00:29.413 INFO:tasks.workunit.client.0.smithi145.stdout:snapset 0=[]:{} 2021-01-15T17:00:29.446 INFO:tasks.workunit.client.0.smithi145.stdout:write_pg epoch 35 info 1.0( v 35'600 (35'500,35'600] local-lis/les=24/25 n=600 ec=21/21 lis/c=24/24 les/c/f=25/25/0 sis=24) 2021-01-15T17:00:29.487 INFO:tasks.workunit.client.0.smithi145.stdout:Import successful 2021-01-15T17:00:30.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:494: TEST_backfill_multi_partial: ceph pg dump pgs 2021-01-15T17:00:30.394 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:00:30.394 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:59:52.264519+0000 0'0 45:36 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:07.000190+0000 0'0 2021-01-15T16:57:07.000190+0000 0 2021-01-15T17:00:30.394 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 554 0 0 0 0 1446494 0 0 54 54 stale+active+clean 2021-01-15T16:57:04.288669+0000 35'554 35:582 [1] 1 [1] 1 0'0 2021-01-15T16:57:01.996196+0000 0'0 2021-01-15T16:57:01.996196+0000 0 2021-01-15T17:00:30.395 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 600 0 0 0 0 614400 0 0 100 100 active+clean 2021-01-15T16:59:52.230906+0000 40'600 45:656 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:04.485874+0000 0'0 2021-01-15T16:57:04.485874+0000 0 2021-01-15T17:00:30.395 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:00:30.395 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:00:30.396 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:00:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:495: TEST_backfill_multi_partial: sleep 20 2021-01-15T17:00:50.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:496: TEST_backfill_multi_partial: ceph pg dump pgs 2021-01-15T17:00:50.784 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:00:50.784 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T16:59:52.264519+0000 0'0 45:36 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:07.000190+0000 0'0 2021-01-15T16:57:07.000190+0000 0 2021-01-15T17:00:50.785 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 554 0 0 0 0 1446494 0 0 54 54 stale+active+clean 2021-01-15T16:57:04.288669+0000 35'554 35:582 [1] 1 [1] 1 0'0 2021-01-15T16:57:01.996196+0000 0'0 2021-01-15T16:57:01.996196+0000 0 2021-01-15T17:00:50.785 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 600 0 0 0 0 614400 0 0 100 100 active+clean 2021-01-15T16:59:52.230906+0000 40'600 45:656 [2,0] 2 [2,0] 2 0'0 2021-01-15T16:57:04.485874+0000 0'0 2021-01-15T16:57:04.485874+0000 0 2021-01-15T17:00:50.785 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:00:50.785 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:00:50.786 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:00:50.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:499: TEST_backfill_multi_partial: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=2611 count=1 2021-01-15T17:00:50.796 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T17:00:50.796 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T17:00:50.797 INFO:tasks.workunit.client.0.smithi145.stderr:2611 bytes (2.6 kB, 2.5 KiB) copied, 8.7122e-05 s, 30.0 MB/s 2021-01-15T17:00:50.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: seq 1 600 2021-01-15T17:00:50.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:50.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:50.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:50.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-1 td/osd-backfill-space/datafile 2021-01-15T17:00:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-1 td/osd-backfill-space/datafile 2021-01-15T17:00:51.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:51.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-2 td/osd-backfill-space/datafile 2021-01-15T17:00:51.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-2 td/osd-backfill-space/datafile 2021-01-15T17:00:51.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:51.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-3 td/osd-backfill-space/datafile 2021-01-15T17:00:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-3 td/osd-backfill-space/datafile 2021-01-15T17:00:51.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:51.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:51.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-4 td/osd-backfill-space/datafile 2021-01-15T17:00:51.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-4 td/osd-backfill-space/datafile 2021-01-15T17:00:51.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:51.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:51.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-5 td/osd-backfill-space/datafile 2021-01-15T17:00:51.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-5 td/osd-backfill-space/datafile 2021-01-15T17:00:52.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:52.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-6 td/osd-backfill-space/datafile 2021-01-15T17:00:52.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-6 td/osd-backfill-space/datafile 2021-01-15T17:00:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:52.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:52.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-7 td/osd-backfill-space/datafile 2021-01-15T17:00:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-7 td/osd-backfill-space/datafile 2021-01-15T17:00:52.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:52.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:52.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-8 td/osd-backfill-space/datafile 2021-01-15T17:00:52.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-8 td/osd-backfill-space/datafile 2021-01-15T17:00:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:52.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-9 td/osd-backfill-space/datafile 2021-01-15T17:00:52.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:52.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-9 td/osd-backfill-space/datafile 2021-01-15T17:00:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:53.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:53.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-10 td/osd-backfill-space/datafile 2021-01-15T17:00:53.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-10 td/osd-backfill-space/datafile 2021-01-15T17:00:53.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:53.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:53.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-11 td/osd-backfill-space/datafile 2021-01-15T17:00:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-11 td/osd-backfill-space/datafile 2021-01-15T17:00:53.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:53.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:53.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-12 td/osd-backfill-space/datafile 2021-01-15T17:00:53.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-12 td/osd-backfill-space/datafile 2021-01-15T17:00:53.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:53.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:53.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-13 td/osd-backfill-space/datafile 2021-01-15T17:00:53.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:53.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-13 td/osd-backfill-space/datafile 2021-01-15T17:00:54.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:54.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:54.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-14 td/osd-backfill-space/datafile 2021-01-15T17:00:54.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-14 td/osd-backfill-space/datafile 2021-01-15T17:00:54.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:54.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:54.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-15 td/osd-backfill-space/datafile 2021-01-15T17:00:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-15 td/osd-backfill-space/datafile 2021-01-15T17:00:54.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:54.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-16 td/osd-backfill-space/datafile 2021-01-15T17:00:54.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-16 td/osd-backfill-space/datafile 2021-01-15T17:00:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:54.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-17 td/osd-backfill-space/datafile 2021-01-15T17:00:54.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:54.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-17 td/osd-backfill-space/datafile 2021-01-15T17:00:55.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:55.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-18 td/osd-backfill-space/datafile 2021-01-15T17:00:55.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-18 td/osd-backfill-space/datafile 2021-01-15T17:00:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:55.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:55.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-19 td/osd-backfill-space/datafile 2021-01-15T17:00:55.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-19 td/osd-backfill-space/datafile 2021-01-15T17:00:55.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:55.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-20 td/osd-backfill-space/datafile 2021-01-15T17:00:55.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-20 td/osd-backfill-space/datafile 2021-01-15T17:00:55.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:55.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:55.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-21 td/osd-backfill-space/datafile 2021-01-15T17:00:55.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-21 td/osd-backfill-space/datafile 2021-01-15T17:00:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:55.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:55.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:55.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-22 td/osd-backfill-space/datafile 2021-01-15T17:00:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-22 td/osd-backfill-space/datafile 2021-01-15T17:00:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:56.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-23 td/osd-backfill-space/datafile 2021-01-15T17:00:56.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-23 td/osd-backfill-space/datafile 2021-01-15T17:00:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:56.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:56.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-24 td/osd-backfill-space/datafile 2021-01-15T17:00:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-24 td/osd-backfill-space/datafile 2021-01-15T17:00:56.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:56.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:56.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-25 td/osd-backfill-space/datafile 2021-01-15T17:00:56.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:56.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-25 td/osd-backfill-space/datafile 2021-01-15T17:00:57.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:57.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:57.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-26 td/osd-backfill-space/datafile 2021-01-15T17:00:57.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-26 td/osd-backfill-space/datafile 2021-01-15T17:00:57.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:57.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:57.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-27 td/osd-backfill-space/datafile 2021-01-15T17:00:57.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-27 td/osd-backfill-space/datafile 2021-01-15T17:00:57.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:57.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-28 td/osd-backfill-space/datafile 2021-01-15T17:00:57.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-28 td/osd-backfill-space/datafile 2021-01-15T17:00:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:57.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-29 td/osd-backfill-space/datafile 2021-01-15T17:00:57.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:57.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-29 td/osd-backfill-space/datafile 2021-01-15T17:00:58.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:58.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-30 td/osd-backfill-space/datafile 2021-01-15T17:00:58.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-30 td/osd-backfill-space/datafile 2021-01-15T17:00:58.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:58.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:58.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-31 td/osd-backfill-space/datafile 2021-01-15T17:00:58.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-31 td/osd-backfill-space/datafile 2021-01-15T17:00:58.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:58.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-32 td/osd-backfill-space/datafile 2021-01-15T17:00:58.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-32 td/osd-backfill-space/datafile 2021-01-15T17:00:58.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:58.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:58.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-33 td/osd-backfill-space/datafile 2021-01-15T17:00:58.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:58.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-33 td/osd-backfill-space/datafile 2021-01-15T17:00:59.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:59.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:59.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-34 td/osd-backfill-space/datafile 2021-01-15T17:00:59.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-34 td/osd-backfill-space/datafile 2021-01-15T17:00:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:59.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:59.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-35 td/osd-backfill-space/datafile 2021-01-15T17:00:59.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-35 td/osd-backfill-space/datafile 2021-01-15T17:00:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-36 td/osd-backfill-space/datafile 2021-01-15T17:00:59.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-36 td/osd-backfill-space/datafile 2021-01-15T17:00:59.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:59.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-37 td/osd-backfill-space/datafile 2021-01-15T17:00:59.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-37 td/osd-backfill-space/datafile 2021-01-15T17:00:59.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:00:59.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:00:59.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:00:59.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-38 td/osd-backfill-space/datafile 2021-01-15T17:01:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-38 td/osd-backfill-space/datafile 2021-01-15T17:01:00.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:00.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:00.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-39 td/osd-backfill-space/datafile 2021-01-15T17:01:00.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-39 td/osd-backfill-space/datafile 2021-01-15T17:01:00.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:00.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-40 td/osd-backfill-space/datafile 2021-01-15T17:01:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-40 td/osd-backfill-space/datafile 2021-01-15T17:01:00.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:00.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:00.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-41 td/osd-backfill-space/datafile 2021-01-15T17:01:00.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-41 td/osd-backfill-space/datafile 2021-01-15T17:01:00.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:00.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:00.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:00.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-42 td/osd-backfill-space/datafile 2021-01-15T17:01:01.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-42 td/osd-backfill-space/datafile 2021-01-15T17:01:01.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:01.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:01.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-43 td/osd-backfill-space/datafile 2021-01-15T17:01:01.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-43 td/osd-backfill-space/datafile 2021-01-15T17:01:01.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:01.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:01.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-44 td/osd-backfill-space/datafile 2021-01-15T17:01:01.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-44 td/osd-backfill-space/datafile 2021-01-15T17:01:01.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:01.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:01.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-45 td/osd-backfill-space/datafile 2021-01-15T17:01:01.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-45 td/osd-backfill-space/datafile 2021-01-15T17:01:01.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:01.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-46 td/osd-backfill-space/datafile 2021-01-15T17:01:02.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-46 td/osd-backfill-space/datafile 2021-01-15T17:01:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:02.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-47 td/osd-backfill-space/datafile 2021-01-15T17:01:02.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-47 td/osd-backfill-space/datafile 2021-01-15T17:01:02.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:02.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:02.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-48 td/osd-backfill-space/datafile 2021-01-15T17:01:02.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-48 td/osd-backfill-space/datafile 2021-01-15T17:01:02.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:02.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:02.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-49 td/osd-backfill-space/datafile 2021-01-15T17:01:02.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-49 td/osd-backfill-space/datafile 2021-01-15T17:01:02.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:02.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:02.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:02.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-50 td/osd-backfill-space/datafile 2021-01-15T17:01:03.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-50 td/osd-backfill-space/datafile 2021-01-15T17:01:03.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:03.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:03.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-51 td/osd-backfill-space/datafile 2021-01-15T17:01:03.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-51 td/osd-backfill-space/datafile 2021-01-15T17:01:03.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:03.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:03.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-52 td/osd-backfill-space/datafile 2021-01-15T17:01:03.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-52 td/osd-backfill-space/datafile 2021-01-15T17:01:03.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:03.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:03.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-53 td/osd-backfill-space/datafile 2021-01-15T17:01:03.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-53 td/osd-backfill-space/datafile 2021-01-15T17:01:03.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:03.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:03.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-54 td/osd-backfill-space/datafile 2021-01-15T17:01:03.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:03.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-54 td/osd-backfill-space/datafile 2021-01-15T17:01:04.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:04.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:04.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-55 td/osd-backfill-space/datafile 2021-01-15T17:01:04.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-55 td/osd-backfill-space/datafile 2021-01-15T17:01:04.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:04.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-56 td/osd-backfill-space/datafile 2021-01-15T17:01:04.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-56 td/osd-backfill-space/datafile 2021-01-15T17:01:04.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:04.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:04.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-57 td/osd-backfill-space/datafile 2021-01-15T17:01:04.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-57 td/osd-backfill-space/datafile 2021-01-15T17:01:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:04.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:04.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-58 td/osd-backfill-space/datafile 2021-01-15T17:01:04.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:04.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-58 td/osd-backfill-space/datafile 2021-01-15T17:01:05.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:05.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-59 td/osd-backfill-space/datafile 2021-01-15T17:01:05.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-59 td/osd-backfill-space/datafile 2021-01-15T17:01:05.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:05.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:05.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-60 td/osd-backfill-space/datafile 2021-01-15T17:01:05.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-60 td/osd-backfill-space/datafile 2021-01-15T17:01:05.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:05.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:05.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-61 td/osd-backfill-space/datafile 2021-01-15T17:01:05.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-61 td/osd-backfill-space/datafile 2021-01-15T17:01:05.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:05.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:05.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-62 td/osd-backfill-space/datafile 2021-01-15T17:01:05.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-62 td/osd-backfill-space/datafile 2021-01-15T17:01:05.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:05.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:05.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:05.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-63 td/osd-backfill-space/datafile 2021-01-15T17:01:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-63 td/osd-backfill-space/datafile 2021-01-15T17:01:06.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:06.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:06.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-64 td/osd-backfill-space/datafile 2021-01-15T17:01:06.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-64 td/osd-backfill-space/datafile 2021-01-15T17:01:06.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:06.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:06.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-65 td/osd-backfill-space/datafile 2021-01-15T17:01:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-65 td/osd-backfill-space/datafile 2021-01-15T17:01:06.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:06.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:06.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-66 td/osd-backfill-space/datafile 2021-01-15T17:01:06.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-66 td/osd-backfill-space/datafile 2021-01-15T17:01:06.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:06.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:06.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:06.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-67 td/osd-backfill-space/datafile 2021-01-15T17:01:07.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-67 td/osd-backfill-space/datafile 2021-01-15T17:01:07.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:07.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:07.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-68 td/osd-backfill-space/datafile 2021-01-15T17:01:07.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-68 td/osd-backfill-space/datafile 2021-01-15T17:01:07.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:07.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-69 td/osd-backfill-space/datafile 2021-01-15T17:01:07.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-69 td/osd-backfill-space/datafile 2021-01-15T17:01:07.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:07.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:07.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-70 td/osd-backfill-space/datafile 2021-01-15T17:01:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-70 td/osd-backfill-space/datafile 2021-01-15T17:01:07.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:07.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-71 td/osd-backfill-space/datafile 2021-01-15T17:01:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-71 td/osd-backfill-space/datafile 2021-01-15T17:01:08.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:08.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:08.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-72 td/osd-backfill-space/datafile 2021-01-15T17:01:08.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-72 td/osd-backfill-space/datafile 2021-01-15T17:01:08.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:08.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-73 td/osd-backfill-space/datafile 2021-01-15T17:01:08.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-73 td/osd-backfill-space/datafile 2021-01-15T17:01:08.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:08.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:08.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-74 td/osd-backfill-space/datafile 2021-01-15T17:01:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-74 td/osd-backfill-space/datafile 2021-01-15T17:01:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:08.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-75 td/osd-backfill-space/datafile 2021-01-15T17:01:08.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:08.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-75 td/osd-backfill-space/datafile 2021-01-15T17:01:09.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:09.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:09.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-76 td/osd-backfill-space/datafile 2021-01-15T17:01:09.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-76 td/osd-backfill-space/datafile 2021-01-15T17:01:09.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:09.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:09.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-77 td/osd-backfill-space/datafile 2021-01-15T17:01:09.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-77 td/osd-backfill-space/datafile 2021-01-15T17:01:09.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:09.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:09.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-78 td/osd-backfill-space/datafile 2021-01-15T17:01:09.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-78 td/osd-backfill-space/datafile 2021-01-15T17:01:09.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:09.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:09.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-79 td/osd-backfill-space/datafile 2021-01-15T17:01:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:09.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-79 td/osd-backfill-space/datafile 2021-01-15T17:01:10.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:10.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:10.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-80 td/osd-backfill-space/datafile 2021-01-15T17:01:10.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-80 td/osd-backfill-space/datafile 2021-01-15T17:01:10.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:10.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:10.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-81 td/osd-backfill-space/datafile 2021-01-15T17:01:10.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-81 td/osd-backfill-space/datafile 2021-01-15T17:01:10.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:10.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:10.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-82 td/osd-backfill-space/datafile 2021-01-15T17:01:10.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-82 td/osd-backfill-space/datafile 2021-01-15T17:01:10.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:10.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:10.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:10.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-83 td/osd-backfill-space/datafile 2021-01-15T17:01:11.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-83 td/osd-backfill-space/datafile 2021-01-15T17:01:11.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:11.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:11.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-84 td/osd-backfill-space/datafile 2021-01-15T17:01:11.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-84 td/osd-backfill-space/datafile 2021-01-15T17:01:11.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:11.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:11.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-85 td/osd-backfill-space/datafile 2021-01-15T17:01:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-85 td/osd-backfill-space/datafile 2021-01-15T17:01:11.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:11.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:11.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-86 td/osd-backfill-space/datafile 2021-01-15T17:01:11.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-86 td/osd-backfill-space/datafile 2021-01-15T17:01:11.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:11.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:11.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:11.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-87 td/osd-backfill-space/datafile 2021-01-15T17:01:12.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-87 td/osd-backfill-space/datafile 2021-01-15T17:01:12.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:12.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:12.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-88 td/osd-backfill-space/datafile 2021-01-15T17:01:12.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-88 td/osd-backfill-space/datafile 2021-01-15T17:01:12.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:12.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:12.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-89 td/osd-backfill-space/datafile 2021-01-15T17:01:12.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-89 td/osd-backfill-space/datafile 2021-01-15T17:01:12.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:12.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:12.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-90 td/osd-backfill-space/datafile 2021-01-15T17:01:12.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-90 td/osd-backfill-space/datafile 2021-01-15T17:01:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:12.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:12.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:12.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-91 td/osd-backfill-space/datafile 2021-01-15T17:01:13.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-91 td/osd-backfill-space/datafile 2021-01-15T17:01:13.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:13.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:13.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-92 td/osd-backfill-space/datafile 2021-01-15T17:01:13.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-92 td/osd-backfill-space/datafile 2021-01-15T17:01:13.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:13.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:13.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-93 td/osd-backfill-space/datafile 2021-01-15T17:01:13.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-93 td/osd-backfill-space/datafile 2021-01-15T17:01:13.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:13.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-94 td/osd-backfill-space/datafile 2021-01-15T17:01:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-94 td/osd-backfill-space/datafile 2021-01-15T17:01:13.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:13.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:13.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:13.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-95 td/osd-backfill-space/datafile 2021-01-15T17:01:14.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-95 td/osd-backfill-space/datafile 2021-01-15T17:01:14.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:14.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:14.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-96 td/osd-backfill-space/datafile 2021-01-15T17:01:14.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-96 td/osd-backfill-space/datafile 2021-01-15T17:01:14.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:14.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:14.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-97 td/osd-backfill-space/datafile 2021-01-15T17:01:14.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-97 td/osd-backfill-space/datafile 2021-01-15T17:01:14.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:14.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:14.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-98 td/osd-backfill-space/datafile 2021-01-15T17:01:14.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-98 td/osd-backfill-space/datafile 2021-01-15T17:01:14.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:14.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:14.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-99 td/osd-backfill-space/datafile 2021-01-15T17:01:14.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:14.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-99 td/osd-backfill-space/datafile 2021-01-15T17:01:15.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:15.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:15.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-100 td/osd-backfill-space/datafile 2021-01-15T17:01:15.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-100 td/osd-backfill-space/datafile 2021-01-15T17:01:15.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:15.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:15.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-101 td/osd-backfill-space/datafile 2021-01-15T17:01:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-101 td/osd-backfill-space/datafile 2021-01-15T17:01:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:15.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:15.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-102 td/osd-backfill-space/datafile 2021-01-15T17:01:15.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-102 td/osd-backfill-space/datafile 2021-01-15T17:01:15.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:15.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:15.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-103 td/osd-backfill-space/datafile 2021-01-15T17:01:16.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-103 td/osd-backfill-space/datafile 2021-01-15T17:01:16.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:16.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-104 td/osd-backfill-space/datafile 2021-01-15T17:01:16.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-104 td/osd-backfill-space/datafile 2021-01-15T17:01:16.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:16.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-105 td/osd-backfill-space/datafile 2021-01-15T17:01:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-105 td/osd-backfill-space/datafile 2021-01-15T17:01:16.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:16.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:16.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-106 td/osd-backfill-space/datafile 2021-01-15T17:01:16.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:16.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-106 td/osd-backfill-space/datafile 2021-01-15T17:01:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:17.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-107 td/osd-backfill-space/datafile 2021-01-15T17:01:17.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-107 td/osd-backfill-space/datafile 2021-01-15T17:01:17.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:17.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:17.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-108 td/osd-backfill-space/datafile 2021-01-15T17:01:17.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-108 td/osd-backfill-space/datafile 2021-01-15T17:01:17.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:17.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-109 td/osd-backfill-space/datafile 2021-01-15T17:01:17.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-109 td/osd-backfill-space/datafile 2021-01-15T17:01:17.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:17.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:17.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-110 td/osd-backfill-space/datafile 2021-01-15T17:01:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-110 td/osd-backfill-space/datafile 2021-01-15T17:01:17.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:17.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-111 td/osd-backfill-space/datafile 2021-01-15T17:01:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-111 td/osd-backfill-space/datafile 2021-01-15T17:01:18.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:18.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-112 td/osd-backfill-space/datafile 2021-01-15T17:01:18.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-112 td/osd-backfill-space/datafile 2021-01-15T17:01:18.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:18.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:18.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-113 td/osd-backfill-space/datafile 2021-01-15T17:01:18.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-113 td/osd-backfill-space/datafile 2021-01-15T17:01:18.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:18.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-114 td/osd-backfill-space/datafile 2021-01-15T17:01:18.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-114 td/osd-backfill-space/datafile 2021-01-15T17:01:18.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:18.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:18.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:18.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-115 td/osd-backfill-space/datafile 2021-01-15T17:01:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-115 td/osd-backfill-space/datafile 2021-01-15T17:01:19.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:19.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:19.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-116 td/osd-backfill-space/datafile 2021-01-15T17:01:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-116 td/osd-backfill-space/datafile 2021-01-15T17:01:19.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:19.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:19.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-117 td/osd-backfill-space/datafile 2021-01-15T17:01:19.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-117 td/osd-backfill-space/datafile 2021-01-15T17:01:19.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:19.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:19.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-118 td/osd-backfill-space/datafile 2021-01-15T17:01:19.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-118 td/osd-backfill-space/datafile 2021-01-15T17:01:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:19.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-119 td/osd-backfill-space/datafile 2021-01-15T17:01:19.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:19.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-119 td/osd-backfill-space/datafile 2021-01-15T17:01:20.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:20.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:20.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-120 td/osd-backfill-space/datafile 2021-01-15T17:01:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-120 td/osd-backfill-space/datafile 2021-01-15T17:01:20.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:20.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:20.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-121 td/osd-backfill-space/datafile 2021-01-15T17:01:20.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-121 td/osd-backfill-space/datafile 2021-01-15T17:01:20.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:20.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:20.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-122 td/osd-backfill-space/datafile 2021-01-15T17:01:20.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-122 td/osd-backfill-space/datafile 2021-01-15T17:01:20.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:20.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:20.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-123 td/osd-backfill-space/datafile 2021-01-15T17:01:20.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:20.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-123 td/osd-backfill-space/datafile 2021-01-15T17:01:20.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:20.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:21.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-124 td/osd-backfill-space/datafile 2021-01-15T17:01:21.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-124 td/osd-backfill-space/datafile 2021-01-15T17:01:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:21.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-125 td/osd-backfill-space/datafile 2021-01-15T17:01:21.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-125 td/osd-backfill-space/datafile 2021-01-15T17:01:21.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:21.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:21.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-126 td/osd-backfill-space/datafile 2021-01-15T17:01:21.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-126 td/osd-backfill-space/datafile 2021-01-15T17:01:21.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:21.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-127 td/osd-backfill-space/datafile 2021-01-15T17:01:21.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-127 td/osd-backfill-space/datafile 2021-01-15T17:01:21.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:21.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:21.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:21.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-128 td/osd-backfill-space/datafile 2021-01-15T17:01:22.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-128 td/osd-backfill-space/datafile 2021-01-15T17:01:22.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:22.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:22.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-129 td/osd-backfill-space/datafile 2021-01-15T17:01:22.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-129 td/osd-backfill-space/datafile 2021-01-15T17:01:22.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:22.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:22.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-130 td/osd-backfill-space/datafile 2021-01-15T17:01:22.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-130 td/osd-backfill-space/datafile 2021-01-15T17:01:22.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:22.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:22.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-131 td/osd-backfill-space/datafile 2021-01-15T17:01:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-131 td/osd-backfill-space/datafile 2021-01-15T17:01:22.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:22.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:22.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:22.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-132 td/osd-backfill-space/datafile 2021-01-15T17:01:23.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-132 td/osd-backfill-space/datafile 2021-01-15T17:01:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:23.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:23.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-133 td/osd-backfill-space/datafile 2021-01-15T17:01:23.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-133 td/osd-backfill-space/datafile 2021-01-15T17:01:23.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:23.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:23.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-134 td/osd-backfill-space/datafile 2021-01-15T17:01:23.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-134 td/osd-backfill-space/datafile 2021-01-15T17:01:23.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:23.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:23.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-135 td/osd-backfill-space/datafile 2021-01-15T17:01:23.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-135 td/osd-backfill-space/datafile 2021-01-15T17:01:23.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:23.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:23.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:23.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-136 td/osd-backfill-space/datafile 2021-01-15T17:01:24.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-136 td/osd-backfill-space/datafile 2021-01-15T17:01:24.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:24.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:24.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-137 td/osd-backfill-space/datafile 2021-01-15T17:01:24.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-137 td/osd-backfill-space/datafile 2021-01-15T17:01:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:24.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:24.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-138 td/osd-backfill-space/datafile 2021-01-15T17:01:24.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-138 td/osd-backfill-space/datafile 2021-01-15T17:01:24.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:24.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:24.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-139 td/osd-backfill-space/datafile 2021-01-15T17:01:24.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-139 td/osd-backfill-space/datafile 2021-01-15T17:01:24.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:24.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:24.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:24.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-140 td/osd-backfill-space/datafile 2021-01-15T17:01:25.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-140 td/osd-backfill-space/datafile 2021-01-15T17:01:25.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:25.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:25.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-141 td/osd-backfill-space/datafile 2021-01-15T17:01:25.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-141 td/osd-backfill-space/datafile 2021-01-15T17:01:25.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:25.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-142 td/osd-backfill-space/datafile 2021-01-15T17:01:25.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-142 td/osd-backfill-space/datafile 2021-01-15T17:01:25.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:25.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:25.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-143 td/osd-backfill-space/datafile 2021-01-15T17:01:25.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-143 td/osd-backfill-space/datafile 2021-01-15T17:01:25.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:25.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:25.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:25.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-144 td/osd-backfill-space/datafile 2021-01-15T17:01:26.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-144 td/osd-backfill-space/datafile 2021-01-15T17:01:26.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:26.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:26.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-145 td/osd-backfill-space/datafile 2021-01-15T17:01:26.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-145 td/osd-backfill-space/datafile 2021-01-15T17:01:26.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:26.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:26.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-146 td/osd-backfill-space/datafile 2021-01-15T17:01:26.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-146 td/osd-backfill-space/datafile 2021-01-15T17:01:26.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:26.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:26.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-147 td/osd-backfill-space/datafile 2021-01-15T17:01:26.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-147 td/osd-backfill-space/datafile 2021-01-15T17:01:26.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:26.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:26.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:26.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-148 td/osd-backfill-space/datafile 2021-01-15T17:01:27.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-148 td/osd-backfill-space/datafile 2021-01-15T17:01:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:27.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:27.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-149 td/osd-backfill-space/datafile 2021-01-15T17:01:27.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-149 td/osd-backfill-space/datafile 2021-01-15T17:01:27.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:27.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:27.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-150 td/osd-backfill-space/datafile 2021-01-15T17:01:27.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-150 td/osd-backfill-space/datafile 2021-01-15T17:01:27.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:27.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:27.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-151 td/osd-backfill-space/datafile 2021-01-15T17:01:27.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-151 td/osd-backfill-space/datafile 2021-01-15T17:01:27.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:27.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-152 td/osd-backfill-space/datafile 2021-01-15T17:01:28.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-152 td/osd-backfill-space/datafile 2021-01-15T17:01:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:28.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-153 td/osd-backfill-space/datafile 2021-01-15T17:01:28.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-153 td/osd-backfill-space/datafile 2021-01-15T17:01:28.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:28.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:28.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-154 td/osd-backfill-space/datafile 2021-01-15T17:01:28.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-154 td/osd-backfill-space/datafile 2021-01-15T17:01:28.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:28.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-155 td/osd-backfill-space/datafile 2021-01-15T17:01:28.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-155 td/osd-backfill-space/datafile 2021-01-15T17:01:28.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:28.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:28.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-156 td/osd-backfill-space/datafile 2021-01-15T17:01:29.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-156 td/osd-backfill-space/datafile 2021-01-15T17:01:29.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:29.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-157 td/osd-backfill-space/datafile 2021-01-15T17:01:29.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-157 td/osd-backfill-space/datafile 2021-01-15T17:01:29.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:29.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:29.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-158 td/osd-backfill-space/datafile 2021-01-15T17:01:29.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-158 td/osd-backfill-space/datafile 2021-01-15T17:01:29.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:29.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:29.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-159 td/osd-backfill-space/datafile 2021-01-15T17:01:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-159 td/osd-backfill-space/datafile 2021-01-15T17:01:29.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:29.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-160 td/osd-backfill-space/datafile 2021-01-15T17:01:30.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-160 td/osd-backfill-space/datafile 2021-01-15T17:01:30.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:30.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:30.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-161 td/osd-backfill-space/datafile 2021-01-15T17:01:30.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-161 td/osd-backfill-space/datafile 2021-01-15T17:01:30.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:30.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:30.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-162 td/osd-backfill-space/datafile 2021-01-15T17:01:30.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-162 td/osd-backfill-space/datafile 2021-01-15T17:01:30.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:30.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:30.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-163 td/osd-backfill-space/datafile 2021-01-15T17:01:30.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-163 td/osd-backfill-space/datafile 2021-01-15T17:01:30.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:30.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:30.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-164 td/osd-backfill-space/datafile 2021-01-15T17:01:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-164 td/osd-backfill-space/datafile 2021-01-15T17:01:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:31.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-165 td/osd-backfill-space/datafile 2021-01-15T17:01:31.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-165 td/osd-backfill-space/datafile 2021-01-15T17:01:31.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:31.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:31.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-166 td/osd-backfill-space/datafile 2021-01-15T17:01:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-166 td/osd-backfill-space/datafile 2021-01-15T17:01:31.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:31.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:31.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-167 td/osd-backfill-space/datafile 2021-01-15T17:01:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:31.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-167 td/osd-backfill-space/datafile 2021-01-15T17:01:32.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:32.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:32.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-168 td/osd-backfill-space/datafile 2021-01-15T17:01:32.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-168 td/osd-backfill-space/datafile 2021-01-15T17:01:32.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:32.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-169 td/osd-backfill-space/datafile 2021-01-15T17:01:32.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-169 td/osd-backfill-space/datafile 2021-01-15T17:01:32.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:32.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:32.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-170 td/osd-backfill-space/datafile 2021-01-15T17:01:32.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-170 td/osd-backfill-space/datafile 2021-01-15T17:01:32.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:32.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:32.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-171 td/osd-backfill-space/datafile 2021-01-15T17:01:32.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:32.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-171 td/osd-backfill-space/datafile 2021-01-15T17:01:33.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:33.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:33.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-172 td/osd-backfill-space/datafile 2021-01-15T17:01:33.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-172 td/osd-backfill-space/datafile 2021-01-15T17:01:33.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:33.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:33.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-173 td/osd-backfill-space/datafile 2021-01-15T17:01:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-173 td/osd-backfill-space/datafile 2021-01-15T17:01:33.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:33.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-174 td/osd-backfill-space/datafile 2021-01-15T17:01:33.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-174 td/osd-backfill-space/datafile 2021-01-15T17:01:33.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:33.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:33.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-175 td/osd-backfill-space/datafile 2021-01-15T17:01:33.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:33.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-175 td/osd-backfill-space/datafile 2021-01-15T17:01:34.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:34.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-176 td/osd-backfill-space/datafile 2021-01-15T17:01:34.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-176 td/osd-backfill-space/datafile 2021-01-15T17:01:34.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:34.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:34.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-177 td/osd-backfill-space/datafile 2021-01-15T17:01:34.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-177 td/osd-backfill-space/datafile 2021-01-15T17:01:34.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:34.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:34.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-178 td/osd-backfill-space/datafile 2021-01-15T17:01:34.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-178 td/osd-backfill-space/datafile 2021-01-15T17:01:34.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:34.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:34.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-179 td/osd-backfill-space/datafile 2021-01-15T17:01:34.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:34.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-179 td/osd-backfill-space/datafile 2021-01-15T17:01:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:35.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-180 td/osd-backfill-space/datafile 2021-01-15T17:01:35.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-180 td/osd-backfill-space/datafile 2021-01-15T17:01:35.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:35.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:35.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-181 td/osd-backfill-space/datafile 2021-01-15T17:01:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-181 td/osd-backfill-space/datafile 2021-01-15T17:01:35.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:35.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:35.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-182 td/osd-backfill-space/datafile 2021-01-15T17:01:35.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-182 td/osd-backfill-space/datafile 2021-01-15T17:01:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:35.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-183 td/osd-backfill-space/datafile 2021-01-15T17:01:35.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:35.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-183 td/osd-backfill-space/datafile 2021-01-15T17:01:36.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:36.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:36.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-184 td/osd-backfill-space/datafile 2021-01-15T17:01:36.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-184 td/osd-backfill-space/datafile 2021-01-15T17:01:36.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:36.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:36.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-185 td/osd-backfill-space/datafile 2021-01-15T17:01:36.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-185 td/osd-backfill-space/datafile 2021-01-15T17:01:36.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:36.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:36.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-186 td/osd-backfill-space/datafile 2021-01-15T17:01:36.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-186 td/osd-backfill-space/datafile 2021-01-15T17:01:36.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:36.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:36.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-187 td/osd-backfill-space/datafile 2021-01-15T17:01:36.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:36.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-187 td/osd-backfill-space/datafile 2021-01-15T17:01:37.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:37.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:37.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-188 td/osd-backfill-space/datafile 2021-01-15T17:01:37.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-188 td/osd-backfill-space/datafile 2021-01-15T17:01:37.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:37.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-189 td/osd-backfill-space/datafile 2021-01-15T17:01:37.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-189 td/osd-backfill-space/datafile 2021-01-15T17:01:37.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:37.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-190 td/osd-backfill-space/datafile 2021-01-15T17:01:37.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-190 td/osd-backfill-space/datafile 2021-01-15T17:01:37.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:37.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:37.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-191 td/osd-backfill-space/datafile 2021-01-15T17:01:37.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-191 td/osd-backfill-space/datafile 2021-01-15T17:01:37.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:37.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:37.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:37.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-192 td/osd-backfill-space/datafile 2021-01-15T17:01:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-192 td/osd-backfill-space/datafile 2021-01-15T17:01:38.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:38.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:38.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-193 td/osd-backfill-space/datafile 2021-01-15T17:01:38.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-193 td/osd-backfill-space/datafile 2021-01-15T17:01:38.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:38.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:38.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-194 td/osd-backfill-space/datafile 2021-01-15T17:01:38.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-194 td/osd-backfill-space/datafile 2021-01-15T17:01:38.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:38.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:38.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-195 td/osd-backfill-space/datafile 2021-01-15T17:01:38.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-195 td/osd-backfill-space/datafile 2021-01-15T17:01:38.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:38.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:38.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:38.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-196 td/osd-backfill-space/datafile 2021-01-15T17:01:39.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-196 td/osd-backfill-space/datafile 2021-01-15T17:01:39.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:39.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:39.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-197 td/osd-backfill-space/datafile 2021-01-15T17:01:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-197 td/osd-backfill-space/datafile 2021-01-15T17:01:39.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:39.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-198 td/osd-backfill-space/datafile 2021-01-15T17:01:39.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-198 td/osd-backfill-space/datafile 2021-01-15T17:01:39.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:39.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:39.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-199 td/osd-backfill-space/datafile 2021-01-15T17:01:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-199 td/osd-backfill-space/datafile 2021-01-15T17:01:39.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:39.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:39.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:39.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-200 td/osd-backfill-space/datafile 2021-01-15T17:01:40.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-200 td/osd-backfill-space/datafile 2021-01-15T17:01:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:40.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-201 td/osd-backfill-space/datafile 2021-01-15T17:01:40.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-201 td/osd-backfill-space/datafile 2021-01-15T17:01:40.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:40.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:40.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-202 td/osd-backfill-space/datafile 2021-01-15T17:01:40.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-202 td/osd-backfill-space/datafile 2021-01-15T17:01:40.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:40.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-203 td/osd-backfill-space/datafile 2021-01-15T17:01:40.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-203 td/osd-backfill-space/datafile 2021-01-15T17:01:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:40.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:40.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:40.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-204 td/osd-backfill-space/datafile 2021-01-15T17:01:41.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-204 td/osd-backfill-space/datafile 2021-01-15T17:01:41.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:41.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:41.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-205 td/osd-backfill-space/datafile 2021-01-15T17:01:41.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-205 td/osd-backfill-space/datafile 2021-01-15T17:01:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:41.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-206 td/osd-backfill-space/datafile 2021-01-15T17:01:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-206 td/osd-backfill-space/datafile 2021-01-15T17:01:41.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:41.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:41.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-207 td/osd-backfill-space/datafile 2021-01-15T17:01:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-207 td/osd-backfill-space/datafile 2021-01-15T17:01:41.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:41.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:41.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-208 td/osd-backfill-space/datafile 2021-01-15T17:01:41.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:41.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-208 td/osd-backfill-space/datafile 2021-01-15T17:01:42.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:42.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-209 td/osd-backfill-space/datafile 2021-01-15T17:01:42.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-209 td/osd-backfill-space/datafile 2021-01-15T17:01:42.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:42.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:42.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-210 td/osd-backfill-space/datafile 2021-01-15T17:01:42.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-210 td/osd-backfill-space/datafile 2021-01-15T17:01:42.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:42.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:42.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-211 td/osd-backfill-space/datafile 2021-01-15T17:01:42.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-211 td/osd-backfill-space/datafile 2021-01-15T17:01:42.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:42.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:42.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-212 td/osd-backfill-space/datafile 2021-01-15T17:01:42.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:42.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-212 td/osd-backfill-space/datafile 2021-01-15T17:01:43.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:43.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:43.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-213 td/osd-backfill-space/datafile 2021-01-15T17:01:43.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-213 td/osd-backfill-space/datafile 2021-01-15T17:01:43.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:43.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:43.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-214 td/osd-backfill-space/datafile 2021-01-15T17:01:43.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-214 td/osd-backfill-space/datafile 2021-01-15T17:01:43.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:43.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:43.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-215 td/osd-backfill-space/datafile 2021-01-15T17:01:43.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-215 td/osd-backfill-space/datafile 2021-01-15T17:01:43.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:43.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:43.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-216 td/osd-backfill-space/datafile 2021-01-15T17:01:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-216 td/osd-backfill-space/datafile 2021-01-15T17:01:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:43.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:43.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:43.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-217 td/osd-backfill-space/datafile 2021-01-15T17:01:44.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-217 td/osd-backfill-space/datafile 2021-01-15T17:01:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:44.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-218 td/osd-backfill-space/datafile 2021-01-15T17:01:44.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-218 td/osd-backfill-space/datafile 2021-01-15T17:01:44.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:44.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:44.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-219 td/osd-backfill-space/datafile 2021-01-15T17:01:44.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-219 td/osd-backfill-space/datafile 2021-01-15T17:01:44.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:44.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:44.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-220 td/osd-backfill-space/datafile 2021-01-15T17:01:44.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-220 td/osd-backfill-space/datafile 2021-01-15T17:01:44.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:44.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:44.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:44.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-221 td/osd-backfill-space/datafile 2021-01-15T17:01:45.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-221 td/osd-backfill-space/datafile 2021-01-15T17:01:45.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:45.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:45.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-222 td/osd-backfill-space/datafile 2021-01-15T17:01:45.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-222 td/osd-backfill-space/datafile 2021-01-15T17:01:45.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:45.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:45.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-223 td/osd-backfill-space/datafile 2021-01-15T17:01:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-223 td/osd-backfill-space/datafile 2021-01-15T17:01:45.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:45.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:45.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-224 td/osd-backfill-space/datafile 2021-01-15T17:01:45.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-224 td/osd-backfill-space/datafile 2021-01-15T17:01:45.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:45.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:45.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-225 td/osd-backfill-space/datafile 2021-01-15T17:01:45.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:45.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-225 td/osd-backfill-space/datafile 2021-01-15T17:01:46.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:46.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-226 td/osd-backfill-space/datafile 2021-01-15T17:01:46.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-226 td/osd-backfill-space/datafile 2021-01-15T17:01:46.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:46.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:46.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-227 td/osd-backfill-space/datafile 2021-01-15T17:01:46.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-227 td/osd-backfill-space/datafile 2021-01-15T17:01:46.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:46.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:46.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-228 td/osd-backfill-space/datafile 2021-01-15T17:01:46.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-228 td/osd-backfill-space/datafile 2021-01-15T17:01:46.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:46.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:46.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-229 td/osd-backfill-space/datafile 2021-01-15T17:01:46.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-229 td/osd-backfill-space/datafile 2021-01-15T17:01:46.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:46.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:46.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:46.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-230 td/osd-backfill-space/datafile 2021-01-15T17:01:47.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:47.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-230 td/osd-backfill-space/datafile 2021-01-15T17:01:47.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:47.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-231 td/osd-backfill-space/datafile 2021-01-15T17:01:47.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:47.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-231 td/osd-backfill-space/datafile 2021-01-15T17:01:47.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:47.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:47.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:47.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-232 td/osd-backfill-space/datafile 2021-01-15T17:01:47.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:47.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-232 td/osd-backfill-space/datafile 2021-01-15T17:01:48.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:48.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:48.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-233 td/osd-backfill-space/datafile 2021-01-15T17:01:48.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-233 td/osd-backfill-space/datafile 2021-01-15T17:01:48.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:48.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:48.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-234 td/osd-backfill-space/datafile 2021-01-15T17:01:48.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-234 td/osd-backfill-space/datafile 2021-01-15T17:01:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:48.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-235 td/osd-backfill-space/datafile 2021-01-15T17:01:48.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-235 td/osd-backfill-space/datafile 2021-01-15T17:01:48.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:48.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:48.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-236 td/osd-backfill-space/datafile 2021-01-15T17:01:48.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:48.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-236 td/osd-backfill-space/datafile 2021-01-15T17:01:49.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:49.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:49.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-237 td/osd-backfill-space/datafile 2021-01-15T17:01:49.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-237 td/osd-backfill-space/datafile 2021-01-15T17:01:49.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:49.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:49.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-238 td/osd-backfill-space/datafile 2021-01-15T17:01:49.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-238 td/osd-backfill-space/datafile 2021-01-15T17:01:49.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:49.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:49.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-239 td/osd-backfill-space/datafile 2021-01-15T17:01:49.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-239 td/osd-backfill-space/datafile 2021-01-15T17:01:49.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:49.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:49.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-240 td/osd-backfill-space/datafile 2021-01-15T17:01:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-240 td/osd-backfill-space/datafile 2021-01-15T17:01:49.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:49.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:49.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:49.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-241 td/osd-backfill-space/datafile 2021-01-15T17:01:50.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-241 td/osd-backfill-space/datafile 2021-01-15T17:01:50.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:50.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:50.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-242 td/osd-backfill-space/datafile 2021-01-15T17:01:50.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-242 td/osd-backfill-space/datafile 2021-01-15T17:01:50.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:50.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:50.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-243 td/osd-backfill-space/datafile 2021-01-15T17:01:50.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-243 td/osd-backfill-space/datafile 2021-01-15T17:01:50.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:50.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:50.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-244 td/osd-backfill-space/datafile 2021-01-15T17:01:50.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-244 td/osd-backfill-space/datafile 2021-01-15T17:01:50.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:50.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:50.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:50.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-245 td/osd-backfill-space/datafile 2021-01-15T17:01:51.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-245 td/osd-backfill-space/datafile 2021-01-15T17:01:51.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:51.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-246 td/osd-backfill-space/datafile 2021-01-15T17:01:51.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-246 td/osd-backfill-space/datafile 2021-01-15T17:01:51.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:51.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-247 td/osd-backfill-space/datafile 2021-01-15T17:01:51.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-247 td/osd-backfill-space/datafile 2021-01-15T17:01:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:51.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-248 td/osd-backfill-space/datafile 2021-01-15T17:01:51.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-248 td/osd-backfill-space/datafile 2021-01-15T17:01:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:51.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-249 td/osd-backfill-space/datafile 2021-01-15T17:01:51.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:51.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-249 td/osd-backfill-space/datafile 2021-01-15T17:01:52.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:52.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:52.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-250 td/osd-backfill-space/datafile 2021-01-15T17:01:52.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-250 td/osd-backfill-space/datafile 2021-01-15T17:01:52.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:52.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-251 td/osd-backfill-space/datafile 2021-01-15T17:01:52.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-251 td/osd-backfill-space/datafile 2021-01-15T17:01:52.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:52.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:52.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-252 td/osd-backfill-space/datafile 2021-01-15T17:01:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-252 td/osd-backfill-space/datafile 2021-01-15T17:01:52.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:52.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:52.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:52.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-253 td/osd-backfill-space/datafile 2021-01-15T17:01:53.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-253 td/osd-backfill-space/datafile 2021-01-15T17:01:53.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:53.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:53.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-254 td/osd-backfill-space/datafile 2021-01-15T17:01:53.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-254 td/osd-backfill-space/datafile 2021-01-15T17:01:53.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:53.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:53.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-255 td/osd-backfill-space/datafile 2021-01-15T17:01:53.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-255 td/osd-backfill-space/datafile 2021-01-15T17:01:53.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:53.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:53.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-256 td/osd-backfill-space/datafile 2021-01-15T17:01:53.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-256 td/osd-backfill-space/datafile 2021-01-15T17:01:53.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:53.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:53.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:53.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-257 td/osd-backfill-space/datafile 2021-01-15T17:01:54.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-257 td/osd-backfill-space/datafile 2021-01-15T17:01:54.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:54.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:54.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-258 td/osd-backfill-space/datafile 2021-01-15T17:01:54.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-258 td/osd-backfill-space/datafile 2021-01-15T17:01:54.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:54.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-259 td/osd-backfill-space/datafile 2021-01-15T17:01:54.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-259 td/osd-backfill-space/datafile 2021-01-15T17:01:54.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:54.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-260 td/osd-backfill-space/datafile 2021-01-15T17:01:54.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:54.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-260 td/osd-backfill-space/datafile 2021-01-15T17:01:55.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:55.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:55.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-261 td/osd-backfill-space/datafile 2021-01-15T17:01:55.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-261 td/osd-backfill-space/datafile 2021-01-15T17:01:55.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:55.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:55.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-262 td/osd-backfill-space/datafile 2021-01-15T17:01:55.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-262 td/osd-backfill-space/datafile 2021-01-15T17:01:55.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:55.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-263 td/osd-backfill-space/datafile 2021-01-15T17:01:55.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-263 td/osd-backfill-space/datafile 2021-01-15T17:01:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:55.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:55.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-264 td/osd-backfill-space/datafile 2021-01-15T17:01:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-264 td/osd-backfill-space/datafile 2021-01-15T17:01:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:56.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:56.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-265 td/osd-backfill-space/datafile 2021-01-15T17:01:56.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-265 td/osd-backfill-space/datafile 2021-01-15T17:01:56.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:56.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:56.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-266 td/osd-backfill-space/datafile 2021-01-15T17:01:56.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-266 td/osd-backfill-space/datafile 2021-01-15T17:01:56.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:56.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:56.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-267 td/osd-backfill-space/datafile 2021-01-15T17:01:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-267 td/osd-backfill-space/datafile 2021-01-15T17:01:56.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:56.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:56.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:56.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-268 td/osd-backfill-space/datafile 2021-01-15T17:01:57.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-268 td/osd-backfill-space/datafile 2021-01-15T17:01:57.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:57.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:57.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-269 td/osd-backfill-space/datafile 2021-01-15T17:01:57.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-269 td/osd-backfill-space/datafile 2021-01-15T17:01:57.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:57.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:57.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-270 td/osd-backfill-space/datafile 2021-01-15T17:01:57.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-270 td/osd-backfill-space/datafile 2021-01-15T17:01:57.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:57.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:57.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-271 td/osd-backfill-space/datafile 2021-01-15T17:01:57.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:57.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-271 td/osd-backfill-space/datafile 2021-01-15T17:01:58.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:58.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-272 td/osd-backfill-space/datafile 2021-01-15T17:01:58.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-272 td/osd-backfill-space/datafile 2021-01-15T17:01:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:58.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:58.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-273 td/osd-backfill-space/datafile 2021-01-15T17:01:58.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-273 td/osd-backfill-space/datafile 2021-01-15T17:01:58.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:58.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:58.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-274 td/osd-backfill-space/datafile 2021-01-15T17:01:58.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-274 td/osd-backfill-space/datafile 2021-01-15T17:01:58.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:58.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:58.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-275 td/osd-backfill-space/datafile 2021-01-15T17:01:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-275 td/osd-backfill-space/datafile 2021-01-15T17:01:59.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:59.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-276 td/osd-backfill-space/datafile 2021-01-15T17:01:59.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-276 td/osd-backfill-space/datafile 2021-01-15T17:01:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:59.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:59.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-277 td/osd-backfill-space/datafile 2021-01-15T17:01:59.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-277 td/osd-backfill-space/datafile 2021-01-15T17:01:59.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:59.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:59.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-278 td/osd-backfill-space/datafile 2021-01-15T17:01:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-278 td/osd-backfill-space/datafile 2021-01-15T17:01:59.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:01:59.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:01:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:01:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-279 td/osd-backfill-space/datafile 2021-01-15T17:02:00.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-279 td/osd-backfill-space/datafile 2021-01-15T17:02:00.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:00.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:00.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-280 td/osd-backfill-space/datafile 2021-01-15T17:02:00.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-280 td/osd-backfill-space/datafile 2021-01-15T17:02:00.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:00.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:00.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-281 td/osd-backfill-space/datafile 2021-01-15T17:02:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-281 td/osd-backfill-space/datafile 2021-01-15T17:02:00.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:00.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-282 td/osd-backfill-space/datafile 2021-01-15T17:02:00.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-282 td/osd-backfill-space/datafile 2021-01-15T17:02:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:00.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:00.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:00.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-283 td/osd-backfill-space/datafile 2021-01-15T17:02:01.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-283 td/osd-backfill-space/datafile 2021-01-15T17:02:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:01.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:01.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-284 td/osd-backfill-space/datafile 2021-01-15T17:02:01.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-284 td/osd-backfill-space/datafile 2021-01-15T17:02:01.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:01.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:01.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-285 td/osd-backfill-space/datafile 2021-01-15T17:02:01.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-285 td/osd-backfill-space/datafile 2021-01-15T17:02:01.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:01.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:01.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-286 td/osd-backfill-space/datafile 2021-01-15T17:02:01.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-286 td/osd-backfill-space/datafile 2021-01-15T17:02:01.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:01.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-287 td/osd-backfill-space/datafile 2021-01-15T17:02:02.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-287 td/osd-backfill-space/datafile 2021-01-15T17:02:02.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:02.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:02.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-288 td/osd-backfill-space/datafile 2021-01-15T17:02:02.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-288 td/osd-backfill-space/datafile 2021-01-15T17:02:02.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:02.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:02.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-289 td/osd-backfill-space/datafile 2021-01-15T17:02:02.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-289 td/osd-backfill-space/datafile 2021-01-15T17:02:02.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:02.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:02.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-290 td/osd-backfill-space/datafile 2021-01-15T17:02:02.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:02.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-290 td/osd-backfill-space/datafile 2021-01-15T17:02:03.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:03.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-291 td/osd-backfill-space/datafile 2021-01-15T17:02:03.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-291 td/osd-backfill-space/datafile 2021-01-15T17:02:03.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:03.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:03.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-292 td/osd-backfill-space/datafile 2021-01-15T17:02:03.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-292 td/osd-backfill-space/datafile 2021-01-15T17:02:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:03.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:03.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-293 td/osd-backfill-space/datafile 2021-01-15T17:02:03.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-293 td/osd-backfill-space/datafile 2021-01-15T17:02:03.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:03.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:03.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:03.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-294 td/osd-backfill-space/datafile 2021-01-15T17:02:04.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-294 td/osd-backfill-space/datafile 2021-01-15T17:02:04.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:04.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:04.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-295 td/osd-backfill-space/datafile 2021-01-15T17:02:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-295 td/osd-backfill-space/datafile 2021-01-15T17:02:04.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:04.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:04.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-296 td/osd-backfill-space/datafile 2021-01-15T17:02:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-296 td/osd-backfill-space/datafile 2021-01-15T17:02:04.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:04.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:04.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-297 td/osd-backfill-space/datafile 2021-01-15T17:02:04.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-297 td/osd-backfill-space/datafile 2021-01-15T17:02:04.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:04.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:04.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:04.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-298 td/osd-backfill-space/datafile 2021-01-15T17:02:05.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-298 td/osd-backfill-space/datafile 2021-01-15T17:02:05.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:05.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:05.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-299 td/osd-backfill-space/datafile 2021-01-15T17:02:05.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-299 td/osd-backfill-space/datafile 2021-01-15T17:02:05.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:05.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:05.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-300 td/osd-backfill-space/datafile 2021-01-15T17:02:05.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-300 td/osd-backfill-space/datafile 2021-01-15T17:02:05.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:05.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:05.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-301 td/osd-backfill-space/datafile 2021-01-15T17:02:05.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-301 td/osd-backfill-space/datafile 2021-01-15T17:02:05.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:05.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:05.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:05.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-302 td/osd-backfill-space/datafile 2021-01-15T17:02:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-302 td/osd-backfill-space/datafile 2021-01-15T17:02:06.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:06.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:06.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-303 td/osd-backfill-space/datafile 2021-01-15T17:02:06.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-303 td/osd-backfill-space/datafile 2021-01-15T17:02:06.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:06.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:06.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-304 td/osd-backfill-space/datafile 2021-01-15T17:02:06.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-304 td/osd-backfill-space/datafile 2021-01-15T17:02:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:06.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:06.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-305 td/osd-backfill-space/datafile 2021-01-15T17:02:06.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:06.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-305 td/osd-backfill-space/datafile 2021-01-15T17:02:07.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:07.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:07.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-306 td/osd-backfill-space/datafile 2021-01-15T17:02:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-306 td/osd-backfill-space/datafile 2021-01-15T17:02:07.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:07.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:07.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-307 td/osd-backfill-space/datafile 2021-01-15T17:02:07.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-307 td/osd-backfill-space/datafile 2021-01-15T17:02:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:07.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-308 td/osd-backfill-space/datafile 2021-01-15T17:02:07.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-308 td/osd-backfill-space/datafile 2021-01-15T17:02:07.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:07.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:07.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-309 td/osd-backfill-space/datafile 2021-01-15T17:02:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:07.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-309 td/osd-backfill-space/datafile 2021-01-15T17:02:08.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:08.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:08.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-310 td/osd-backfill-space/datafile 2021-01-15T17:02:08.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-310 td/osd-backfill-space/datafile 2021-01-15T17:02:08.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:08.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-311 td/osd-backfill-space/datafile 2021-01-15T17:02:08.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-311 td/osd-backfill-space/datafile 2021-01-15T17:02:08.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:08.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:08.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-312 td/osd-backfill-space/datafile 2021-01-15T17:02:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-312 td/osd-backfill-space/datafile 2021-01-15T17:02:08.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:08.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:08.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:08.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-313 td/osd-backfill-space/datafile 2021-01-15T17:02:09.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-313 td/osd-backfill-space/datafile 2021-01-15T17:02:09.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:09.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-314 td/osd-backfill-space/datafile 2021-01-15T17:02:09.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-314 td/osd-backfill-space/datafile 2021-01-15T17:02:09.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:09.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-315 td/osd-backfill-space/datafile 2021-01-15T17:02:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-315 td/osd-backfill-space/datafile 2021-01-15T17:02:09.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:09.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:09.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-316 td/osd-backfill-space/datafile 2021-01-15T17:02:09.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-316 td/osd-backfill-space/datafile 2021-01-15T17:02:09.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:09.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:09.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-317 td/osd-backfill-space/datafile 2021-01-15T17:02:10.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-317 td/osd-backfill-space/datafile 2021-01-15T17:02:10.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:10.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:10.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-318 td/osd-backfill-space/datafile 2021-01-15T17:02:10.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-318 td/osd-backfill-space/datafile 2021-01-15T17:02:10.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:10.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-319 td/osd-backfill-space/datafile 2021-01-15T17:02:10.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-319 td/osd-backfill-space/datafile 2021-01-15T17:02:10.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:10.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-320 td/osd-backfill-space/datafile 2021-01-15T17:02:10.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-320 td/osd-backfill-space/datafile 2021-01-15T17:02:10.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:10.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:10.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:10.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-321 td/osd-backfill-space/datafile 2021-01-15T17:02:11.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-321 td/osd-backfill-space/datafile 2021-01-15T17:02:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:11.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:11.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-322 td/osd-backfill-space/datafile 2021-01-15T17:02:11.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-322 td/osd-backfill-space/datafile 2021-01-15T17:02:11.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:11.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:11.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-323 td/osd-backfill-space/datafile 2021-01-15T17:02:11.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-323 td/osd-backfill-space/datafile 2021-01-15T17:02:11.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:11.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:11.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-324 td/osd-backfill-space/datafile 2021-01-15T17:02:11.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-324 td/osd-backfill-space/datafile 2021-01-15T17:02:11.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:11.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:11.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:11.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-325 td/osd-backfill-space/datafile 2021-01-15T17:02:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-325 td/osd-backfill-space/datafile 2021-01-15T17:02:12.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:12.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:12.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-326 td/osd-backfill-space/datafile 2021-01-15T17:02:12.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-326 td/osd-backfill-space/datafile 2021-01-15T17:02:12.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:12.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:12.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-327 td/osd-backfill-space/datafile 2021-01-15T17:02:12.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-327 td/osd-backfill-space/datafile 2021-01-15T17:02:12.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:12.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:12.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-328 td/osd-backfill-space/datafile 2021-01-15T17:02:12.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:12.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-328 td/osd-backfill-space/datafile 2021-01-15T17:02:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:13.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-329 td/osd-backfill-space/datafile 2021-01-15T17:02:13.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-329 td/osd-backfill-space/datafile 2021-01-15T17:02:13.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:13.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:13.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-330 td/osd-backfill-space/datafile 2021-01-15T17:02:13.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-330 td/osd-backfill-space/datafile 2021-01-15T17:02:13.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:13.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-331 td/osd-backfill-space/datafile 2021-01-15T17:02:13.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-331 td/osd-backfill-space/datafile 2021-01-15T17:02:13.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:13.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:13.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:13.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-332 td/osd-backfill-space/datafile 2021-01-15T17:02:14.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-332 td/osd-backfill-space/datafile 2021-01-15T17:02:14.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:14.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:14.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-333 td/osd-backfill-space/datafile 2021-01-15T17:02:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-333 td/osd-backfill-space/datafile 2021-01-15T17:02:14.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:14.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:14.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-334 td/osd-backfill-space/datafile 2021-01-15T17:02:14.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-334 td/osd-backfill-space/datafile 2021-01-15T17:02:14.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:14.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:14.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-335 td/osd-backfill-space/datafile 2021-01-15T17:02:14.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-335 td/osd-backfill-space/datafile 2021-01-15T17:02:14.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:14.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:14.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:14.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-336 td/osd-backfill-space/datafile 2021-01-15T17:02:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-336 td/osd-backfill-space/datafile 2021-01-15T17:02:15.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:15.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-337 td/osd-backfill-space/datafile 2021-01-15T17:02:15.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-337 td/osd-backfill-space/datafile 2021-01-15T17:02:15.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:15.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:15.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-338 td/osd-backfill-space/datafile 2021-01-15T17:02:15.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-338 td/osd-backfill-space/datafile 2021-01-15T17:02:15.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:15.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:15.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-339 td/osd-backfill-space/datafile 2021-01-15T17:02:15.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-339 td/osd-backfill-space/datafile 2021-01-15T17:02:15.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:15.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:15.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:15.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-340 td/osd-backfill-space/datafile 2021-01-15T17:02:16.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-340 td/osd-backfill-space/datafile 2021-01-15T17:02:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:16.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:16.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-341 td/osd-backfill-space/datafile 2021-01-15T17:02:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-341 td/osd-backfill-space/datafile 2021-01-15T17:02:16.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:16.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:16.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-342 td/osd-backfill-space/datafile 2021-01-15T17:02:16.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-342 td/osd-backfill-space/datafile 2021-01-15T17:02:16.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:16.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:16.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-343 td/osd-backfill-space/datafile 2021-01-15T17:02:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-343 td/osd-backfill-space/datafile 2021-01-15T17:02:16.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:16.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:16.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:16.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-344 td/osd-backfill-space/datafile 2021-01-15T17:02:17.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-344 td/osd-backfill-space/datafile 2021-01-15T17:02:17.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:17.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:17.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-345 td/osd-backfill-space/datafile 2021-01-15T17:02:17.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-345 td/osd-backfill-space/datafile 2021-01-15T17:02:17.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:17.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:17.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-346 td/osd-backfill-space/datafile 2021-01-15T17:02:17.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-346 td/osd-backfill-space/datafile 2021-01-15T17:02:17.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:17.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:17.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-347 td/osd-backfill-space/datafile 2021-01-15T17:02:17.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:17.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-347 td/osd-backfill-space/datafile 2021-01-15T17:02:18.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:18.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:18.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:18.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-348 td/osd-backfill-space/datafile 2021-01-15T17:02:18.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:18.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-348 td/osd-backfill-space/datafile 2021-01-15T17:02:18.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:18.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:18.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:18.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-349 td/osd-backfill-space/datafile 2021-01-15T17:02:18.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:18.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-349 td/osd-backfill-space/datafile 2021-01-15T17:02:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:18.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:18.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:18.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-350 td/osd-backfill-space/datafile 2021-01-15T17:02:19.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-350 td/osd-backfill-space/datafile 2021-01-15T17:02:19.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:19.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:19.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-351 td/osd-backfill-space/datafile 2021-01-15T17:02:19.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-351 td/osd-backfill-space/datafile 2021-01-15T17:02:19.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:19.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:19.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-352 td/osd-backfill-space/datafile 2021-01-15T17:02:19.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-352 td/osd-backfill-space/datafile 2021-01-15T17:02:19.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:19.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:19.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-353 td/osd-backfill-space/datafile 2021-01-15T17:02:19.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-353 td/osd-backfill-space/datafile 2021-01-15T17:02:19.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:19.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:19.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:19.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-354 td/osd-backfill-space/datafile 2021-01-15T17:02:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-354 td/osd-backfill-space/datafile 2021-01-15T17:02:20.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:20.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:20.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-355 td/osd-backfill-space/datafile 2021-01-15T17:02:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-355 td/osd-backfill-space/datafile 2021-01-15T17:02:20.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:20.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:20.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-356 td/osd-backfill-space/datafile 2021-01-15T17:02:20.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-356 td/osd-backfill-space/datafile 2021-01-15T17:02:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:20.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-357 td/osd-backfill-space/datafile 2021-01-15T17:02:20.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:20.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-357 td/osd-backfill-space/datafile 2021-01-15T17:02:21.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:21.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:21.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-358 td/osd-backfill-space/datafile 2021-01-15T17:02:21.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-358 td/osd-backfill-space/datafile 2021-01-15T17:02:21.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:21.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:21.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-359 td/osd-backfill-space/datafile 2021-01-15T17:02:21.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-359 td/osd-backfill-space/datafile 2021-01-15T17:02:21.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:21.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:21.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-360 td/osd-backfill-space/datafile 2021-01-15T17:02:21.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-360 td/osd-backfill-space/datafile 2021-01-15T17:02:21.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:21.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:21.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-361 td/osd-backfill-space/datafile 2021-01-15T17:02:21.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:21.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-361 td/osd-backfill-space/datafile 2021-01-15T17:02:22.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:22.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:22.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-362 td/osd-backfill-space/datafile 2021-01-15T17:02:22.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-362 td/osd-backfill-space/datafile 2021-01-15T17:02:22.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:22.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-363 td/osd-backfill-space/datafile 2021-01-15T17:02:22.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-363 td/osd-backfill-space/datafile 2021-01-15T17:02:22.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:22.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:22.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-364 td/osd-backfill-space/datafile 2021-01-15T17:02:22.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-364 td/osd-backfill-space/datafile 2021-01-15T17:02:22.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:22.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-365 td/osd-backfill-space/datafile 2021-01-15T17:02:23.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-365 td/osd-backfill-space/datafile 2021-01-15T17:02:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:23.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:23.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-366 td/osd-backfill-space/datafile 2021-01-15T17:02:23.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-366 td/osd-backfill-space/datafile 2021-01-15T17:02:23.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:23.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:23.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-367 td/osd-backfill-space/datafile 2021-01-15T17:02:23.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-367 td/osd-backfill-space/datafile 2021-01-15T17:02:23.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:23.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:23.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-368 td/osd-backfill-space/datafile 2021-01-15T17:02:23.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-368 td/osd-backfill-space/datafile 2021-01-15T17:02:23.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:23.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:23.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:23.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-369 td/osd-backfill-space/datafile 2021-01-15T17:02:24.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-369 td/osd-backfill-space/datafile 2021-01-15T17:02:24.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:24.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-370 td/osd-backfill-space/datafile 2021-01-15T17:02:24.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-370 td/osd-backfill-space/datafile 2021-01-15T17:02:24.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:24.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:24.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-371 td/osd-backfill-space/datafile 2021-01-15T17:02:24.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-371 td/osd-backfill-space/datafile 2021-01-15T17:02:24.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:24.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:24.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-372 td/osd-backfill-space/datafile 2021-01-15T17:02:24.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-372 td/osd-backfill-space/datafile 2021-01-15T17:02:24.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:24.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:24.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:24.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-373 td/osd-backfill-space/datafile 2021-01-15T17:02:25.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-373 td/osd-backfill-space/datafile 2021-01-15T17:02:25.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:25.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:25.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-374 td/osd-backfill-space/datafile 2021-01-15T17:02:25.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-374 td/osd-backfill-space/datafile 2021-01-15T17:02:25.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:25.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:25.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-375 td/osd-backfill-space/datafile 2021-01-15T17:02:25.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-375 td/osd-backfill-space/datafile 2021-01-15T17:02:25.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:25.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-376 td/osd-backfill-space/datafile 2021-01-15T17:02:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-376 td/osd-backfill-space/datafile 2021-01-15T17:02:25.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:25.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:25.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:25.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-377 td/osd-backfill-space/datafile 2021-01-15T17:02:26.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-377 td/osd-backfill-space/datafile 2021-01-15T17:02:26.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:26.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:26.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-378 td/osd-backfill-space/datafile 2021-01-15T17:02:26.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-378 td/osd-backfill-space/datafile 2021-01-15T17:02:26.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:26.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:26.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-379 td/osd-backfill-space/datafile 2021-01-15T17:02:26.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-379 td/osd-backfill-space/datafile 2021-01-15T17:02:26.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:26.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:26.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-380 td/osd-backfill-space/datafile 2021-01-15T17:02:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-380 td/osd-backfill-space/datafile 2021-01-15T17:02:27.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:27.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:27.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-381 td/osd-backfill-space/datafile 2021-01-15T17:02:27.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-381 td/osd-backfill-space/datafile 2021-01-15T17:02:27.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:27.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:27.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-382 td/osd-backfill-space/datafile 2021-01-15T17:02:27.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-382 td/osd-backfill-space/datafile 2021-01-15T17:02:27.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:27.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:27.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-383 td/osd-backfill-space/datafile 2021-01-15T17:02:27.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-383 td/osd-backfill-space/datafile 2021-01-15T17:02:27.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:27.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:27.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-384 td/osd-backfill-space/datafile 2021-01-15T17:02:27.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:27.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-384 td/osd-backfill-space/datafile 2021-01-15T17:02:28.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:28.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:28.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-385 td/osd-backfill-space/datafile 2021-01-15T17:02:28.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-385 td/osd-backfill-space/datafile 2021-01-15T17:02:28.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:28.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:28.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-386 td/osd-backfill-space/datafile 2021-01-15T17:02:28.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-386 td/osd-backfill-space/datafile 2021-01-15T17:02:28.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:28.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:28.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-387 td/osd-backfill-space/datafile 2021-01-15T17:02:28.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-387 td/osd-backfill-space/datafile 2021-01-15T17:02:28.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:28.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:28.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:28.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-388 td/osd-backfill-space/datafile 2021-01-15T17:02:29.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-388 td/osd-backfill-space/datafile 2021-01-15T17:02:29.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:29.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:29.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-389 td/osd-backfill-space/datafile 2021-01-15T17:02:29.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-389 td/osd-backfill-space/datafile 2021-01-15T17:02:29.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:29.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:29.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-390 td/osd-backfill-space/datafile 2021-01-15T17:02:29.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-390 td/osd-backfill-space/datafile 2021-01-15T17:02:29.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:29.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:29.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-391 td/osd-backfill-space/datafile 2021-01-15T17:02:29.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-391 td/osd-backfill-space/datafile 2021-01-15T17:02:29.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:29.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:29.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:29.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-392 td/osd-backfill-space/datafile 2021-01-15T17:02:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-392 td/osd-backfill-space/datafile 2021-01-15T17:02:30.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:30.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-393 td/osd-backfill-space/datafile 2021-01-15T17:02:30.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-393 td/osd-backfill-space/datafile 2021-01-15T17:02:30.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:30.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-394 td/osd-backfill-space/datafile 2021-01-15T17:02:30.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-394 td/osd-backfill-space/datafile 2021-01-15T17:02:30.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:30.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:30.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-395 td/osd-backfill-space/datafile 2021-01-15T17:02:30.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:30.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-395 td/osd-backfill-space/datafile 2021-01-15T17:02:31.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:31.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:31.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-396 td/osd-backfill-space/datafile 2021-01-15T17:02:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-396 td/osd-backfill-space/datafile 2021-01-15T17:02:31.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:31.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:31.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-397 td/osd-backfill-space/datafile 2021-01-15T17:02:31.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-397 td/osd-backfill-space/datafile 2021-01-15T17:02:31.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:31.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:31.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-398 td/osd-backfill-space/datafile 2021-01-15T17:02:31.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-398 td/osd-backfill-space/datafile 2021-01-15T17:02:31.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:31.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:31.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-399 td/osd-backfill-space/datafile 2021-01-15T17:02:31.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:31.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-399 td/osd-backfill-space/datafile 2021-01-15T17:02:32.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:32.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-400 td/osd-backfill-space/datafile 2021-01-15T17:02:32.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-400 td/osd-backfill-space/datafile 2021-01-15T17:02:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:32.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:32.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-401 td/osd-backfill-space/datafile 2021-01-15T17:02:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-401 td/osd-backfill-space/datafile 2021-01-15T17:02:32.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:32.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:32.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-402 td/osd-backfill-space/datafile 2021-01-15T17:02:32.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-402 td/osd-backfill-space/datafile 2021-01-15T17:02:32.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:32.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:32.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-403 td/osd-backfill-space/datafile 2021-01-15T17:02:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-403 td/osd-backfill-space/datafile 2021-01-15T17:02:33.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:33.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-404 td/osd-backfill-space/datafile 2021-01-15T17:02:33.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-404 td/osd-backfill-space/datafile 2021-01-15T17:02:33.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:33.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:33.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-405 td/osd-backfill-space/datafile 2021-01-15T17:02:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-405 td/osd-backfill-space/datafile 2021-01-15T17:02:33.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:33.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:33.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-406 td/osd-backfill-space/datafile 2021-01-15T17:02:33.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-406 td/osd-backfill-space/datafile 2021-01-15T17:02:33.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:33.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:33.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-407 td/osd-backfill-space/datafile 2021-01-15T17:02:33.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:33.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-407 td/osd-backfill-space/datafile 2021-01-15T17:02:34.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:34.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:34.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-408 td/osd-backfill-space/datafile 2021-01-15T17:02:34.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-408 td/osd-backfill-space/datafile 2021-01-15T17:02:34.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:34.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:34.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-409 td/osd-backfill-space/datafile 2021-01-15T17:02:34.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-409 td/osd-backfill-space/datafile 2021-01-15T17:02:34.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:34.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:34.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-410 td/osd-backfill-space/datafile 2021-01-15T17:02:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-410 td/osd-backfill-space/datafile 2021-01-15T17:02:34.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:34.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:34.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:34.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-411 td/osd-backfill-space/datafile 2021-01-15T17:02:35.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-411 td/osd-backfill-space/datafile 2021-01-15T17:02:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:35.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-412 td/osd-backfill-space/datafile 2021-01-15T17:02:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-412 td/osd-backfill-space/datafile 2021-01-15T17:02:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:35.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-413 td/osd-backfill-space/datafile 2021-01-15T17:02:35.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-413 td/osd-backfill-space/datafile 2021-01-15T17:02:35.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:35.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:35.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-414 td/osd-backfill-space/datafile 2021-01-15T17:02:35.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-414 td/osd-backfill-space/datafile 2021-01-15T17:02:35.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:35.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:35.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:35.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-415 td/osd-backfill-space/datafile 2021-01-15T17:02:36.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-415 td/osd-backfill-space/datafile 2021-01-15T17:02:36.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:36.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:36.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-416 td/osd-backfill-space/datafile 2021-01-15T17:02:36.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-416 td/osd-backfill-space/datafile 2021-01-15T17:02:36.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:36.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-417 td/osd-backfill-space/datafile 2021-01-15T17:02:36.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-417 td/osd-backfill-space/datafile 2021-01-15T17:02:36.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:36.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:36.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-418 td/osd-backfill-space/datafile 2021-01-15T17:02:36.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:36.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-418 td/osd-backfill-space/datafile 2021-01-15T17:02:37.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:37.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:37.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-419 td/osd-backfill-space/datafile 2021-01-15T17:02:37.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-419 td/osd-backfill-space/datafile 2021-01-15T17:02:37.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:37.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:37.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-420 td/osd-backfill-space/datafile 2021-01-15T17:02:37.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-420 td/osd-backfill-space/datafile 2021-01-15T17:02:37.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:37.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:37.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-421 td/osd-backfill-space/datafile 2021-01-15T17:02:37.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-421 td/osd-backfill-space/datafile 2021-01-15T17:02:37.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:37.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:37.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-422 td/osd-backfill-space/datafile 2021-01-15T17:02:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-422 td/osd-backfill-space/datafile 2021-01-15T17:02:38.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:38.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-423 td/osd-backfill-space/datafile 2021-01-15T17:02:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-423 td/osd-backfill-space/datafile 2021-01-15T17:02:38.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:38.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:38.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-424 td/osd-backfill-space/datafile 2021-01-15T17:02:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-424 td/osd-backfill-space/datafile 2021-01-15T17:02:38.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:38.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-425 td/osd-backfill-space/datafile 2021-01-15T17:02:38.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-425 td/osd-backfill-space/datafile 2021-01-15T17:02:38.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:38.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:38.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-426 td/osd-backfill-space/datafile 2021-01-15T17:02:38.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:38.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-426 td/osd-backfill-space/datafile 2021-01-15T17:02:39.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:39.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:39.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-427 td/osd-backfill-space/datafile 2021-01-15T17:02:39.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-427 td/osd-backfill-space/datafile 2021-01-15T17:02:39.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:39.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:39.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-428 td/osd-backfill-space/datafile 2021-01-15T17:02:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-428 td/osd-backfill-space/datafile 2021-01-15T17:02:39.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:39.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:39.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-429 td/osd-backfill-space/datafile 2021-01-15T17:02:39.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-429 td/osd-backfill-space/datafile 2021-01-15T17:02:39.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:39.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:39.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-430 td/osd-backfill-space/datafile 2021-01-15T17:02:39.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:39.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-430 td/osd-backfill-space/datafile 2021-01-15T17:02:40.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:40.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-431 td/osd-backfill-space/datafile 2021-01-15T17:02:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-431 td/osd-backfill-space/datafile 2021-01-15T17:02:40.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:40.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:40.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-432 td/osd-backfill-space/datafile 2021-01-15T17:02:40.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-432 td/osd-backfill-space/datafile 2021-01-15T17:02:40.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:40.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:40.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-433 td/osd-backfill-space/datafile 2021-01-15T17:02:40.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-433 td/osd-backfill-space/datafile 2021-01-15T17:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-434 td/osd-backfill-space/datafile 2021-01-15T17:02:41.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-434 td/osd-backfill-space/datafile 2021-01-15T17:02:41.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:41.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:41.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-435 td/osd-backfill-space/datafile 2021-01-15T17:02:41.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-435 td/osd-backfill-space/datafile 2021-01-15T17:02:41.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:41.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:41.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-436 td/osd-backfill-space/datafile 2021-01-15T17:02:41.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-436 td/osd-backfill-space/datafile 2021-01-15T17:02:41.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:41.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-437 td/osd-backfill-space/datafile 2021-01-15T17:02:41.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-437 td/osd-backfill-space/datafile 2021-01-15T17:02:41.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:41.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:41.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-438 td/osd-backfill-space/datafile 2021-01-15T17:02:42.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-438 td/osd-backfill-space/datafile 2021-01-15T17:02:42.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:42.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-439 td/osd-backfill-space/datafile 2021-01-15T17:02:42.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-439 td/osd-backfill-space/datafile 2021-01-15T17:02:42.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:42.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:42.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-440 td/osd-backfill-space/datafile 2021-01-15T17:02:42.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-440 td/osd-backfill-space/datafile 2021-01-15T17:02:42.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:42.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:42.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-441 td/osd-backfill-space/datafile 2021-01-15T17:02:42.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:42.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-441 td/osd-backfill-space/datafile 2021-01-15T17:02:43.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:43.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:43.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-442 td/osd-backfill-space/datafile 2021-01-15T17:02:43.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-442 td/osd-backfill-space/datafile 2021-01-15T17:02:43.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:43.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:43.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-443 td/osd-backfill-space/datafile 2021-01-15T17:02:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-443 td/osd-backfill-space/datafile 2021-01-15T17:02:43.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:43.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:43.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-444 td/osd-backfill-space/datafile 2021-01-15T17:02:43.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-444 td/osd-backfill-space/datafile 2021-01-15T17:02:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:43.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:43.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-445 td/osd-backfill-space/datafile 2021-01-15T17:02:43.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:43.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-445 td/osd-backfill-space/datafile 2021-01-15T17:02:44.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:44.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:44.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-446 td/osd-backfill-space/datafile 2021-01-15T17:02:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-446 td/osd-backfill-space/datafile 2021-01-15T17:02:44.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:44.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:44.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-447 td/osd-backfill-space/datafile 2021-01-15T17:02:44.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-447 td/osd-backfill-space/datafile 2021-01-15T17:02:44.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:44.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:44.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-448 td/osd-backfill-space/datafile 2021-01-15T17:02:44.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-448 td/osd-backfill-space/datafile 2021-01-15T17:02:44.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:44.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:44.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:44.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-449 td/osd-backfill-space/datafile 2021-01-15T17:02:45.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-449 td/osd-backfill-space/datafile 2021-01-15T17:02:45.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:45.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:45.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-450 td/osd-backfill-space/datafile 2021-01-15T17:02:45.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-450 td/osd-backfill-space/datafile 2021-01-15T17:02:45.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:45.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:45.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-451 td/osd-backfill-space/datafile 2021-01-15T17:02:45.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-451 td/osd-backfill-space/datafile 2021-01-15T17:02:45.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:45.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:45.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-452 td/osd-backfill-space/datafile 2021-01-15T17:02:45.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-452 td/osd-backfill-space/datafile 2021-01-15T17:02:45.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:45.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:45.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:45.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-453 td/osd-backfill-space/datafile 2021-01-15T17:02:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-453 td/osd-backfill-space/datafile 2021-01-15T17:02:46.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:46.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:46.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-454 td/osd-backfill-space/datafile 2021-01-15T17:02:46.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-454 td/osd-backfill-space/datafile 2021-01-15T17:02:46.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:46.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:46.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-455 td/osd-backfill-space/datafile 2021-01-15T17:02:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-455 td/osd-backfill-space/datafile 2021-01-15T17:02:46.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:46.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:46.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-456 td/osd-backfill-space/datafile 2021-01-15T17:02:46.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:46.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-456 td/osd-backfill-space/datafile 2021-01-15T17:02:47.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:47.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:47.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-457 td/osd-backfill-space/datafile 2021-01-15T17:02:47.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-457 td/osd-backfill-space/datafile 2021-01-15T17:02:47.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:47.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-458 td/osd-backfill-space/datafile 2021-01-15T17:02:47.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-458 td/osd-backfill-space/datafile 2021-01-15T17:02:47.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:47.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:47.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-459 td/osd-backfill-space/datafile 2021-01-15T17:02:47.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-459 td/osd-backfill-space/datafile 2021-01-15T17:02:47.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:47.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:47.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-460 td/osd-backfill-space/datafile 2021-01-15T17:02:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-460 td/osd-backfill-space/datafile 2021-01-15T17:02:48.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:48.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:48.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-461 td/osd-backfill-space/datafile 2021-01-15T17:02:48.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-461 td/osd-backfill-space/datafile 2021-01-15T17:02:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:48.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:48.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-462 td/osd-backfill-space/datafile 2021-01-15T17:02:48.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-462 td/osd-backfill-space/datafile 2021-01-15T17:02:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:48.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:48.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-463 td/osd-backfill-space/datafile 2021-01-15T17:02:48.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-463 td/osd-backfill-space/datafile 2021-01-15T17:02:48.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:48.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:48.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-464 td/osd-backfill-space/datafile 2021-01-15T17:02:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:48.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-464 td/osd-backfill-space/datafile 2021-01-15T17:02:49.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:49.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:49.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-465 td/osd-backfill-space/datafile 2021-01-15T17:02:49.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:49.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-465 td/osd-backfill-space/datafile 2021-01-15T17:02:49.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:49.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:49.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:49.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-466 td/osd-backfill-space/datafile 2021-01-15T17:02:49.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:49.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-466 td/osd-backfill-space/datafile 2021-01-15T17:02:49.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:49.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:49.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:49.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-467 td/osd-backfill-space/datafile 2021-01-15T17:02:50.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-467 td/osd-backfill-space/datafile 2021-01-15T17:02:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-468 td/osd-backfill-space/datafile 2021-01-15T17:02:50.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-468 td/osd-backfill-space/datafile 2021-01-15T17:02:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:50.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-469 td/osd-backfill-space/datafile 2021-01-15T17:02:50.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-469 td/osd-backfill-space/datafile 2021-01-15T17:02:50.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:50.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:50.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-470 td/osd-backfill-space/datafile 2021-01-15T17:02:50.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:50.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-470 td/osd-backfill-space/datafile 2021-01-15T17:02:51.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:51.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:51.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-471 td/osd-backfill-space/datafile 2021-01-15T17:02:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-471 td/osd-backfill-space/datafile 2021-01-15T17:02:51.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:51.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:51.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-472 td/osd-backfill-space/datafile 2021-01-15T17:02:51.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-472 td/osd-backfill-space/datafile 2021-01-15T17:02:51.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:51.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:51.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-473 td/osd-backfill-space/datafile 2021-01-15T17:02:51.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-473 td/osd-backfill-space/datafile 2021-01-15T17:02:51.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:51.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:51.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-474 td/osd-backfill-space/datafile 2021-01-15T17:02:51.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:51.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-474 td/osd-backfill-space/datafile 2021-01-15T17:02:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-475 td/osd-backfill-space/datafile 2021-01-15T17:02:52.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-475 td/osd-backfill-space/datafile 2021-01-15T17:02:52.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:52.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-476 td/osd-backfill-space/datafile 2021-01-15T17:02:52.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-476 td/osd-backfill-space/datafile 2021-01-15T17:02:52.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:52.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:52.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-477 td/osd-backfill-space/datafile 2021-01-15T17:02:52.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-477 td/osd-backfill-space/datafile 2021-01-15T17:02:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:52.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:52.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:52.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-478 td/osd-backfill-space/datafile 2021-01-15T17:02:53.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-478 td/osd-backfill-space/datafile 2021-01-15T17:02:53.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:53.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:53.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-479 td/osd-backfill-space/datafile 2021-01-15T17:02:53.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-479 td/osd-backfill-space/datafile 2021-01-15T17:02:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:53.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-480 td/osd-backfill-space/datafile 2021-01-15T17:02:53.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-480 td/osd-backfill-space/datafile 2021-01-15T17:02:53.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:53.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:53.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-481 td/osd-backfill-space/datafile 2021-01-15T17:02:53.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-481 td/osd-backfill-space/datafile 2021-01-15T17:02:53.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:53.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:53.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:53.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-482 td/osd-backfill-space/datafile 2021-01-15T17:02:54.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-482 td/osd-backfill-space/datafile 2021-01-15T17:02:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:54.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-483 td/osd-backfill-space/datafile 2021-01-15T17:02:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-483 td/osd-backfill-space/datafile 2021-01-15T17:02:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:54.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-484 td/osd-backfill-space/datafile 2021-01-15T17:02:54.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-484 td/osd-backfill-space/datafile 2021-01-15T17:02:54.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:54.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:54.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-485 td/osd-backfill-space/datafile 2021-01-15T17:02:54.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:54.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-485 td/osd-backfill-space/datafile 2021-01-15T17:02:55.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:55.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:55.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-486 td/osd-backfill-space/datafile 2021-01-15T17:02:55.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-486 td/osd-backfill-space/datafile 2021-01-15T17:02:55.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:55.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:55.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-487 td/osd-backfill-space/datafile 2021-01-15T17:02:55.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-487 td/osd-backfill-space/datafile 2021-01-15T17:02:55.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:55.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-488 td/osd-backfill-space/datafile 2021-01-15T17:02:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-488 td/osd-backfill-space/datafile 2021-01-15T17:02:55.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:55.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:55.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-489 td/osd-backfill-space/datafile 2021-01-15T17:02:55.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:55.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-489 td/osd-backfill-space/datafile 2021-01-15T17:02:56.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:56.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:56.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-490 td/osd-backfill-space/datafile 2021-01-15T17:02:56.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-490 td/osd-backfill-space/datafile 2021-01-15T17:02:56.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:56.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:56.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-491 td/osd-backfill-space/datafile 2021-01-15T17:02:56.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-491 td/osd-backfill-space/datafile 2021-01-15T17:02:56.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:56.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:56.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-492 td/osd-backfill-space/datafile 2021-01-15T17:02:56.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-492 td/osd-backfill-space/datafile 2021-01-15T17:02:56.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:56.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-493 td/osd-backfill-space/datafile 2021-01-15T17:02:56.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-493 td/osd-backfill-space/datafile 2021-01-15T17:02:57.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:57.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:57.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-494 td/osd-backfill-space/datafile 2021-01-15T17:02:57.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-494 td/osd-backfill-space/datafile 2021-01-15T17:02:57.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:57.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:57.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-495 td/osd-backfill-space/datafile 2021-01-15T17:02:57.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-495 td/osd-backfill-space/datafile 2021-01-15T17:02:57.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:57.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-496 td/osd-backfill-space/datafile 2021-01-15T17:02:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-496 td/osd-backfill-space/datafile 2021-01-15T17:02:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:57.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:57.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-497 td/osd-backfill-space/datafile 2021-01-15T17:02:57.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:57.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-497 td/osd-backfill-space/datafile 2021-01-15T17:02:58.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:58.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:58.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-498 td/osd-backfill-space/datafile 2021-01-15T17:02:58.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-498 td/osd-backfill-space/datafile 2021-01-15T17:02:58.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:58.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:58.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-499 td/osd-backfill-space/datafile 2021-01-15T17:02:58.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-499 td/osd-backfill-space/datafile 2021-01-15T17:02:58.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:58.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:58.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-500 td/osd-backfill-space/datafile 2021-01-15T17:02:58.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-500 td/osd-backfill-space/datafile 2021-01-15T17:02:58.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:58.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-501 td/osd-backfill-space/datafile 2021-01-15T17:02:59.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-501 td/osd-backfill-space/datafile 2021-01-15T17:02:59.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:59.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:59.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-502 td/osd-backfill-space/datafile 2021-01-15T17:02:59.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-502 td/osd-backfill-space/datafile 2021-01-15T17:02:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:59.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:59.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-503 td/osd-backfill-space/datafile 2021-01-15T17:02:59.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-503 td/osd-backfill-space/datafile 2021-01-15T17:02:59.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:59.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-504 td/osd-backfill-space/datafile 2021-01-15T17:02:59.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-504 td/osd-backfill-space/datafile 2021-01-15T17:02:59.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:02:59.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:02:59.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:02:59.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-505 td/osd-backfill-space/datafile 2021-01-15T17:03:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-505 td/osd-backfill-space/datafile 2021-01-15T17:03:00.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:00.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:00.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-506 td/osd-backfill-space/datafile 2021-01-15T17:03:00.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-506 td/osd-backfill-space/datafile 2021-01-15T17:03:00.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:00.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:00.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-507 td/osd-backfill-space/datafile 2021-01-15T17:03:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-507 td/osd-backfill-space/datafile 2021-01-15T17:03:00.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:00.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:00.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-508 td/osd-backfill-space/datafile 2021-01-15T17:03:00.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:00.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-508 td/osd-backfill-space/datafile 2021-01-15T17:03:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:01.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-509 td/osd-backfill-space/datafile 2021-01-15T17:03:01.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-509 td/osd-backfill-space/datafile 2021-01-15T17:03:01.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:01.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:01.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-510 td/osd-backfill-space/datafile 2021-01-15T17:03:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-510 td/osd-backfill-space/datafile 2021-01-15T17:03:01.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:01.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:01.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-511 td/osd-backfill-space/datafile 2021-01-15T17:03:01.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-511 td/osd-backfill-space/datafile 2021-01-15T17:03:01.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:01.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:01.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-512 td/osd-backfill-space/datafile 2021-01-15T17:03:01.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:01.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-512 td/osd-backfill-space/datafile 2021-01-15T17:03:02.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:02.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:02.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-513 td/osd-backfill-space/datafile 2021-01-15T17:03:02.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-513 td/osd-backfill-space/datafile 2021-01-15T17:03:02.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:02.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-514 td/osd-backfill-space/datafile 2021-01-15T17:03:02.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-514 td/osd-backfill-space/datafile 2021-01-15T17:03:02.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:02.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:02.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-515 td/osd-backfill-space/datafile 2021-01-15T17:03:02.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-515 td/osd-backfill-space/datafile 2021-01-15T17:03:02.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:02.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:02.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-516 td/osd-backfill-space/datafile 2021-01-15T17:03:02.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:02.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-516 td/osd-backfill-space/datafile 2021-01-15T17:03:03.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:03.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:03.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-517 td/osd-backfill-space/datafile 2021-01-15T17:03:03.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-517 td/osd-backfill-space/datafile 2021-01-15T17:03:03.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:03.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:03.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-518 td/osd-backfill-space/datafile 2021-01-15T17:03:03.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-518 td/osd-backfill-space/datafile 2021-01-15T17:03:03.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:03.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-519 td/osd-backfill-space/datafile 2021-01-15T17:03:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-519 td/osd-backfill-space/datafile 2021-01-15T17:03:03.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:03.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:03.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:03.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-520 td/osd-backfill-space/datafile 2021-01-15T17:03:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-520 td/osd-backfill-space/datafile 2021-01-15T17:03:04.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:04.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:04.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-521 td/osd-backfill-space/datafile 2021-01-15T17:03:04.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-521 td/osd-backfill-space/datafile 2021-01-15T17:03:04.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:04.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:04.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-522 td/osd-backfill-space/datafile 2021-01-15T17:03:04.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-522 td/osd-backfill-space/datafile 2021-01-15T17:03:04.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:04.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:04.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-523 td/osd-backfill-space/datafile 2021-01-15T17:03:04.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-523 td/osd-backfill-space/datafile 2021-01-15T17:03:04.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:04.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:04.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:04.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-524 td/osd-backfill-space/datafile 2021-01-15T17:03:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-524 td/osd-backfill-space/datafile 2021-01-15T17:03:05.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:05.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:05.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-525 td/osd-backfill-space/datafile 2021-01-15T17:03:05.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-525 td/osd-backfill-space/datafile 2021-01-15T17:03:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:05.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:05.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-526 td/osd-backfill-space/datafile 2021-01-15T17:03:05.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-526 td/osd-backfill-space/datafile 2021-01-15T17:03:05.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:05.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:05.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-527 td/osd-backfill-space/datafile 2021-01-15T17:03:05.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:05.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-527 td/osd-backfill-space/datafile 2021-01-15T17:03:06.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:06.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:06.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-528 td/osd-backfill-space/datafile 2021-01-15T17:03:06.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-528 td/osd-backfill-space/datafile 2021-01-15T17:03:06.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:06.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:06.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-529 td/osd-backfill-space/datafile 2021-01-15T17:03:06.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-529 td/osd-backfill-space/datafile 2021-01-15T17:03:06.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:06.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-530 td/osd-backfill-space/datafile 2021-01-15T17:03:06.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-530 td/osd-backfill-space/datafile 2021-01-15T17:03:06.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:06.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:06.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-531 td/osd-backfill-space/datafile 2021-01-15T17:03:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-531 td/osd-backfill-space/datafile 2021-01-15T17:03:07.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:07.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:07.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-532 td/osd-backfill-space/datafile 2021-01-15T17:03:07.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-532 td/osd-backfill-space/datafile 2021-01-15T17:03:07.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:07.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:07.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-533 td/osd-backfill-space/datafile 2021-01-15T17:03:07.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-533 td/osd-backfill-space/datafile 2021-01-15T17:03:07.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:07.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:07.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-534 td/osd-backfill-space/datafile 2021-01-15T17:03:07.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-534 td/osd-backfill-space/datafile 2021-01-15T17:03:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:07.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:07.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:07.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-535 td/osd-backfill-space/datafile 2021-01-15T17:03:08.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-535 td/osd-backfill-space/datafile 2021-01-15T17:03:08.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:08.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:08.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-536 td/osd-backfill-space/datafile 2021-01-15T17:03:08.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-536 td/osd-backfill-space/datafile 2021-01-15T17:03:08.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:08.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:08.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-537 td/osd-backfill-space/datafile 2021-01-15T17:03:08.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-537 td/osd-backfill-space/datafile 2021-01-15T17:03:08.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:08.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-538 td/osd-backfill-space/datafile 2021-01-15T17:03:08.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:08.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-538 td/osd-backfill-space/datafile 2021-01-15T17:03:09.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:09.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-539 td/osd-backfill-space/datafile 2021-01-15T17:03:09.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-539 td/osd-backfill-space/datafile 2021-01-15T17:03:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:09.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:09.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-540 td/osd-backfill-space/datafile 2021-01-15T17:03:09.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-540 td/osd-backfill-space/datafile 2021-01-15T17:03:09.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:09.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-541 td/osd-backfill-space/datafile 2021-01-15T17:03:09.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-541 td/osd-backfill-space/datafile 2021-01-15T17:03:09.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:09.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:09.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:09.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-542 td/osd-backfill-space/datafile 2021-01-15T17:03:10.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-542 td/osd-backfill-space/datafile 2021-01-15T17:03:10.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:10.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:10.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-543 td/osd-backfill-space/datafile 2021-01-15T17:03:10.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-543 td/osd-backfill-space/datafile 2021-01-15T17:03:10.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:10.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:10.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-544 td/osd-backfill-space/datafile 2021-01-15T17:03:10.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-544 td/osd-backfill-space/datafile 2021-01-15T17:03:10.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:10.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:10.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-545 td/osd-backfill-space/datafile 2021-01-15T17:03:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-545 td/osd-backfill-space/datafile 2021-01-15T17:03:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:11.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:11.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-546 td/osd-backfill-space/datafile 2021-01-15T17:03:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-546 td/osd-backfill-space/datafile 2021-01-15T17:03:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:11.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-547 td/osd-backfill-space/datafile 2021-01-15T17:03:11.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-547 td/osd-backfill-space/datafile 2021-01-15T17:03:11.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:11.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:11.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-548 td/osd-backfill-space/datafile 2021-01-15T17:03:11.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-548 td/osd-backfill-space/datafile 2021-01-15T17:03:11.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:11.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:11.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-549 td/osd-backfill-space/datafile 2021-01-15T17:03:11.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:11.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-549 td/osd-backfill-space/datafile 2021-01-15T17:03:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:12.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:12.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-550 td/osd-backfill-space/datafile 2021-01-15T17:03:12.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-550 td/osd-backfill-space/datafile 2021-01-15T17:03:12.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:12.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:12.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-551 td/osd-backfill-space/datafile 2021-01-15T17:03:12.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-551 td/osd-backfill-space/datafile 2021-01-15T17:03:12.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:12.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:12.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-552 td/osd-backfill-space/datafile 2021-01-15T17:03:12.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-552 td/osd-backfill-space/datafile 2021-01-15T17:03:12.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:12.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:12.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:12.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-553 td/osd-backfill-space/datafile 2021-01-15T17:03:13.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-553 td/osd-backfill-space/datafile 2021-01-15T17:03:13.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:13.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:13.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-554 td/osd-backfill-space/datafile 2021-01-15T17:03:13.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-554 td/osd-backfill-space/datafile 2021-01-15T17:03:13.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:13.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:13.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-555 td/osd-backfill-space/datafile 2021-01-15T17:03:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-555 td/osd-backfill-space/datafile 2021-01-15T17:03:13.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:13.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:13.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-556 td/osd-backfill-space/datafile 2021-01-15T17:03:13.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:13.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-556 td/osd-backfill-space/datafile 2021-01-15T17:03:14.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:14.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-557 td/osd-backfill-space/datafile 2021-01-15T17:03:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-557 td/osd-backfill-space/datafile 2021-01-15T17:03:14.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:14.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:14.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-558 td/osd-backfill-space/datafile 2021-01-15T17:03:14.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-558 td/osd-backfill-space/datafile 2021-01-15T17:03:14.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:14.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:14.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-559 td/osd-backfill-space/datafile 2021-01-15T17:03:14.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-559 td/osd-backfill-space/datafile 2021-01-15T17:03:14.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:14.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:14.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:14.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-560 td/osd-backfill-space/datafile 2021-01-15T17:03:15.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-560 td/osd-backfill-space/datafile 2021-01-15T17:03:15.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:15.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:15.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-561 td/osd-backfill-space/datafile 2021-01-15T17:03:15.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-561 td/osd-backfill-space/datafile 2021-01-15T17:03:15.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:15.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-562 td/osd-backfill-space/datafile 2021-01-15T17:03:15.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-562 td/osd-backfill-space/datafile 2021-01-15T17:03:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:15.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:15.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-563 td/osd-backfill-space/datafile 2021-01-15T17:03:15.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:15.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-563 td/osd-backfill-space/datafile 2021-01-15T17:03:15.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:15.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:16.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-564 td/osd-backfill-space/datafile 2021-01-15T17:03:16.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-564 td/osd-backfill-space/datafile 2021-01-15T17:03:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:16.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:16.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-565 td/osd-backfill-space/datafile 2021-01-15T17:03:16.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-565 td/osd-backfill-space/datafile 2021-01-15T17:03:16.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:16.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-566 td/osd-backfill-space/datafile 2021-01-15T17:03:16.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-566 td/osd-backfill-space/datafile 2021-01-15T17:03:16.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:16.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:16.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-567 td/osd-backfill-space/datafile 2021-01-15T17:03:16.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:16.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-567 td/osd-backfill-space/datafile 2021-01-15T17:03:17.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:17.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:17.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-568 td/osd-backfill-space/datafile 2021-01-15T17:03:17.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-568 td/osd-backfill-space/datafile 2021-01-15T17:03:17.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:17.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:17.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-569 td/osd-backfill-space/datafile 2021-01-15T17:03:17.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-569 td/osd-backfill-space/datafile 2021-01-15T17:03:17.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:17.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:17.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-570 td/osd-backfill-space/datafile 2021-01-15T17:03:17.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-570 td/osd-backfill-space/datafile 2021-01-15T17:03:17.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:17.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-571 td/osd-backfill-space/datafile 2021-01-15T17:03:18.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-571 td/osd-backfill-space/datafile 2021-01-15T17:03:18.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:18.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:18.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-572 td/osd-backfill-space/datafile 2021-01-15T17:03:18.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-572 td/osd-backfill-space/datafile 2021-01-15T17:03:18.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:18.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:18.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-573 td/osd-backfill-space/datafile 2021-01-15T17:03:18.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-573 td/osd-backfill-space/datafile 2021-01-15T17:03:18.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:18.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:18.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-574 td/osd-backfill-space/datafile 2021-01-15T17:03:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-574 td/osd-backfill-space/datafile 2021-01-15T17:03:18.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:18.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:18.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-575 td/osd-backfill-space/datafile 2021-01-15T17:03:19.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-575 td/osd-backfill-space/datafile 2021-01-15T17:03:19.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:19.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:19.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-576 td/osd-backfill-space/datafile 2021-01-15T17:03:19.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-576 td/osd-backfill-space/datafile 2021-01-15T17:03:19.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:19.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:19.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-577 td/osd-backfill-space/datafile 2021-01-15T17:03:19.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-577 td/osd-backfill-space/datafile 2021-01-15T17:03:19.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:19.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:19.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-578 td/osd-backfill-space/datafile 2021-01-15T17:03:19.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:19.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-578 td/osd-backfill-space/datafile 2021-01-15T17:03:20.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:20.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:20.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-579 td/osd-backfill-space/datafile 2021-01-15T17:03:20.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-579 td/osd-backfill-space/datafile 2021-01-15T17:03:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:20.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:20.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-580 td/osd-backfill-space/datafile 2021-01-15T17:03:20.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-580 td/osd-backfill-space/datafile 2021-01-15T17:03:20.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:20.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:20.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-581 td/osd-backfill-space/datafile 2021-01-15T17:03:20.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:20.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-581 td/osd-backfill-space/datafile 2021-01-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:21.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-582 td/osd-backfill-space/datafile 2021-01-15T17:03:21.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-582 td/osd-backfill-space/datafile 2021-01-15T17:03:21.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:21.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:21.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-583 td/osd-backfill-space/datafile 2021-01-15T17:03:21.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-583 td/osd-backfill-space/datafile 2021-01-15T17:03:21.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:21.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-584 td/osd-backfill-space/datafile 2021-01-15T17:03:21.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:21.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-584 td/osd-backfill-space/datafile 2021-01-15T17:03:22.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:22.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:22.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-585 td/osd-backfill-space/datafile 2021-01-15T17:03:22.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-585 td/osd-backfill-space/datafile 2021-01-15T17:03:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:22.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:22.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-586 td/osd-backfill-space/datafile 2021-01-15T17:03:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-586 td/osd-backfill-space/datafile 2021-01-15T17:03:22.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:22.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:22.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-587 td/osd-backfill-space/datafile 2021-01-15T17:03:22.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-587 td/osd-backfill-space/datafile 2021-01-15T17:03:22.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:22.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:22.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-588 td/osd-backfill-space/datafile 2021-01-15T17:03:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:22.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-588 td/osd-backfill-space/datafile 2021-01-15T17:03:23.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:23.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:23.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-589 td/osd-backfill-space/datafile 2021-01-15T17:03:23.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-589 td/osd-backfill-space/datafile 2021-01-15T17:03:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:23.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-590 td/osd-backfill-space/datafile 2021-01-15T17:03:23.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-590 td/osd-backfill-space/datafile 2021-01-15T17:03:23.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:23.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:23.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-591 td/osd-backfill-space/datafile 2021-01-15T17:03:23.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-591 td/osd-backfill-space/datafile 2021-01-15T17:03:23.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:23.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:23.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:23.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-592 td/osd-backfill-space/datafile 2021-01-15T17:03:24.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-592 td/osd-backfill-space/datafile 2021-01-15T17:03:24.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:24.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-593 td/osd-backfill-space/datafile 2021-01-15T17:03:24.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-593 td/osd-backfill-space/datafile 2021-01-15T17:03:24.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:24.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:24.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-594 td/osd-backfill-space/datafile 2021-01-15T17:03:24.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-594 td/osd-backfill-space/datafile 2021-01-15T17:03:24.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:24.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:24.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-595 td/osd-backfill-space/datafile 2021-01-15T17:03:24.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:24.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-595 td/osd-backfill-space/datafile 2021-01-15T17:03:25.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:25.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:25.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-596 td/osd-backfill-space/datafile 2021-01-15T17:03:25.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-596 td/osd-backfill-space/datafile 2021-01-15T17:03:25.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:25.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:25.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-597 td/osd-backfill-space/datafile 2021-01-15T17:03:25.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-597 td/osd-backfill-space/datafile 2021-01-15T17:03:25.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:25.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-598 td/osd-backfill-space/datafile 2021-01-15T17:03:25.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-598 td/osd-backfill-space/datafile 2021-01-15T17:03:25.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:25.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:25.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-599 td/osd-backfill-space/datafile 2021-01-15T17:03:25.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:25.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-599 td/osd-backfill-space/datafile 2021-01-15T17:03:26.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:500: TEST_backfill_multi_partial: for o in $(seq 1 $objects) 2021-01-15T17:03:26.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:03:26.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:26.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test1 put obj-1-600 td/osd-backfill-space/datafile 2021-01-15T17:03:26.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:502: TEST_backfill_multi_partial: for p in $(seq 1 $pools) 2021-01-15T17:03:26.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:504: TEST_backfill_multi_partial: rados -p test2 put obj-2-600 td/osd-backfill-space/datafile 2021-01-15T17:03:26.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:508: TEST_backfill_multi_partial: kill_daemon td/osd-backfill-space/osd.2.pid TERM 2021-01-15T17:03:26.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.2.pid 2021-01-15T17:03:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=237063 2021-01-15T17:03:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2021-01-15T17:03:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2021-01-15T17:03:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2021-01-15T17:03:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:03:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 237063 2021-01-15T17:03:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2021-01-15T17:03:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2021-01-15T17:03:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2021-01-15T17:03:26.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:03:26.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 237063 2021-01-15T17:03:26.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2021-01-15T17:03:26.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2021-01-15T17:03:26.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2021-01-15T17:03:26.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:509: TEST_backfill_multi_partial: ceph osd out osd.2 2021-01-15T17:03:27.290 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T17:03:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:511: TEST_backfill_multi_partial: activate_osd td/osd-backfill-space 1 2021-01-15T17:03:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T17:03:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T17:03:27.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T17:03:27.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T17:03:27.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:03:27.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:03:27.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:03:27.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:03:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:03:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:03:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:03:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T17:03:27.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T17:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T17:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:03:27.306 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:03:27.306 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:03:27.306 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:03:27.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:03:27.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:03:27.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:03:27.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:03:27.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:03:27.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:03:27.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:03:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:03:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T17:03:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T17:03:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:03:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T17:03:27.309 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:03:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:03:27.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2021-01-15T17:03:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T17:03:27.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T17:03:27.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T17:03:27.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T17:03:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:03:27.327+0000 7f87a5ee5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:03:27.329 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:03:27.329+0000 7f87a5ee5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:03:27.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:03:27.330+0000 7f87a5ee5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:03:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T17:03:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:03:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:03:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:03:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:03:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:03:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:03:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:03:27.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:03:27.700 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:03:27.853 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:03:27.853+0000 7f87a5ee5080 -1 Falling back to public interface 2021-01-15T17:03:28.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:03:29.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:03:29.124+0000 7f87a5ee5080 -1 osd.1 44 log_to_monitors {default=true} 2021-01-15T17:03:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:03:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:03:29.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:03:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:03:29.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:03:29.133 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:03:29.586 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up out weight 0 up_from 49 up_thru 33 down_at 45 last_clean_interval [40,44) [v2:127.0.0.1:6810/263346,v1:127.0.0.1:6811/263346] [v2:127.0.0.1:6812/263346,v1:127.0.0.1:6813/263346] exists,up ba9894f0-c157-464b-ad69-8661d42507f9 2021-01-15T17:03:29.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:03:29.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:03:29.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:03:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:512: TEST_backfill_multi_partial: ceph osd in osd.1 2021-01-15T17:03:30.196 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already in. 2021-01-15T17:03:30.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:513: TEST_backfill_multi_partial: sleep 30 2021-01-15T17:04:00.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:515: TEST_backfill_multi_partial: wait_for_not_backfilling 240 2021-01-15T17:04:00.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:04:00.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:04:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:04:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:04:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:04:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:04:00.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:04:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:04:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:04:00.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:04:00.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:04:00.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:04:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:04:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:04:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:04:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:04:00.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:04:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:04:00.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:04:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836572 2021-01-15T17:04:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836572 2021-01-15T17:04:00.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836572' 2021-01-15T17:04:00.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:04:00.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:04:01.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397513 2021-01-15T17:04:01.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397513 2021-01-15T17:04:01.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836572 1-210453397513' 2021-01-15T17:04:01.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:04:01.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:04:01.122 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-01-15T17:04:01.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:04:01.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:04:01.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:04:01.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:04:01.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836572 2021-01-15T17:04:01.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:04:01.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:04:01.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836572 2021-01-15T17:04:01.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:04:01.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836572 2021-01-15T17:04:01.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836572' 2021-01-15T17:04:01.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:04:01.127 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836572 2021-01-15T17:04:01.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836570 -lt 21474836572 2021-01-15T17:04:01.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:04:02.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:04:02.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:04:02.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836572 -lt 21474836572 2021-01-15T17:04:02.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:04:02.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397513 2021-01-15T17:04:02.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:04:02.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:04:02.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397513 2021-01-15T17:04:02.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:04:02.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397513 2021-01-15T17:04:02.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397513' 2021-01-15T17:04:02.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:04:02.948 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 210453397513 2021-01-15T17:04:03.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397513 -lt 210453397513 2021-01-15T17:04:03.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:04:03.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:04:03.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:04:03.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2021-01-15T17:04:03.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:04:03.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:04:03.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:04:03.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:04:03.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:04:03.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:04:03.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:04:04.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:04:04.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:04:04.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2021-01-15T17:04:04.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:04:04.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2021-01-15T17:04:04.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:04:09.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:04:09.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:04:09.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:04:09.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:04:09.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:04:09.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:04:09.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:04:09.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:04:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:04:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:04:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:04:09.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2021-01-15T17:04:09.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2021-01-15T17:04:19.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:04:19.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:04:19.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:04:19.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:04:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:04:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:04:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:04:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:04:20.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:04:20.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:04:20.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:04:20.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2021-01-15T17:04:20.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:04:35.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:04:35.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:04:35.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:04:35.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:04:35.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:04:35.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:04:35.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:04:35.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:04:35.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:04:35.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:04:35.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:04:35.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2021-01-15T17:04:35.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:04:50.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:04:50.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:04:50.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:04:50.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:04:50.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:04:50.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:04:50.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:04:50.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:04:50.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:04:50.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:04:50.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:04:50.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2021-01-15T17:04:50.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:05:05.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:05:05.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:05:05.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:05:05.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:05:05.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:05:05.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:05:05.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:05:05.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:05:06.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:05:06.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:05:06.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:05:06.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:05:06.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:516: TEST_backfill_multi_partial: wait_for_not_activating 60 2021-01-15T17:05:06.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:05:06.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:05:06.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:05:06.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:05:06.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:05:06.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:05:06.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:05:06.240 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:05:06.241 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:05:06.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:05:06.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:05:06.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:05:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:05:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:05:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:05:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:05:06.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:05:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:05:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:05:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:05:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:05:06.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:06.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:05:06.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836587 2021-01-15T17:05:06.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836587 2021-01-15T17:05:06.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836587' 2021-01-15T17:05:06.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:06.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:05:06.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397528 2021-01-15T17:05:06.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397528 2021-01-15T17:05:06.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836587 1-210453397528' 2021-01-15T17:05:06.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:06.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:05:07.062 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-01-15T17:05:07.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:05:07.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:05:07.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:05:07.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:05:07.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836587 2021-01-15T17:05:07.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:05:07.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:05:07.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836587 2021-01-15T17:05:07.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:05:07.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836587 2021-01-15T17:05:07.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836587' 2021-01-15T17:05:07.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:05:07.067 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836587 2021-01-15T17:05:07.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836585 -lt 21474836587 2021-01-15T17:05:07.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:05:08.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:05:08.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:05:08.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836587 -lt 21474836587 2021-01-15T17:05:08.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:05:08.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397528 2021-01-15T17:05:08.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:05:08.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:05:08.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397528 2021-01-15T17:05:08.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:05:08.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397528 2021-01-15T17:05:08.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397528' 2021-01-15T17:05:08.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:05:08.865 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 210453397528 2021-01-15T17:05:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397528 -lt 210453397528 2021-01-15T17:05:09.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:05:09.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:05:09.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:05:09.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2021-01-15T17:05:09.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:05:09.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:05:09.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:05:09.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:05:09.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:05:09.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:05:09.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:05:10.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:05:10.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:05:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:05:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:05:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:518: TEST_backfill_multi_partial: flush_pg_stats 2021-01-15T17:05:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:05:10.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:05:10.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:05:10.614 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:05:10.614 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:05:10.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:05:10.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:10.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:05:10.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836590 2021-01-15T17:05:10.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836590 2021-01-15T17:05:10.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590' 2021-01-15T17:05:10.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:10.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:05:10.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397531 2021-01-15T17:05:10.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397531 2021-01-15T17:05:10.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836590 1-210453397531' 2021-01-15T17:05:10.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:05:10.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:05:10.994 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-01-15T17:05:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:05:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:05:10.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:05:10.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:05:10.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836590 2021-01-15T17:05:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:05:10.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:05:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836590 2021-01-15T17:05:10.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:05:10.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836590 2021-01-15T17:05:10.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2021-01-15T17:05:10.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:05:10.999 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836590 2021-01-15T17:05:11.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836588 -lt 21474836590 2021-01-15T17:05:11.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:05:12.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:05:12.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:05:12.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836590 -lt 21474836590 2021-01-15T17:05:12.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:05:12.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397531 2021-01-15T17:05:12.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:05:12.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:05:12.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397531 2021-01-15T17:05:12.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:05:12.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397531 2021-01-15T17:05:12.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397531' 2021-01-15T17:05:12.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:05:12.795 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 210453397531 2021-01-15T17:05:13.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397531 -lt 210453397531 2021-01-15T17:05:13.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:519: TEST_backfill_multi_partial: ceph pg dump pgs 2021-01-15T17:05:13.600 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:05:13.601 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 600 0 600 0 0 1566600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:05:01.210068+0000 45'600 55:671 [1,0] 1 [0] 0 0'0 2021-01-15T16:57:07.000190+0000 0'0 2021-01-15T16:57:07.000190+0000 0 2021-01-15T17:05:13.601 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 0 0 0 1566600 0 0 100 100 active+clean 2021-01-15T17:03:30.551293+0000 35'600 55:650 [1] 1 [1] 1 0'0 2021-01-15T16:57:01.996196+0000 0'0 2021-01-15T16:57:01.996196+0000 0 2021-01-15T17:05:13.601 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 600 0 0 0 0 1566600 0 0 10 10 active+clean 2021-01-15T17:05:03.463242+0000 45'1200 56:2487 [1,0] 1 [1,0] 1 0'0 2021-01-15T16:57:04.485874+0000 0'0 2021-01-15T16:57:04.485874+0000 0 2021-01-15T17:05:13.601 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:05:13.602 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:05:13.602 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:05:13.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:521: TEST_backfill_multi_partial: ERRORS=0 2021-01-15T17:05:13.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:522: TEST_backfill_multi_partial: get_num_in_state backfill_toofull 2021-01-15T17:05:13.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfill_toofull 2021-01-15T17:05:13.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:05:13.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfill_toofull"))' 2021-01-15T17:05:13.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:05:13.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfill_toofull"))] | length' 2021-01-15T17:05:13.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:522: TEST_backfill_multi_partial: '[' 1 '!=' 1 ']' 2021-01-15T17:05:13.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:528: TEST_backfill_multi_partial: get_num_in_state active+clean 2021-01-15T17:05:13.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=active+clean 2021-01-15T17:05:13.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:05:13.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("active+clean"))' 2021-01-15T17:05:13.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:05:13.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("active+clean"))] | length' 2021-01-15T17:05:14.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:528: TEST_backfill_multi_partial: '[' 2 '!=' 2 ']' 2021-01-15T17:05:14.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:534: TEST_backfill_multi_partial: '[' 0 '!=' 0 ']' 2021-01-15T17:05:14.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:539: TEST_backfill_multi_partial: delete_pool fillpool 2021-01-15T17:05:14.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2021-01-15T17:05:14.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2021-01-15T17:05:14.934 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' does not exist 2021-01-15T17:05:14.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: seq 1 2 2021-01-15T17:05:14.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: for i in $(seq 1 $pools) 2021-01-15T17:05:14.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:542: TEST_backfill_multi_partial: delete_pool test1 2021-01-15T17:05:14.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:05:14.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:05:15.488 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:05:15.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:540: TEST_backfill_multi_partial: for i in $(seq 1 $pools) 2021-01-15T17:05:15.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:542: TEST_backfill_multi_partial: delete_pool test2 2021-01-15T17:05:15.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T17:05:15.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T17:05:16.064 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T17:05:16.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:544: TEST_backfill_multi_partial: kill_daemons td/osd-backfill-space 2021-01-15T17:05:16.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:05:16.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:05:16.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:05:16.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:05:16.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:05:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:05:16.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:545: TEST_backfill_multi_partial: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2021-01-15T17:05:16.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:05:16.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:05:16.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:05:16.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:05:16.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:05:16.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:05:16.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:05:16.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:05:16.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:05:16.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:05:16.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:05:16.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:05:16.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:05:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:05:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:05:16.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:05:16.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:05:16.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:05:16.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:05:16.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:05:16.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:05:16.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:05:16.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:05:16.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:05:16.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:05:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:05:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:05:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:05:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:05:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:05:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:05:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:05:16.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:05:16.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:05:16.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:05:16.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:05:16.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:05:16.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:05:16.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:05:16.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:05:16.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:05:16.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:05:16.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:05:16.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:05:16.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:05:16.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:05:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:05:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:05:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:05:16.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:05:16.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:05:16.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:05:16.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:05:16.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:05:16.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:05:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:05:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:05:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:05:16.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:05:16.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:05:16.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:05:16.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:05:16.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:05:16.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:05:16.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_multi td/osd-backfill-space 2021-01-15T17:05:16.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:191: TEST_backfill_test_multi: local dir=td/osd-backfill-space 2021-01-15T17:05:16.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:192: TEST_backfill_test_multi: local pools=8 2021-01-15T17:05:16.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:193: TEST_backfill_test_multi: local OSDS=20 2021-01-15T17:05:16.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:195: TEST_backfill_test_multi: run_mon td/osd-backfill-space a 2021-01-15T17:05:16.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:05:16.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:05:16.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:05:16.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:05:16.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:05:16.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:05:16.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:05:16.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:16.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:16.680 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:16.680 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.681 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:16.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:05:16.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:05:16.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:05:16.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:05:16.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:05:16.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:05:16.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:05:16.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:05:16.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:05:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:05:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:05:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:05:16.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:05:16.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:05:16.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:05:16.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:05:16.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:05:16.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:05:16.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:05:16.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:05:16.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:05:16.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:05:16.961 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:05:16.961 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:16.962 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:16.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:05:16.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:05:16.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:05:17.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:196: TEST_backfill_test_multi: run_mgr td/osd-backfill-space x 2021-01-15T17:05:17.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:05:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:05:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:05:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:05:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:05:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:05:17.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:05:17.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:17.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:17.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:17.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:05:17.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:05:17.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:197: TEST_backfill_test_multi: export CEPH_ARGS 2021-01-15T17:05:17.998 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: expr 20 - 1 2021-01-15T17:05:17.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: seq 0 19 2021-01-15T17:05:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 0 2021-01-15T17:05:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:17.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:05:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:05:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:05:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:05:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:18.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:18.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:18.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:18.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:18.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:18.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:18.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:18.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:05:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:18.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ab9b1061-84c5-4817-829a-4a75903ab59d 2021-01-15T17:05:18.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 ab9b1061-84c5-4817-829a-4a75903ab59d' 2021-01-15T17:05:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:18.007 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 ab9b1061-84c5-4817-829a-4a75903ab59d 2021-01-15T17:05:18.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBOywFg8//bABAAv4IcIThAfQrI9betZcg03g== 2021-01-15T17:05:18.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBOywFg8//bABAAv4IcIThAfQrI9betZcg03g=="}' 2021-01-15T17:05:18.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ab9b1061-84c5-4817-829a-4a75903ab59d -i td/osd-backfill-space/0/new.json 2021-01-15T17:05:18.357 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:18.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:05:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBOywFg8//bABAAv4IcIThAfQrI9betZcg03g== --osd-uuid ab9b1061-84c5-4817-829a-4a75903ab59d 2021-01-15T17:05:18.396 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.396+0000 7fc03c829080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:18.398 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.398+0000 7fc03c829080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:18.399 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.399+0000 7fc03c829080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:18.399 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.399+0000 7fc03c829080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:05:18.400 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.399+0000 7fc03c829080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:05:18.400 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.399+0000 7fc03c829080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:05:18.497 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:18.497+0000 7fc03c829080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:05:23.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:05:23.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:05:23.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:05:23.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:05:23.064 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:05:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:05:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:05:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:05:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:05:23.595 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:05:23.619 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:23.618+0000 7fdf5dbd2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:23.645 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:23.645+0000 7fdf5dbd2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:23.646 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:23.646+0000 7fdf5dbd2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:05:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:05:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:05:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:05:24.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:05:24.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:24.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:05:24.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:05:24.009 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:24.165 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:24.165+0000 7fdf5dbd2080 -1 Falling back to public interface 2021-01-15T17:05:24.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:24.997 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:24.997+0000 7fdf5dbd2080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:05:25.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:05:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:05:25.406 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:25.818 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/266062,v1:127.0.0.1:6803/266062] [v2:127.0.0.1:6804/266062,v1:127.0.0.1:6805/266062] exists,up ab9b1061-84c5-4817-829a-4a75903ab59d 2021-01-15T17:05:25.818 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 e37b9115-717c-4669-9a91-94d8155bcb48 2021-01-15T17:05:25.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:05:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:05:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:05:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 1 2021-01-15T17:05:25.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:05:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:05:25.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:05:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:05:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:25.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:25.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:25.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:25.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:25.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:25.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:25.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:25.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:25.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:05:25.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:25.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e37b9115-717c-4669-9a91-94d8155bcb48 2021-01-15T17:05:25.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 e37b9115-717c-4669-9a91-94d8155bcb48' 2021-01-15T17:05:25.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:25.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBVywFgjQ++MRAAw7Ajr6XlDAuCAZPcNDnDMw== 2021-01-15T17:05:25.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBVywFgjQ++MRAAw7Ajr6XlDAuCAZPcNDnDMw=="}' 2021-01-15T17:05:25.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e37b9115-717c-4669-9a91-94d8155bcb48 -i td/osd-backfill-space/1/new.json 2021-01-15T17:05:26.422 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:26.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:05:26.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBVywFgjQ++MRAAw7Ajr6XlDAuCAZPcNDnDMw== --osd-uuid e37b9115-717c-4669-9a91-94d8155bcb48 2021-01-15T17:05:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.462+0000 7fb84f565080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.463+0000 7fb84f565080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:26.464 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.465+0000 7fb84f565080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:26.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.465+0000 7fb84f565080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:05:26.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.465+0000 7fb84f565080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:05:26.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.465+0000 7fb84f565080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:05:26.519 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:26.518+0000 7fb84f565080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:05:31.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:05:31.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:05:31.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:05:31.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:05:31.111 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:05:31.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:05:31.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:05:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:05:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:05:31.645 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:05:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:31.667+0000 7f01067b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:31.692 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:31.691+0000 7f01067b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:31.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:31.693+0000 7f01067b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:32.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:05:32.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:05:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:05:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:05:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:05:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:05:32.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:32.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:05:32.064 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:32.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:32.208+0000 7f01067b1080 -1 Falling back to public interface 2021-01-15T17:05:32.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:33.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:33.009+0000 7f01067b1080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:05:33.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:33.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:33.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:05:33.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:33.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:05:33.460 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:33.888 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/267326,v1:127.0.0.1:6811/267326] [v2:127.0.0.1:6812/267326,v1:127.0.0.1:6813/267326] exists,up e37b9115-717c-4669-9a91-94d8155bcb48 2021-01-15T17:05:33.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:05:33.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:05:33.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:05:33.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 2 2021-01-15T17:05:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:05:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:05:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:05:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:05:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:33.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:33.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:33.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:33.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:33.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:33.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:33.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:33.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:33.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:33.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:33.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:33.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:33.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:05:33.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:33.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2a332673-463e-4f6a-ad39-b75f83e65b27 2021-01-15T17:05:33.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 2a332673-463e-4f6a-ad39-b75f83e65b27' 2021-01-15T17:05:33.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:33.899 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 2a332673-463e-4f6a-ad39-b75f83e65b27 2021-01-15T17:05:33.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBdywFgc8oBNhAASw7BapPG1/TULvsfHYNdKw== 2021-01-15T17:05:33.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBdywFgc8oBNhAASw7BapPG1/TULvsfHYNdKw=="}' 2021-01-15T17:05:33.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2a332673-463e-4f6a-ad39-b75f83e65b27 -i td/osd-backfill-space/2/new.json 2021-01-15T17:05:34.492 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:05:34.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:05:34.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBdywFgc8oBNhAASw7BapPG1/TULvsfHYNdKw== --osd-uuid 2a332673-463e-4f6a-ad39-b75f83e65b27 2021-01-15T17:05:34.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.533+0000 7f7f8e856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:34.534 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.534+0000 7f7f8e856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:34.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.535+0000 7f7f8e856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:34.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.537+0000 7f7f8e856080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:05:34.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.537+0000 7f7f8e856080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:05:34.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.537+0000 7f7f8e856080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:05:34.574 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:34.574+0000 7f7f8e856080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:05:39.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:05:39.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:05:39.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:05:39.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:05:39.334 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:05:39.863 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:05:39.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:05:39.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:39.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:05:39.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:05:39.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:05:39.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:39.889+0000 7f1bd7fa1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:39.890 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:39.890+0000 7f1bd7fa1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:39.892 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:39.891+0000 7f1bd7fa1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:40.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:05:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:05:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:05:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:05:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:05:40.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:40.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:05:40.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:40.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:05:40.296 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:40.414 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:40.414+0000 7f1bd7fa1080 -1 Falling back to public interface 2021-01-15T17:05:40.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:41.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:41.191+0000 7f1bd7fa1080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:05:41.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:05:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:05:41.695 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:42.127 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/268598,v1:127.0.0.1:6819/268598] [v2:127.0.0.1:6820/268598,v1:127.0.0.1:6821/268598] exists,up 2a332673-463e-4f6a-ad39-b75f83e65b27 2021-01-15T17:05:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:05:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:05:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:05:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 3 2021-01-15T17:05:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:05:42.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:42.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2021-01-15T17:05:42.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:42.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:42.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:42.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:42.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2021-01-15T17:05:42.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2021-01-15T17:05:42.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:42.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:42.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:42.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:42.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:42.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:42.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:42.133 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:42.133 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:42.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:42.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:42.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:42.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:42.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:42.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:42.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:42.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2021-01-15T17:05:42.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5df739dc-adbc-426d-9aa6-dd9a2a592f99 2021-01-15T17:05:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 5df739dc-adbc-426d-9aa6-dd9a2a592f99' 2021-01-15T17:05:42.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:42.137 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 5df739dc-adbc-426d-9aa6-dd9a2a592f99 2021-01-15T17:05:42.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBmywFgJZqoCBAASPYxjE/r/Xn4Nh/hx/1bgg== 2021-01-15T17:05:42.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBmywFgJZqoCBAASPYxjE/r/Xn4Nh/hx/1bgg=="}' 2021-01-15T17:05:42.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5df739dc-adbc-426d-9aa6-dd9a2a592f99 -i td/osd-backfill-space/3/new.json 2021-01-15T17:05:42.657 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:05:42.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2021-01-15T17:05:42.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBmywFgJZqoCBAASPYxjE/r/Xn4Nh/hx/1bgg== --osd-uuid 5df739dc-adbc-426d-9aa6-dd9a2a592f99 2021-01-15T17:05:42.696 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.696+0000 7f75fa408080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:42.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.697+0000 7f75fa408080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:42.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.698+0000 7f75fa408080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:42.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.699+0000 7f75fa408080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:05:42.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.699+0000 7f75fa408080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:05:42.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.699+0000 7f75fa408080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:05:42.797 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:42.797+0000 7f75fa408080 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2021-01-15T17:05:47.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2021-01-15T17:05:47.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:05:47.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:05:47.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:05:47.555 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:05:48.087 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:05:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:05:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:05:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:05:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:05:48.112 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:48.112+0000 7f506c96a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:48.114 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:48.114+0000 7f506c96a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:48.115 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:48.115+0000 7f506c96a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:48.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:05:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:05:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:05:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:05:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:05:48.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:48.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:05:48.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:48.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:05:48.489 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:48.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:48.633+0000 7f506c96a080 -1 Falling back to public interface 2021-01-15T17:05:48.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:49.423 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:49.423+0000 7f506c96a080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:05:49.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:49.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:49.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:05:49.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:49.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:05:49.899 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:50.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:51.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:05:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:05:51.378 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:05:51.786 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/269862,v1:127.0.0.1:6827/269862] [v2:127.0.0.1:6828/269862,v1:127.0.0.1:6829/269862] exists,up 5df739dc-adbc-426d-9aa6-dd9a2a592f99 2021-01-15T17:05:51.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:05:51.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:05:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:05:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 4 2021-01-15T17:05:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:51.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:51.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:05:51.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:51.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2021-01-15T17:05:51.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:51.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2021-01-15T17:05:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2021-01-15T17:05:51.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:51.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:51.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:51.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:51.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:51.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:51.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:51.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:51.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:51.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:51.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:51.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2021-01-15T17:05:51.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e0bad634-b2bd-4667-a260-9cdb1d16f8b4 2021-01-15T17:05:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 e0bad634-b2bd-4667-a260-9cdb1d16f8b4' 2021-01-15T17:05:51.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:51.795 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 e0bad634-b2bd-4667-a260-9cdb1d16f8b4 2021-01-15T17:05:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBvywFgbYLnLxAAr45oNiXkDryMalgZoYezcA== 2021-01-15T17:05:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBvywFgbYLnLxAAr45oNiXkDryMalgZoYezcA=="}' 2021-01-15T17:05:51.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e0bad634-b2bd-4667-a260-9cdb1d16f8b4 -i td/osd-backfill-space/4/new.json 2021-01-15T17:05:52.338 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:05:52.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2021-01-15T17:05:52.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBvywFgbYLnLxAAr45oNiXkDryMalgZoYezcA== --osd-uuid e0bad634-b2bd-4667-a260-9cdb1d16f8b4 2021-01-15T17:05:52.376 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.376+0000 7f32b8d17080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:52.378 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.378+0000 7f32b8d17080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.379+0000 7f32b8d17080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:52.380 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.380+0000 7f32b8d17080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:05:52.380 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.380+0000 7f32b8d17080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:05:52.380 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.380+0000 7f32b8d17080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:05:52.509 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:52.509+0000 7f32b8d17080 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2021-01-15T17:05:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2021-01-15T17:05:57.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:05:57.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:05:57.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:05:57.056 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:05:57.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:05:57.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:57.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:05:57.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:05:57.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:05:57.588 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:05:57.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:57.612+0000 7fa7ea2c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:57.614 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:57.614+0000 7fa7ea2c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:57.615 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:57.615+0000 7fa7ea2c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:05:57.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:05:57.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:05:57.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:05:57.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:05:57.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:05:57.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:57.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:05:57.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:57.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:05:57.988 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:05:58.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:58.132+0000 7fa7ea2c4080 -1 Falling back to public interface 2021-01-15T17:05:58.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:05:58.994 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:05:58.993+0000 7fa7ea2c4080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:05:59.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:05:59.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:05:59.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:05:59.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:05:59.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:05:59.441 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:05:59.846 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/271152,v1:127.0.0.1:6835/271152] [v2:127.0.0.1:6836/271152,v1:127.0.0.1:6837/271152] exists,up e0bad634-b2bd-4667-a260-9cdb1d16f8b4 2021-01-15T17:05:59.847 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 23ba462f-a1ac-4a4d-9376-8378d098e230 2021-01-15T17:05:59.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:05:59.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:05:59.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:05:59.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:05:59.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 5 2021-01-15T17:05:59.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:05:59.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:05:59.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:05:59.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:05:59.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2021-01-15T17:05:59.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:05:59.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:05:59.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:05:59.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:05:59.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2021-01-15T17:05:59.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2021-01-15T17:05:59.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:05:59.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:05:59.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:05:59.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:05:59.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:05:59.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:05:59.852 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:05:59.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:05:59.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:05:59.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:59.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:05:59.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:05:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:05:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:05:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:05:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:05:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:05:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:05:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:05:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:05:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:05:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2021-01-15T17:05:59.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:05:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=23ba462f-a1ac-4a4d-9376-8378d098e230 2021-01-15T17:05:59.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 23ba462f-a1ac-4a4d-9376-8378d098e230' 2021-01-15T17:05:59.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:05:59.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB3ywFgSpJ5MxAAbE4oqygvIR/oWJKrCDQhXA== 2021-01-15T17:05:59.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB3ywFgSpJ5MxAAbE4oqygvIR/oWJKrCDQhXA=="}' 2021-01-15T17:05:59.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 23ba462f-a1ac-4a4d-9376-8378d098e230 -i td/osd-backfill-space/5/new.json 2021-01-15T17:06:00.447 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:06:00.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2021-01-15T17:06:00.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB3ywFgSpJ5MxAAbE4oqygvIR/oWJKrCDQhXA== --osd-uuid 23ba462f-a1ac-4a4d-9376-8378d098e230 2021-01-15T17:06:00.486 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.486+0000 7effef521080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:00.487 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.488+0000 7effef521080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:00.489 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.489+0000 7effef521080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.490+0000 7effef521080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:06:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.490+0000 7effef521080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:06:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.490+0000 7effef521080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:06:00.594 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:00.594+0000 7effef521080 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2021-01-15T17:06:05.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2021-01-15T17:06:05.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:05.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:06:05.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:05.379 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:06:05.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:06:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:05.906 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:06:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:05.929+0000 7ff51512f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:05.931+0000 7ff51512f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:05.933 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:05.933+0000 7ff51512f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:06.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:06:06.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:06.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:06:06.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:06.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:06.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:06.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:06.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:06.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:06:06.325 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:06.452 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:06.451+0000 7ff51512f080 -1 Falling back to public interface 2021-01-15T17:06:06.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:07.345 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:07.344+0000 7ff51512f080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:06:07.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:07.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:07.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:07.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:07.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:06:07.724 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:08.200 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/272417,v1:127.0.0.1:6843/272417] [v2:127.0.0.1:6844/272417,v1:127.0.0.1:6845/272417] exists,up 23ba462f-a1ac-4a4d-9376-8378d098e230 2021-01-15T17:06:08.200 INFO:tasks.workunit.client.0.smithi145.stdout:add osd6 a3230948-c14f-4458-bc77-b469680cc44f 2021-01-15T17:06:08.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:08.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:08.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:08.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:08.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 6 2021-01-15T17:06:08.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:08.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:08.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=6 2021-01-15T17:06:08.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:08.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/6 2021-01-15T17:06:08.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:08.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:08.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:08.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:08.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/6' 2021-01-15T17:06:08.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/6/journal' 2021-01-15T17:06:08.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:08.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:08.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:08.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:08.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:08.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:08.206 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:08.206 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:08.206 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:08.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:08.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:08.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:08.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:08.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/6 2021-01-15T17:06:08.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:08.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a3230948-c14f-4458-bc77-b469680cc44f 2021-01-15T17:06:08.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd6 a3230948-c14f-4458-bc77-b469680cc44f' 2021-01-15T17:06:08.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:08.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCAywFgXO/tDBAAuzN+ZUqqYCI0okD8uAE7sw== 2021-01-15T17:06:08.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCAywFgXO/tDBAAuzN+ZUqqYCI0okD8uAE7sw=="}' 2021-01-15T17:06:08.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a3230948-c14f-4458-bc77-b469680cc44f -i td/osd-backfill-space/6/new.json 2021-01-15T17:06:08.765 INFO:tasks.workunit.client.0.smithi145.stdout:6 2021-01-15T17:06:08.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/6/new.json 2021-01-15T17:06:08.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 6 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/6 --osd-journal=td/osd-backfill-space/6/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCAywFgXO/tDBAAuzN+ZUqqYCI0okD8uAE7sw== --osd-uuid a3230948-c14f-4458-bc77-b469680cc44f 2021-01-15T17:06:08.804 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.804+0000 7f1dde76b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:08.820 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.820+0000 7f1dde76b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:08.821 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.821+0000 7f1dde76b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:08.822 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.822+0000 7f1dde76b080 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2021-01-15T17:06:08.822 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.822+0000 7f1dde76b080 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2021-01-15T17:06:08.822 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.822+0000 7f1dde76b080 -1 bluestore(td/osd-backfill-space/6/block) _read_bdev_label failed to open td/osd-backfill-space/6/block: (2) No such file or directory 2021-01-15T17:06:08.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:08.887+0000 7f1dde76b080 -1 bluestore(td/osd-backfill-space/6) _read_fsid unparsable uuid 2021-01-15T17:06:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/6/keyring 2021-01-15T17:06:13.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:13.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd6 key to auth repository 2021-01-15T17:06:13.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:13.611 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd6 key to auth repository 2021-01-15T17:06:14.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.6 2021-01-15T17:06:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 6 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/6 --osd-journal=td/osd-backfill-space/6/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:14.147 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.6 2021-01-15T17:06:14.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:14.172+0000 7f860e3b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:14.205+0000 7f860e3b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:14.207 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:14.207+0000 7f860e3b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:14.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 6 2021-01-15T17:06:14.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:14.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=6 2021-01-15T17:06:14.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:14.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2021-01-15T17:06:14.548 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:14.723 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:14.723+0000 7f860e3b8080 -1 Falling back to public interface 2021-01-15T17:06:14.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:15.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:15.599+0000 7f860e3b8080 -1 osd.6 0 log_to_monitors {default=true} 2021-01-15T17:06:15.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.6 up' 2021-01-15T17:06:15.975 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:16.394 INFO:tasks.workunit.client.0.smithi145.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/273681,v1:127.0.0.1:6851/273681] [v2:127.0.0.1:6852/273681,v1:127.0.0.1:6853/273681] exists,up a3230948-c14f-4458-bc77-b469680cc44f 2021-01-15T17:06:16.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 7 2021-01-15T17:06:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=7 2021-01-15T17:06:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/7 2021-01-15T17:06:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/7' 2021-01-15T17:06:16.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/7/journal' 2021-01-15T17:06:16.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:16.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:16.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:16.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:16.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:16.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:16.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:16.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:16.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:16.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:16.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:16.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:16.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:16.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:16.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:16.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:16.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/7 2021-01-15T17:06:16.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1eca892e-c8ff-4c89-b5a7-c958f446c647 2021-01-15T17:06:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd7 1eca892e-c8ff-4c89-b5a7-c958f446c647' 2021-01-15T17:06:16.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:16.404 INFO:tasks.workunit.client.0.smithi145.stdout:add osd7 1eca892e-c8ff-4c89-b5a7-c958f446c647 2021-01-15T17:06:16.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCIywFg7XyaGBAADPA0fsDHVRY5QnZvblnu5A== 2021-01-15T17:06:16.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCIywFg7XyaGBAADPA0fsDHVRY5QnZvblnu5A=="}' 2021-01-15T17:06:16.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1eca892e-c8ff-4c89-b5a7-c958f446c647 -i td/osd-backfill-space/7/new.json 2021-01-15T17:06:17.020 INFO:tasks.workunit.client.0.smithi145.stdout:7 2021-01-15T17:06:17.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/7/new.json 2021-01-15T17:06:17.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 7 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/7 --osd-journal=td/osd-backfill-space/7/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCIywFg7XyaGBAADPA0fsDHVRY5QnZvblnu5A== --osd-uuid 1eca892e-c8ff-4c89-b5a7-c958f446c647 2021-01-15T17:06:17.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.059+0000 7f876cb02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:17.060 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.060+0000 7f876cb02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.062+0000 7f876cb02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.063+0000 7f876cb02080 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2021-01-15T17:06:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.063+0000 7f876cb02080 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2021-01-15T17:06:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.063+0000 7f876cb02080 -1 bluestore(td/osd-backfill-space/7/block) _read_bdev_label failed to open td/osd-backfill-space/7/block: (2) No such file or directory 2021-01-15T17:06:17.118 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:17.118+0000 7f876cb02080 -1 bluestore(td/osd-backfill-space/7) _read_fsid unparsable uuid 2021-01-15T17:06:22.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/7/keyring 2021-01-15T17:06:22.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:22.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd7 key to auth repository 2021-01-15T17:06:22.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:22.389 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd7 key to auth repository 2021-01-15T17:06:22.920 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.7 2021-01-15T17:06:22.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.7 2021-01-15T17:06:22.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 7 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/7 --osd-journal=td/osd-backfill-space/7/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:22.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:22.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:22.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:22.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:22.945+0000 7f49a70ce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:22.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:22.947+0000 7f49a70ce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:22.948 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:22.948+0000 7f49a70ce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:23.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 7 2021-01-15T17:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=7 2021-01-15T17:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2021-01-15T17:06:23.322 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:23.468 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:23.467+0000 7f49a70ce080 -1 Falling back to public interface 2021-01-15T17:06:23.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:24.318 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:24.317+0000 7f49a70ce080 -1 osd.7 0 log_to_monitors {default=true} 2021-01-15T17:06:24.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:24.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.7 up' 2021-01-15T17:06:24.725 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:25.129 INFO:tasks.workunit.client.0.smithi145.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/274946,v1:127.0.0.1:6859/274946] [v2:127.0.0.1:6860/274946,v1:127.0.0.1:6861/274946] exists,up 1eca892e-c8ff-4c89-b5a7-c958f446c647 2021-01-15T17:06:25.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:25.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:25.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:25.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:25.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 8 2021-01-15T17:06:25.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:25.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:25.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=8 2021-01-15T17:06:25.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:25.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/8 2021-01-15T17:06:25.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:25.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:25.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:25.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:25.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/8' 2021-01-15T17:06:25.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/8/journal' 2021-01-15T17:06:25.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:25.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:25.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:25.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:25.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:25.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:25.134 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:25.134 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:25.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/8 2021-01-15T17:06:25.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:25.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1ec52340-5a53-494e-b26a-2c9ad7d5ab78 2021-01-15T17:06:25.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd8 1ec52340-5a53-494e-b26a-2c9ad7d5ab78' 2021-01-15T17:06:25.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:25.139 INFO:tasks.workunit.client.0.smithi145.stdout:add osd8 1ec52340-5a53-494e-b26a-2c9ad7d5ab78 2021-01-15T17:06:25.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCRywFgFHXDCBAAsJBBKNBDp9aioHXBrBo+TQ== 2021-01-15T17:06:25.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCRywFgFHXDCBAAsJBBKNBDp9aioHXBrBo+TQ=="}' 2021-01-15T17:06:25.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1ec52340-5a53-494e-b26a-2c9ad7d5ab78 -i td/osd-backfill-space/8/new.json 2021-01-15T17:06:25.956 INFO:tasks.workunit.client.0.smithi145.stdout:8 2021-01-15T17:06:25.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/8/new.json 2021-01-15T17:06:25.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 8 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/8 --osd-journal=td/osd-backfill-space/8/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCRywFgFHXDCBAAsJBBKNBDp9aioHXBrBo+TQ== --osd-uuid 1ec52340-5a53-494e-b26a-2c9ad7d5ab78 2021-01-15T17:06:25.995 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.995+0000 7f389e6b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:25.997 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.997+0000 7f389e6b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:25.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.998+0000 7f389e6b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:25.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.998+0000 7f389e6b5080 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2021-01-15T17:06:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.998+0000 7f389e6b5080 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2021-01-15T17:06:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:25.998+0000 7f389e6b5080 -1 bluestore(td/osd-backfill-space/8/block) _read_bdev_label failed to open td/osd-backfill-space/8/block: (2) No such file or directory 2021-01-15T17:06:26.122 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:26.121+0000 7f389e6b5080 -1 bluestore(td/osd-backfill-space/8) _read_fsid unparsable uuid 2021-01-15T17:06:31.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/8/keyring 2021-01-15T17:06:31.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:31.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd8 key to auth repository 2021-01-15T17:06:31.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:31.163 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd8 key to auth repository 2021-01-15T17:06:31.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.8 2021-01-15T17:06:31.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 8 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/8 --osd-journal=td/osd-backfill-space/8/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:31.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:31.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:31.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:31.695 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.8 2021-01-15T17:06:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:31.719+0000 7fafb74a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:31.736 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:31.736+0000 7fafb74a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:31.738 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:31.738+0000 7fafb74a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:32.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 8 2021-01-15T17:06:32.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:32.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=8 2021-01-15T17:06:32.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:32.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:32.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:32.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:32.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:32.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2021-01-15T17:06:32.146 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:32.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:32.255+0000 7fafb74a3080 -1 Falling back to public interface 2021-01-15T17:06:32.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:33.063+0000 7fafb74a3080 -1 osd.8 0 log_to_monitors {default=true} 2021-01-15T17:06:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.8 up' 2021-01-15T17:06:33.549 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:34.033 INFO:tasks.workunit.client.0.smithi145.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/276211,v1:127.0.0.1:6867/276211] [v2:127.0.0.1:6868/276211,v1:127.0.0.1:6869/276211] exists,up 1ec52340-5a53-494e-b26a-2c9ad7d5ab78 2021-01-15T17:06:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 9 2021-01-15T17:06:34.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:34.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:34.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=9 2021-01-15T17:06:34.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/9 2021-01-15T17:06:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/9' 2021-01-15T17:06:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/9/journal' 2021-01-15T17:06:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:34.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:34.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:34.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:34.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:34.038 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:34.038 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:34.039 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:34.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:34.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:34.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:34.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:34.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:34.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:34.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:34.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:34.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/9 2021-01-15T17:06:34.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:34.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f704ad84-a073-48fc-8ab4-1e2978c3966b 2021-01-15T17:06:34.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd9 f704ad84-a073-48fc-8ab4-1e2978c3966b' 2021-01-15T17:06:34.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:34.043 INFO:tasks.workunit.client.0.smithi145.stdout:add osd9 f704ad84-a073-48fc-8ab4-1e2978c3966b 2021-01-15T17:06:34.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCaywFgGeUNAxAAjD/sSOQ5V6fDf2d9rJRV2A== 2021-01-15T17:06:34.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCaywFgGeUNAxAAjD/sSOQ5V6fDf2d9rJRV2A=="}' 2021-01-15T17:06:34.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f704ad84-a073-48fc-8ab4-1e2978c3966b -i td/osd-backfill-space/9/new.json 2021-01-15T17:06:34.600 INFO:tasks.workunit.client.0.smithi145.stdout:9 2021-01-15T17:06:34.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/9/new.json 2021-01-15T17:06:34.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 9 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/9 --osd-journal=td/osd-backfill-space/9/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCaywFgGeUNAxAAjD/sSOQ5V6fDf2d9rJRV2A== --osd-uuid f704ad84-a073-48fc-8ab4-1e2978c3966b 2021-01-15T17:06:34.640 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.639+0000 7f8c9f611080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:34.641 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.641+0000 7f8c9f611080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:34.642 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.642+0000 7f8c9f611080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:34.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.643+0000 7f8c9f611080 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2021-01-15T17:06:34.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.643+0000 7f8c9f611080 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2021-01-15T17:06:34.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.643+0000 7f8c9f611080 -1 bluestore(td/osd-backfill-space/9/block) _read_bdev_label failed to open td/osd-backfill-space/9/block: (2) No such file or directory 2021-01-15T17:06:34.703 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:34.703+0000 7f8c9f611080 -1 bluestore(td/osd-backfill-space/9) _read_fsid unparsable uuid 2021-01-15T17:06:39.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/9/keyring 2021-01-15T17:06:39.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:39.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd9 key to auth repository 2021-01-15T17:06:39.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:39.975 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd9 key to auth repository 2021-01-15T17:06:40.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.9 2021-01-15T17:06:40.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 9 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/9 --osd-journal=td/osd-backfill-space/9/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:40.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:40.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:40.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:40.509 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.9 2021-01-15T17:06:40.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:40.533+0000 7f0d0d274080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:40.554 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:40.554+0000 7f0d0d274080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:40.554 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:40.554+0000 7f0d0d274080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:40.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 9 2021-01-15T17:06:40.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:40.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=9 2021-01-15T17:06:40.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:40.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2021-01-15T17:06:40.912 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:41.074 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:41.074+0000 7f0d0d274080 -1 Falling back to public interface 2021-01-15T17:06:41.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:41.922 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:41.922+0000 7f0d0d274080 -1 osd.9 0 log_to_monitors {default=true} 2021-01-15T17:06:42.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:42.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:42.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:42.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:42.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2021-01-15T17:06:42.333 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:42.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:43.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:43.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:43.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:06:43.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:43.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.9 up' 2021-01-15T17:06:43.746 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:06:44.206 INFO:tasks.workunit.client.0.smithi145.stdout:osd.9 up in weight 1 up_from 50 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/277483,v1:127.0.0.1:6875/277483] [v2:127.0.0.1:6876/277483,v1:127.0.0.1:6877/277483] exists,up f704ad84-a073-48fc-8ab4-1e2978c3966b 2021-01-15T17:06:44.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:44.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:44.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:44.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:44.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 10 2021-01-15T17:06:44.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:44.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=10 2021-01-15T17:06:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/10 2021-01-15T17:06:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:44.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/10' 2021-01-15T17:06:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/10/journal' 2021-01-15T17:06:44.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:44.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:44.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:44.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:44.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:44.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:44.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:44.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:44.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:44.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:44.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:44.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:44.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/10 2021-01-15T17:06:44.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:44.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ca594892-22b3-400a-ac25-2473281e1586 2021-01-15T17:06:44.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd10 ca594892-22b3-400a-ac25-2473281e1586' 2021-01-15T17:06:44.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:44.215 INFO:tasks.workunit.client.0.smithi145.stdout:add osd10 ca594892-22b3-400a-ac25-2473281e1586 2021-01-15T17:06:44.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCkywFg90JZDRAAlKqgodbuwhnDy5WcgmoedQ== 2021-01-15T17:06:44.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCkywFg90JZDRAAlKqgodbuwhnDy5WcgmoedQ=="}' 2021-01-15T17:06:44.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ca594892-22b3-400a-ac25-2473281e1586 -i td/osd-backfill-space/10/new.json 2021-01-15T17:06:44.715 INFO:tasks.workunit.client.0.smithi145.stdout:10 2021-01-15T17:06:44.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/10/new.json 2021-01-15T17:06:44.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 10 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/10 --osd-journal=td/osd-backfill-space/10/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCkywFg90JZDRAAlKqgodbuwhnDy5WcgmoedQ== --osd-uuid ca594892-22b3-400a-ac25-2473281e1586 2021-01-15T17:06:44.755 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.754+0000 7f32204fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:44.756 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.756+0000 7f32204fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:44.758 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.758+0000 7f32204fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.759+0000 7f32204fb080 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2021-01-15T17:06:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.759+0000 7f32204fb080 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2021-01-15T17:06:44.760 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.759+0000 7f32204fb080 -1 bluestore(td/osd-backfill-space/10/block) _read_bdev_label failed to open td/osd-backfill-space/10/block: (2) No such file or directory 2021-01-15T17:06:44.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:44.930+0000 7f32204fb080 -1 bluestore(td/osd-backfill-space/10) _read_fsid unparsable uuid 2021-01-15T17:06:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/10/keyring 2021-01-15T17:06:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:06:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd10 key to auth repository 2021-01-15T17:06:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:06:50.229 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd10 key to auth repository 2021-01-15T17:06:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.10 2021-01-15T17:06:50.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 10 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/10 --osd-journal=td/osd-backfill-space/10/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:50.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:06:50.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:06:50.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:06:50.765 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.10 2021-01-15T17:06:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:50.789+0000 7fcb05a5e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:50.827 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:50.827+0000 7fcb05a5e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:50.827 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:50.828+0000 7fcb05a5e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 10 2021-01-15T17:06:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:06:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=10 2021-01-15T17:06:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:06:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:06:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:06:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:51.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2021-01-15T17:06:51.165 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:06:51.343 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:51.343+0000 7fcb05a5e080 -1 Falling back to public interface 2021-01-15T17:06:51.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:52.213 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:52.213+0000 7fcb05a5e080 -1 osd.10 0 log_to_monitors {default=true} 2021-01-15T17:06:52.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:52.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:52.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:06:52.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:52.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2021-01-15T17:06:52.571 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:06:52.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:06:53.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:06:53.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:06:53.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:06:53.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:06:53.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.10 up' 2021-01-15T17:06:53.988 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:06:54.482 INFO:tasks.workunit.client.0.smithi145.stdout:osd.10 up in weight 1 up_from 55 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6882/278778,v1:127.0.0.1:6883/278778] [v2:127.0.0.1:6884/278778,v1:127.0.0.1:6885/278778] exists,up ca594892-22b3-400a-ac25-2473281e1586 2021-01-15T17:06:54.482 INFO:tasks.workunit.client.0.smithi145.stdout:add osd11 d48df733-97fa-4bda-b817-68b7acc38208 2021-01-15T17:06:54.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:06:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:06:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:06:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:06:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 11 2021-01-15T17:06:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:06:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:06:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=11 2021-01-15T17:06:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:06:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/11 2021-01-15T17:06:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:06:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:06:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:06:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:06:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/11' 2021-01-15T17:06:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/11/journal' 2021-01-15T17:06:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:06:54.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:06:54.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:06:54.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:06:54.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:06:54.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:06:54.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:06:54.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:06:54.488 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:06:54.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:54.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:06:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:06:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:06:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:06:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:06:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:06:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:06:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:06:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:06:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:06:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:06:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/11 2021-01-15T17:06:54.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:06:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d48df733-97fa-4bda-b817-68b7acc38208 2021-01-15T17:06:54.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd11 d48df733-97fa-4bda-b817-68b7acc38208' 2021-01-15T17:06:54.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:06:54.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCuywFgTem8HRAAPGoLEBSwdibPkgu/d0a+Gg== 2021-01-15T17:06:54.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCuywFgTem8HRAAPGoLEBSwdibPkgu/d0a+Gg=="}' 2021-01-15T17:06:54.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d48df733-97fa-4bda-b817-68b7acc38208 -i td/osd-backfill-space/11/new.json 2021-01-15T17:06:55.005 INFO:tasks.workunit.client.0.smithi145.stdout:11 2021-01-15T17:06:55.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/11/new.json 2021-01-15T17:06:55.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 11 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/11 --osd-journal=td/osd-backfill-space/11/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCuywFgTem8HRAAPGoLEBSwdibPkgu/d0a+Gg== --osd-uuid d48df733-97fa-4bda-b817-68b7acc38208 2021-01-15T17:06:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.043+0000 7f38e82ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:55.046 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.045+0000 7f38e82ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:55.048 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.047+0000 7f38e82ac080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:06:55.049 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.048+0000 7f38e82ac080 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2021-01-15T17:06:55.049 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.048+0000 7f38e82ac080 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2021-01-15T17:06:55.049 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.048+0000 7f38e82ac080 -1 bluestore(td/osd-backfill-space/11/block) _read_bdev_label failed to open td/osd-backfill-space/11/block: (2) No such file or directory 2021-01-15T17:06:55.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:06:55.152+0000 7f38e82ac080 -1 bluestore(td/osd-backfill-space/11) _read_fsid unparsable uuid 2021-01-15T17:07:00.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/11/keyring 2021-01-15T17:07:00.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd11 key to auth repository 2021-01-15T17:07:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/11/keyring auth add osd.11 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:00.142 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd11 key to auth repository 2021-01-15T17:07:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.11 2021-01-15T17:07:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 11 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/11 --osd-journal=td/osd-backfill-space/11/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:00.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:00.672 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.11 2021-01-15T17:07:00.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:00.694+0000 7efd1943f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:00.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:00.720+0000 7efd1943f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:00.721+0000 7efd1943f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:01.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 11 2021-01-15T17:07:01.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=11 2021-01-15T17:07:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2021-01-15T17:07:01.071 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:01.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:01.241+0000 7efd1943f080 -1 Falling back to public interface 2021-01-15T17:07:01.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:02.057 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:02.057+0000 7efd1943f080 -1 osd.11 0 log_to_monitors {default=true} 2021-01-15T17:07:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:02.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:02.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:02.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:02.472 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:02.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2021-01-15T17:07:02.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:03.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:07:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.11 up' 2021-01-15T17:07:03.888 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:07:04.325 INFO:tasks.workunit.client.0.smithi145.stdout:osd.11 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6890/280068,v1:127.0.0.1:6891/280068] [v2:127.0.0.1:6892/280068,v1:127.0.0.1:6893/280068] exists,up d48df733-97fa-4bda-b817-68b7acc38208 2021-01-15T17:07:04.325 INFO:tasks.workunit.client.0.smithi145.stdout:add osd12 fa3bb8ec-75ea-45c7-8b90-422106c5c29b 2021-01-15T17:07:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:04.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 12 2021-01-15T17:07:04.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:04.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:04.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=12 2021-01-15T17:07:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/12 2021-01-15T17:07:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:04.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:04.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/12' 2021-01-15T17:07:04.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/12/journal' 2021-01-15T17:07:04.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:04.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:04.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:04.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:04.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:04.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:04.330 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:04.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:04.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:04.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:04.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:04.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:04.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:04.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:04.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:04.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:04.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:04.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:04.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:04.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:04.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:04.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/12 2021-01-15T17:07:04.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:04.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fa3bb8ec-75ea-45c7-8b90-422106c5c29b 2021-01-15T17:07:04.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd12 fa3bb8ec-75ea-45c7-8b90-422106c5c29b' 2021-01-15T17:07:04.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:04.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC4ywFg9ClQFBAAzcmP7zNkEeIxyqZ2USB7cg== 2021-01-15T17:07:04.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC4ywFg9ClQFBAAzcmP7zNkEeIxyqZ2USB7cg=="}' 2021-01-15T17:07:04.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fa3bb8ec-75ea-45c7-8b90-422106c5c29b -i td/osd-backfill-space/12/new.json 2021-01-15T17:07:04.902 INFO:tasks.workunit.client.0.smithi145.stdout:12 2021-01-15T17:07:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/12/new.json 2021-01-15T17:07:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 12 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/12 --osd-journal=td/osd-backfill-space/12/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC4ywFg9ClQFBAAzcmP7zNkEeIxyqZ2USB7cg== --osd-uuid fa3bb8ec-75ea-45c7-8b90-422106c5c29b 2021-01-15T17:07:04.941 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.941+0000 7fe85e3cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:04.943 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.943+0000 7fe85e3cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:04.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.944+0000 7fe85e3cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:04.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.945+0000 7fe85e3cd080 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2021-01-15T17:07:04.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.945+0000 7fe85e3cd080 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2021-01-15T17:07:04.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.945+0000 7fe85e3cd080 -1 bluestore(td/osd-backfill-space/12/block) _read_bdev_label failed to open td/osd-backfill-space/12/block: (2) No such file or directory 2021-01-15T17:07:04.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:04.982+0000 7fe85e3cd080 -1 bluestore(td/osd-backfill-space/12) _read_fsid unparsable uuid 2021-01-15T17:07:10.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/12/keyring 2021-01-15T17:07:10.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:10.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd12 key to auth repository 2021-01-15T17:07:10.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/12/keyring auth add osd.12 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:10.305 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd12 key to auth repository 2021-01-15T17:07:10.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.12 2021-01-15T17:07:10.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 12 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/12 --osd-journal=td/osd-backfill-space/12/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:10.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:10.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:10.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:10.836 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.12 2021-01-15T17:07:10.860 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:10.860+0000 7f2016d1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:10.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:10.875+0000 7f2016d1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:10.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:10.876+0000 7f2016d1f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:11.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 12 2021-01-15T17:07:11.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:11.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=12 2021-01-15T17:07:11.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:11.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:11.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:11.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:11.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:11.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.12 up' 2021-01-15T17:07:11.240 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:11.392+0000 7f2016d1f080 -1 Falling back to public interface 2021-01-15T17:07:11.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:12.254+0000 7f2016d1f080 -1 osd.12 0 log_to_monitors {default=true} 2021-01-15T17:07:12.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:12.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:12.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:12.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:12.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.12 up' 2021-01-15T17:07:12.643 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:13.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:14.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:14.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:14.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:07:14.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:14.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.12 up' 2021-01-15T17:07:14.082 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:07:14.514 INFO:tasks.workunit.client.0.smithi145.stdout:osd.12 up in weight 1 up_from 65 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6898/281359,v1:127.0.0.1:6899/281359] [v2:127.0.0.1:6900/281359,v1:127.0.0.1:6901/281359] exists,up fa3bb8ec-75ea-45c7-8b90-422106c5c29b 2021-01-15T17:07:14.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:14.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 13 2021-01-15T17:07:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=13 2021-01-15T17:07:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/13 2021-01-15T17:07:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:14.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:14.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:14.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/13' 2021-01-15T17:07:14.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/13/journal' 2021-01-15T17:07:14.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:14.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:14.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:14.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:14.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:14.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:14.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:14.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:14.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:14.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:14.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:14.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:14.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:14.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:14.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:14.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:14.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:14.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:14.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/13 2021-01-15T17:07:14.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:14.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9d21c225-aeb1-486d-87bf-2ddecfd1ec34 2021-01-15T17:07:14.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd13 9d21c225-aeb1-486d-87bf-2ddecfd1ec34' 2021-01-15T17:07:14.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:14.524 INFO:tasks.workunit.client.0.smithi145.stdout:add osd13 9d21c225-aeb1-486d-87bf-2ddecfd1ec34 2021-01-15T17:07:14.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDCywFg6NW1HxAAOd3JT4jgwJKnsBtTNydn9g== 2021-01-15T17:07:14.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDCywFg6NW1HxAAOd3JT4jgwJKnsBtTNydn9g=="}' 2021-01-15T17:07:14.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9d21c225-aeb1-486d-87bf-2ddecfd1ec34 -i td/osd-backfill-space/13/new.json 2021-01-15T17:07:15.189 INFO:tasks.workunit.client.0.smithi145.stdout:13 2021-01-15T17:07:15.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/13/new.json 2021-01-15T17:07:15.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 13 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/13 --osd-journal=td/osd-backfill-space/13/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDCywFg6NW1HxAAOd3JT4jgwJKnsBtTNydn9g== --osd-uuid 9d21c225-aeb1-486d-87bf-2ddecfd1ec34 2021-01-15T17:07:15.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.229+0000 7f122ae2c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:15.231 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.231+0000 7f122ae2c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:15.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.232+0000 7f122ae2c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:15.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.232+0000 7f122ae2c080 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2021-01-15T17:07:15.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.232+0000 7f122ae2c080 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2021-01-15T17:07:15.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.232+0000 7f122ae2c080 -1 bluestore(td/osd-backfill-space/13/block) _read_bdev_label failed to open td/osd-backfill-space/13/block: (2) No such file or directory 2021-01-15T17:07:15.363 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:15.363+0000 7f122ae2c080 -1 bluestore(td/osd-backfill-space/13) _read_fsid unparsable uuid 2021-01-15T17:07:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/13/keyring 2021-01-15T17:07:20.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:20.422 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd13 key to auth repository 2021-01-15T17:07:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd13 key to auth repository 2021-01-15T17:07:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/13/keyring auth add osd.13 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:20.951 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.13 2021-01-15T17:07:20.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.13 2021-01-15T17:07:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 13 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/13 --osd-journal=td/osd-backfill-space/13/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:20.976 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:20.976+0000 7f52a3ead080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:21.004 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:21.003+0000 7f52a3ead080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:21.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:21.005+0000 7f52a3ead080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:21.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 13 2021-01-15T17:07:21.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:21.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=13 2021-01-15T17:07:21.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.13 up' 2021-01-15T17:07:21.352 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:21.520 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:21.520+0000 7f52a3ead080 -1 Falling back to public interface 2021-01-15T17:07:21.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:22.374 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:22.374+0000 7f52a3ead080 -1 osd.13 0 log_to_monitors {default=true} 2021-01-15T17:07:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:22.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:22.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:22.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:22.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.13 up' 2021-01-15T17:07:22.752 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:23.590 INFO:tasks.workunit.client.0.smithi145.stdout:osd.13 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6906/282649,v1:127.0.0.1:6907/282649] [v2:127.0.0.1:6908/282649,v1:127.0.0.1:6909/282649] exists,up 9d21c225-aeb1-486d-87bf-2ddecfd1ec34 2021-01-15T17:07:23.591 INFO:tasks.workunit.client.0.smithi145.stdout:add osd14 6c79a8db-bb68-4dfd-bd9a-3e124a608628 2021-01-15T17:07:23.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:23.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:23.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:23.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 14 2021-01-15T17:07:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=14 2021-01-15T17:07:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/14 2021-01-15T17:07:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/14' 2021-01-15T17:07:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/14/journal' 2021-01-15T17:07:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:23.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:23.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:23.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:23.599 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:23.599 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:23.599 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:23.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:23.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:23.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:23.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:23.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:23.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:23.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:23.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:23.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:23.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:23.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:23.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/14 2021-01-15T17:07:23.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6c79a8db-bb68-4dfd-bd9a-3e124a608628 2021-01-15T17:07:23.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd14 6c79a8db-bb68-4dfd-bd9a-3e124a608628' 2021-01-15T17:07:23.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDLywFg2XIaJBAAUBL3sZ8GmmvAVLJfytCz+g== 2021-01-15T17:07:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDLywFg2XIaJBAAUBL3sZ8GmmvAVLJfytCz+g=="}' 2021-01-15T17:07:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6c79a8db-bb68-4dfd-bd9a-3e124a608628 -i td/osd-backfill-space/14/new.json 2021-01-15T17:07:24.208 INFO:tasks.workunit.client.0.smithi145.stdout:14 2021-01-15T17:07:24.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/14/new.json 2021-01-15T17:07:24.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 14 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/14 --osd-journal=td/osd-backfill-space/14/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDLywFg2XIaJBAAUBL3sZ8GmmvAVLJfytCz+g== --osd-uuid 6c79a8db-bb68-4dfd-bd9a-3e124a608628 2021-01-15T17:07:24.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.247+0000 7f826a9e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:24.328 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.328+0000 7f826a9e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:24.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.330+0000 7f826a9e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:24.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.331+0000 7f826a9e8080 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2021-01-15T17:07:24.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.331+0000 7f826a9e8080 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2021-01-15T17:07:24.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.331+0000 7f826a9e8080 -1 bluestore(td/osd-backfill-space/14/block) _read_bdev_label failed to open td/osd-backfill-space/14/block: (2) No such file or directory 2021-01-15T17:07:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:24.420+0000 7f826a9e8080 -1 bluestore(td/osd-backfill-space/14) _read_fsid unparsable uuid 2021-01-15T17:07:29.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/14/keyring 2021-01-15T17:07:29.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:29.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd14 key to auth repository 2021-01-15T17:07:29.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/14/keyring auth add osd.14 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:29.699 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd14 key to auth repository 2021-01-15T17:07:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.14 2021-01-15T17:07:30.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 14 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/14 --osd-journal=td/osd-backfill-space/14/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:30.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:30.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:30.234 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.14 2021-01-15T17:07:30.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:30.258+0000 7f8e03333080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:30.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:30.260+0000 7f8e03333080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:30.262+0000 7f8e03333080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:30.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 14 2021-01-15T17:07:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=14 2021-01-15T17:07:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2021-01-15T17:07:30.636 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:30.778+0000 7f8e03333080 -1 Falling back to public interface 2021-01-15T17:07:31.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:31.634 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:31.634+0000 7f8e03333080 -1 osd.14 0 log_to_monitors {default=true} 2021-01-15T17:07:32.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:32.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:32.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:32.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:32.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2021-01-15T17:07:32.053 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:32.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:07:33.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:33.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.14 up' 2021-01-15T17:07:33.497 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:07:33.953 INFO:tasks.workunit.client.0.smithi145.stdout:osd.14 up in weight 1 up_from 75 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6914/283917,v1:127.0.0.1:6915/283917] [v2:127.0.0.1:6916/283917,v1:127.0.0.1:6917/283917] exists,up 6c79a8db-bb68-4dfd-bd9a-3e124a608628 2021-01-15T17:07:33.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:33.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:33.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:33.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:33.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 15 2021-01-15T17:07:33.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:33.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:33.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=15 2021-01-15T17:07:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/15 2021-01-15T17:07:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/15' 2021-01-15T17:07:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/15/journal' 2021-01-15T17:07:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:33.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:33.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:33.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:33.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:33.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:33.958 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:33.959 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:33.959 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:33.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:33.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:33.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/15 2021-01-15T17:07:33.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:33.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=54ff6446-fd54-41e9-ae73-2c5d71495712 2021-01-15T17:07:33.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd15 54ff6446-fd54-41e9-ae73-2c5d71495712' 2021-01-15T17:07:33.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:33.963 INFO:tasks.workunit.client.0.smithi145.stdout:add osd15 54ff6446-fd54-41e9-ae73-2c5d71495712 2021-01-15T17:07:33.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDVywFgrDzoORAArF6rOZ7gdnB/PA4ulKiSjA== 2021-01-15T17:07:33.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDVywFgrDzoORAArF6rOZ7gdnB/PA4ulKiSjA=="}' 2021-01-15T17:07:33.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 54ff6446-fd54-41e9-ae73-2c5d71495712 -i td/osd-backfill-space/15/new.json 2021-01-15T17:07:34.498 INFO:tasks.workunit.client.0.smithi145.stdout:15 2021-01-15T17:07:34.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/15/new.json 2021-01-15T17:07:34.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 15 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/15 --osd-journal=td/osd-backfill-space/15/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDVywFgrDzoORAArF6rOZ7gdnB/PA4ulKiSjA== --osd-uuid 54ff6446-fd54-41e9-ae73-2c5d71495712 2021-01-15T17:07:34.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.537+0000 7f9d76763080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:34.539 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.539+0000 7f9d76763080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:34.541 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.541+0000 7f9d76763080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:34.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.542+0000 7f9d76763080 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2021-01-15T17:07:34.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.542+0000 7f9d76763080 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2021-01-15T17:07:34.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.542+0000 7f9d76763080 -1 bluestore(td/osd-backfill-space/15/block) _read_bdev_label failed to open td/osd-backfill-space/15/block: (2) No such file or directory 2021-01-15T17:07:34.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:34.637+0000 7f9d76763080 -1 bluestore(td/osd-backfill-space/15) _read_fsid unparsable uuid 2021-01-15T17:07:39.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/15/keyring 2021-01-15T17:07:39.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:39.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd15 key to auth repository 2021-01-15T17:07:39.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/15/keyring auth add osd.15 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:39.660 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd15 key to auth repository 2021-01-15T17:07:40.190 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.15 2021-01-15T17:07:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.15 2021-01-15T17:07:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 15 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/15 --osd-journal=td/osd-backfill-space/15/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:40.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:40.216+0000 7eff17ebc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:40.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:40.230+0000 7eff17ebc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:40.231 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:40.231+0000 7eff17ebc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:40.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 15 2021-01-15T17:07:40.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:40.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=15 2021-01-15T17:07:40.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:40.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:40.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:40.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:40.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:40.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2021-01-15T17:07:40.597 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:40.748 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:40.748+0000 7eff17ebc080 -1 Falling back to public interface 2021-01-15T17:07:40.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:41.566 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:41.566+0000 7eff17ebc080 -1 osd.15 0 log_to_monitors {default=true} 2021-01-15T17:07:41.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:41.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:41.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:41.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:41.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2021-01-15T17:07:41.995 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:42.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:43.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:43.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:43.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:07:43.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:43.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.15 up' 2021-01-15T17:07:43.405 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:07:43.826 INFO:tasks.workunit.client.0.smithi145.stdout:osd.15 up in weight 1 up_from 80 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6922/285215,v1:127.0.0.1:6923/285215] [v2:127.0.0.1:6924/285215,v1:127.0.0.1:6925/285215] exists,up 54ff6446-fd54-41e9-ae73-2c5d71495712 2021-01-15T17:07:43.827 INFO:tasks.workunit.client.0.smithi145.stdout:add osd16 b5ca02d3-aa92-4ca9-a11b-7fb69e95729d 2021-01-15T17:07:43.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:43.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:43.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:43.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:43.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 16 2021-01-15T17:07:43.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:43.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:43.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=16 2021-01-15T17:07:43.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:43.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/16 2021-01-15T17:07:43.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:43.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:43.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:43.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:43.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/16' 2021-01-15T17:07:43.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/16/journal' 2021-01-15T17:07:43.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:43.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:43.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:43.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:43.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:43.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:43.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:43.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:43.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:43.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:43.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:43.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:43.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/16 2021-01-15T17:07:43.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:43.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b5ca02d3-aa92-4ca9-a11b-7fb69e95729d 2021-01-15T17:07:43.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd16 b5ca02d3-aa92-4ca9-a11b-7fb69e95729d' 2021-01-15T17:07:43.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:43.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDfywFg7w8HMhAAyA3l3INUSm6fR0tJf/4ZEw== 2021-01-15T17:07:43.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDfywFg7w8HMhAAyA3l3INUSm6fR0tJf/4ZEw=="}' 2021-01-15T17:07:43.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b5ca02d3-aa92-4ca9-a11b-7fb69e95729d -i td/osd-backfill-space/16/new.json 2021-01-15T17:07:44.455 INFO:tasks.workunit.client.0.smithi145.stdout:16 2021-01-15T17:07:44.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/16/new.json 2021-01-15T17:07:44.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 16 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/16 --osd-journal=td/osd-backfill-space/16/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDfywFg7w8HMhAAyA3l3INUSm6fR0tJf/4ZEw== --osd-uuid b5ca02d3-aa92-4ca9-a11b-7fb69e95729d 2021-01-15T17:07:44.495 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.495+0000 7f8186d07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:44.496 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.496+0000 7f8186d07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:44.497 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.498+0000 7f8186d07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:44.498 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.498+0000 7f8186d07080 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2021-01-15T17:07:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.498+0000 7f8186d07080 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2021-01-15T17:07:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.498+0000 7f8186d07080 -1 bluestore(td/osd-backfill-space/16/block) _read_bdev_label failed to open td/osd-backfill-space/16/block: (2) No such file or directory 2021-01-15T17:07:44.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:44.535+0000 7f8186d07080 -1 bluestore(td/osd-backfill-space/16) _read_fsid unparsable uuid 2021-01-15T17:07:50.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/16/keyring 2021-01-15T17:07:50.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:07:50.736 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd16 key to auth repository 2021-01-15T17:07:50.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd16 key to auth repository 2021-01-15T17:07:50.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/16/keyring auth add osd.16 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:07:51.270 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.16 2021-01-15T17:07:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.16 2021-01-15T17:07:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 16 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/16 --osd-journal=td/osd-backfill-space/16/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:07:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:07:51.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:07:51.296 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:51.296+0000 7fcb89088080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:51.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:51.319+0000 7fcb89088080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:51.321 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:51.321+0000 7fcb89088080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:51.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 16 2021-01-15T17:07:51.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:07:51.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=16 2021-01-15T17:07:51.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:07:51.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:07:51.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:51.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:07:51.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:51.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.16 up' 2021-01-15T17:07:51.676 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:07:51.836 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:51.835+0000 7fcb89088080 -1 Falling back to public interface 2021-01-15T17:07:52.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:07:52.632 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:52.632+0000 7fcb89088080 -1 osd.16 0 log_to_monitors {default=true} 2021-01-15T17:07:53.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:07:53.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:07:53.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:07:53.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:07:53.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.16 up' 2021-01-15T17:07:53.080 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:07:53.600 INFO:tasks.workunit.client.0.smithi145.stdout:osd.16 up in weight 1 up_from 85 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6930/286505,v1:127.0.0.1:6931/286505] [v2:127.0.0.1:6932/286505,v1:127.0.0.1:6933/286505] exists,up b5ca02d3-aa92-4ca9-a11b-7fb69e95729d 2021-01-15T17:07:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:07:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:07:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:07:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:07:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 17 2021-01-15T17:07:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:07:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:07:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=17 2021-01-15T17:07:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:07:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/17 2021-01-15T17:07:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:07:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:07:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:07:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:07:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/17' 2021-01-15T17:07:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/17/journal' 2021-01-15T17:07:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:07:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:07:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:07:53.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:07:53.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:07:53.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:07:53.606 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:07:53.606 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:07:53.606 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:07:53.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:53.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:07:53.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:07:53.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:07:53.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:07:53.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:07:53.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:07:53.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:07:53.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:07:53.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:07:53.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:07:53.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:07:53.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/17 2021-01-15T17:07:53.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:07:53.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=eadfc336-88c2-48fe-9c23-c6d28123c634 2021-01-15T17:07:53.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd17 eadfc336-88c2-48fe-9c23-c6d28123c634' 2021-01-15T17:07:53.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:07:53.610 INFO:tasks.workunit.client.0.smithi145.stdout:add osd17 eadfc336-88c2-48fe-9c23-c6d28123c634 2021-01-15T17:07:53.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDpywFg4MrbJBAAnGG9U4l6PHE9AgdknteMjQ== 2021-01-15T17:07:53.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDpywFg4MrbJBAAnGG9U4l6PHE9AgdknteMjQ=="}' 2021-01-15T17:07:53.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new eadfc336-88c2-48fe-9c23-c6d28123c634 -i td/osd-backfill-space/17/new.json 2021-01-15T17:07:54.149 INFO:tasks.workunit.client.0.smithi145.stdout:17 2021-01-15T17:07:54.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/17/new.json 2021-01-15T17:07:54.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 17 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/17 --osd-journal=td/osd-backfill-space/17/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpywFg4MrbJBAAnGG9U4l6PHE9AgdknteMjQ== --osd-uuid eadfc336-88c2-48fe-9c23-c6d28123c634 2021-01-15T17:07:54.188 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.188+0000 7f72efa68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:54.190 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.190+0000 7f72efa68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:54.192 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.192+0000 7f72efa68080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:07:54.193 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.192+0000 7f72efa68080 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2021-01-15T17:07:54.193 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.192+0000 7f72efa68080 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2021-01-15T17:07:54.193 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.192+0000 7f72efa68080 -1 bluestore(td/osd-backfill-space/17/block) _read_bdev_label failed to open td/osd-backfill-space/17/block: (2) No such file or directory 2021-01-15T17:07:54.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:07:54.526+0000 7f72efa68080 -1 bluestore(td/osd-backfill-space/17) _read_fsid unparsable uuid 2021-01-15T17:08:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/17/keyring 2021-01-15T17:08:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:08:00.035 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd17 key to auth repository 2021-01-15T17:08:00.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd17 key to auth repository 2021-01-15T17:08:00.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/17/keyring auth add osd.17 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:08:00.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.17 2021-01-15T17:08:00.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:08:00.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 17 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/17 --osd-journal=td/osd-backfill-space/17/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:08:00.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:08:00.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:08:00.574 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.17 2021-01-15T17:08:00.598 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:00.598+0000 7fdbfb608080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:00.618 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:00.618+0000 7fdbfb608080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:00.620 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:00.620+0000 7fdbfb608080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:00.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 17 2021-01-15T17:08:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:08:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=17 2021-01-15T17:08:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:08:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:08:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:08:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2021-01-15T17:08:00.977 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:08:01.136 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:01.135+0000 7fdbfb608080 -1 Falling back to public interface 2021-01-15T17:08:01.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:02.023 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:02.023+0000 7fdbfb608080 -1 osd.17 0 log_to_monitors {default=true} 2021-01-15T17:08:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:02.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:02.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:08:02.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:02.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2021-01-15T17:08:02.389 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:08:02.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:03.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:03.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:03.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:08:03.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:03.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.17 up' 2021-01-15T17:08:03.808 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:08:04.276 INFO:tasks.workunit.client.0.smithi145.stdout:osd.17 up in weight 1 up_from 90 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6938/287769,v1:127.0.0.1:6939/287769] [v2:127.0.0.1:6940/287769,v1:127.0.0.1:6941/287769] exists,up eadfc336-88c2-48fe-9c23-c6d28123c634 2021-01-15T17:08:04.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:08:04.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:08:04.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:08:04.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:08:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 18 2021-01-15T17:08:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:08:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:08:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=18 2021-01-15T17:08:04.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:08:04.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/18 2021-01-15T17:08:04.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:08:04.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:08:04.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:08:04.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:08:04.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/18' 2021-01-15T17:08:04.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/18/journal' 2021-01-15T17:08:04.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:08:04.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:08:04.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:08:04.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:08:04.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:08:04.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:08:04.281 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:08:04.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:08:04.282 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:08:04.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:08:04.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:08:04.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:08:04.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:08:04.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:08:04.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:08:04.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:08:04.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:08:04.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:08:04.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:08:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:08:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:08:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/18 2021-01-15T17:08:04.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:08:04.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=619547e4-41be-41d2-8c95-8d38c18e93c5 2021-01-15T17:08:04.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd18 619547e4-41be-41d2-8c95-8d38c18e93c5' 2021-01-15T17:08:04.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:08:04.287 INFO:tasks.workunit.client.0.smithi145.stdout:add osd18 619547e4-41be-41d2-8c95-8d38c18e93c5 2021-01-15T17:08:04.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD0ywFg9PKNERAAhiUAwiWWXOcmX6iV0deenQ== 2021-01-15T17:08:04.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD0ywFg9PKNERAAhiUAwiWWXOcmX6iV0deenQ=="}' 2021-01-15T17:08:04.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 619547e4-41be-41d2-8c95-8d38c18e93c5 -i td/osd-backfill-space/18/new.json 2021-01-15T17:08:04.794 INFO:tasks.workunit.client.0.smithi145.stdout:18 2021-01-15T17:08:04.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/18/new.json 2021-01-15T17:08:04.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 18 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/18 --osd-journal=td/osd-backfill-space/18/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD0ywFg9PKNERAAhiUAwiWWXOcmX6iV0deenQ== --osd-uuid 619547e4-41be-41d2-8c95-8d38c18e93c5 2021-01-15T17:08:04.833 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.833+0000 7f7004867080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:04.835 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.835+0000 7f7004867080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:04.836 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.836+0000 7f7004867080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.837+0000 7f7004867080 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2021-01-15T17:08:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.837+0000 7f7004867080 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2021-01-15T17:08:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:04.837+0000 7f7004867080 -1 bluestore(td/osd-backfill-space/18/block) _read_bdev_label failed to open td/osd-backfill-space/18/block: (2) No such file or directory 2021-01-15T17:08:05.071 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:05.071+0000 7f7004867080 -1 bluestore(td/osd-backfill-space/18) _read_fsid unparsable uuid 2021-01-15T17:08:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/18/keyring 2021-01-15T17:08:10.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:08:10.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd18 key to auth repository 2021-01-15T17:08:10.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/18/keyring auth add osd.18 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:08:10.622 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd18 key to auth repository 2021-01-15T17:08:11.161 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.18 2021-01-15T17:08:11.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.18 2021-01-15T17:08:11.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 18 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/18 --osd-journal=td/osd-backfill-space/18/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:08:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:08:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:08:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:08:11.187 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:11.187+0000 7f7968c94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:11.206+0000 7f7968c94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:11.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:11.208+0000 7f7968c94080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:11.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 18 2021-01-15T17:08:11.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:08:11.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=18 2021-01-15T17:08:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:08:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:08:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:11.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:08:11.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:11.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2021-01-15T17:08:11.567 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:08:11.724 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:11.723+0000 7f7968c94080 -1 Falling back to public interface 2021-01-15T17:08:11.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:12.591 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:12.591+0000 7f7968c94080 -1 osd.18 0 log_to_monitors {default=true} 2021-01-15T17:08:12.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:12.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:12.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:08:12.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:12.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2021-01-15T17:08:12.996 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:08:13.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:14.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:08:14.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:14.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.18 up' 2021-01-15T17:08:14.407 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:08:14.813 INFO:tasks.workunit.client.0.smithi145.stdout:osd.18 up in weight 1 up_from 95 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6946/289059,v1:127.0.0.1:6947/289059] [v2:127.0.0.1:6948/289059,v1:127.0.0.1:6949/289059] exists,up 619547e4-41be-41d2-8c95-8d38c18e93c5 2021-01-15T17:08:14.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:08:14.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:08:14.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:08:14.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:199: TEST_backfill_test_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:08:14.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:201: TEST_backfill_test_multi: run_osd td/osd-backfill-space 19 2021-01-15T17:08:14.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:08:14.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:08:14.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=19 2021-01-15T17:08:14.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:08:14.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/19 2021-01-15T17:08:14.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:08:14.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:08:14.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:08:14.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:08:14.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/19' 2021-01-15T17:08:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/19/journal' 2021-01-15T17:08:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:08:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:08:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:08:14.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:08:14.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:08:14.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:08:14.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:08:14.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:08:14.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:08:14.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:08:14.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:08:14.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:08:14.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:08:14.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:08:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:08:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:08:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:08:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:08:14.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:08:14.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:08:14.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:08:14.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/19 2021-01-15T17:08:14.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:08:14.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e4a9dbec-190d-42cb-af47-8ade9543e6cf 2021-01-15T17:08:14.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd19 e4a9dbec-190d-42cb-af47-8ade9543e6cf' 2021-01-15T17:08:14.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:08:14.823 INFO:tasks.workunit.client.0.smithi145.stdout:add osd19 e4a9dbec-190d-42cb-af47-8ade9543e6cf 2021-01-15T17:08:14.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD+ywFgeRKOMRAAVSjO8++hxNRG4nB5ErT/jg== 2021-01-15T17:08:14.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD+ywFgeRKOMRAAVSjO8++hxNRG4nB5ErT/jg=="}' 2021-01-15T17:08:14.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e4a9dbec-190d-42cb-af47-8ade9543e6cf -i td/osd-backfill-space/19/new.json 2021-01-15T17:08:15.462 INFO:tasks.workunit.client.0.smithi145.stdout:19 2021-01-15T17:08:15.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/19/new.json 2021-01-15T17:08:15.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 19 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/19 --osd-journal=td/osd-backfill-space/19/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+ywFgeRKOMRAAVSjO8++hxNRG4nB5ErT/jg== --osd-uuid e4a9dbec-190d-42cb-af47-8ade9543e6cf 2021-01-15T17:08:15.501 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.501+0000 7f8db40cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:15.503 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.503+0000 7f8db40cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:15.504 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.504+0000 7f8db40cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:15.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.505+0000 7f8db40cd080 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2021-01-15T17:08:15.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.505+0000 7f8db40cd080 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2021-01-15T17:08:15.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.505+0000 7f8db40cd080 -1 bluestore(td/osd-backfill-space/19/block) _read_bdev_label failed to open td/osd-backfill-space/19/block: (2) No such file or directory 2021-01-15T17:08:15.551 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:15.551+0000 7f8db40cd080 -1 bluestore(td/osd-backfill-space/19) _read_fsid unparsable uuid 2021-01-15T17:08:21.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/19/keyring 2021-01-15T17:08:21.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:08:21.228 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd19 key to auth repository 2021-01-15T17:08:21.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd19 key to auth repository 2021-01-15T17:08:21.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/19/keyring auth add osd.19 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:08:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.19 2021-01-15T17:08:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 19 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/19 --osd-journal=td/osd-backfill-space/19/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:08:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:08:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:08:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:08:21.762 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.19 2021-01-15T17:08:21.787 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:21.786+0000 7fa1f4f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:21.802+0000 7fa1f4f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:21.804+0000 7fa1f4f0c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:08:22.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 19 2021-01-15T17:08:22.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:08:22.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=19 2021-01-15T17:08:22.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:08:22.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:08:22.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:22.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:08:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2021-01-15T17:08:22.171 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:08:22.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:22.319+0000 7fa1f4f0c080 -1 Falling back to public interface 2021-01-15T17:08:22.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:23.156 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:08:23.156+0000 7fa1f4f0c080 -1 osd.19 0 log_to_monitors {default=true} 2021-01-15T17:08:23.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:23.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:23.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:08:23.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:23.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2021-01-15T17:08:23.574 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:08:23.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:08:25.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:08:25.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:08:25.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T17:08:25.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:08:25.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.19 up' 2021-01-15T17:08:25.002 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:08:25.404 INFO:tasks.workunit.client.0.smithi145.stdout:osd.19 up in weight 1 up_from 100 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6954/290349,v1:127.0.0.1:6955/290349] [v2:127.0.0.1:6956/290349,v1:127.0.0.1:6957/290349] exists,up e4a9dbec-190d-42cb-af47-8ade9543e6cf 2021-01-15T17:08:25.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:08:25.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:08:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:08:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:204: TEST_backfill_test_multi: ceph osd set-backfillfull-ratio .85 2021-01-15T17:08:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:08:26.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:08:26.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:26.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test1 1 1 2021-01-15T17:08:26.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T17:08:27.175 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T17:08:27.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T17:08:29.362 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:08:29.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:29.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test2 1 1 2021-01-15T17:08:29.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T17:08:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T17:08:30.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-01-15T17:08:32.590 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 1 2021-01-15T17:08:32.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:32.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test3 1 1 2021-01-15T17:08:32.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T17:08:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T17:08:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:34.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-01-15T17:08:35.328 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 1 2021-01-15T17:08:35.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:35.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test4 1 1 2021-01-15T17:08:35.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2021-01-15T17:08:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' already exists 2021-01-15T17:08:36.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:37.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test4 size 1 --yes-i-really-mean-it 2021-01-15T17:08:39.009 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 1 2021-01-15T17:08:39.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:39.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test5 1 1 2021-01-15T17:08:39.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2021-01-15T17:08:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' already exists 2021-01-15T17:08:40.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:41.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test5 size 1 --yes-i-really-mean-it 2021-01-15T17:08:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 1 2021-01-15T17:08:42.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:42.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test6 1 1 2021-01-15T17:08:42.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2021-01-15T17:08:42.830 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' already exists 2021-01-15T17:08:42.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test6 size 1 --yes-i-really-mean-it 2021-01-15T17:08:44.907 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 1 2021-01-15T17:08:44.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:44.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test7 1 1 2021-01-15T17:08:44.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2021-01-15T17:08:46.080 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' already exists 2021-01-15T17:08:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:47.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-01-15T17:08:47.840 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 1 2021-01-15T17:08:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:206: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:08:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:208: TEST_backfill_test_multi: create_pool test8 1 1 2021-01-15T17:08:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2021-01-15T17:08:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' already exists 2021-01-15T17:08:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:08:50.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:209: TEST_backfill_test_multi: ceph osd pool set test8 size 1 --yes-i-really-mean-it 2021-01-15T17:08:51.177 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 1 2021-01-15T17:08:51.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:212: TEST_backfill_test_multi: wait_for_clean 2021-01-15T17:08:51.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:08:51.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:08:51.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:08:51.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:08:51.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:08:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:08:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:08:51.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:08:51.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:08:51.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:08:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:08:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:08:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:08:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:08:51.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:08:51.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:08:51.836 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:08:51.837 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:08:51.837 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:08:51.837 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:08:51.837 INFO:tasks.workunit.client.0.smithi145.stderr:5 2021-01-15T17:08:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:6 2021-01-15T17:08:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:7 2021-01-15T17:08:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:8 2021-01-15T17:08:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:9 2021-01-15T17:08:51.838 INFO:tasks.workunit.client.0.smithi145.stderr:10 2021-01-15T17:08:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:11 2021-01-15T17:08:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:12 2021-01-15T17:08:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:13 2021-01-15T17:08:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:14 2021-01-15T17:08:51.839 INFO:tasks.workunit.client.0.smithi145.stderr:15 2021-01-15T17:08:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:16 2021-01-15T17:08:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:17 2021-01-15T17:08:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:18 2021-01-15T17:08:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:19' 2021-01-15T17:08:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:08:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:51.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:08:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836524 2021-01-15T17:08:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836524 2021-01-15T17:08:51.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524' 2021-01-15T17:08:51.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:51.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:08:52.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673002 2021-01-15T17:08:52.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673002 2021-01-15T17:08:52.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002' 2021-01-15T17:08:52.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:08:52.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509480 2021-01-15T17:08:52.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509480 2021-01-15T17:08:52.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480' 2021-01-15T17:08:52.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:08:52.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345959 2021-01-15T17:08:52.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345959 2021-01-15T17:08:52.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959' 2021-01-15T17:08:52.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:08:52.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182437 2021-01-15T17:08:52.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182437 2021-01-15T17:08:52.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437' 2021-01-15T17:08:52.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:08:52.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018915 2021-01-15T17:08:52.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018915 2021-01-15T17:08:52.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915' 2021-01-15T17:08:52.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-01-15T17:08:52.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855394 2021-01-15T17:08:52.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855394 2021-01-15T17:08:52.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394' 2021-01-15T17:08:52.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-01-15T17:08:52.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691872 2021-01-15T17:08:52.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691872 2021-01-15T17:08:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872' 2021-01-15T17:08:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:52.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-01-15T17:08:53.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528351 2021-01-15T17:08:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528351 2021-01-15T17:08:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351' 2021-01-15T17:08:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-01-15T17:08:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=214748364828 2021-01-15T17:08:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 214748364828 2021-01-15T17:08:53.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828' 2021-01-15T17:08:53.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2021-01-15T17:08:53.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=236223201306 2021-01-15T17:08:53.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 236223201306 2021-01-15T17:08:53.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306' 2021-01-15T17:08:53.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2021-01-15T17:08:53.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=257698037784 2021-01-15T17:08:53.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 257698037784 2021-01-15T17:08:53.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784' 2021-01-15T17:08:53.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2021-01-15T17:08:53.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=279172874264 2021-01-15T17:08:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 279172874264 2021-01-15T17:08:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264' 2021-01-15T17:08:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2021-01-15T17:08:53.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=300647710741 2021-01-15T17:08:53.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 300647710741 2021-01-15T17:08:53.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741' 2021-01-15T17:08:53.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:53.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2021-01-15T17:08:54.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=322122547219 2021-01-15T17:08:54.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 322122547219 2021-01-15T17:08:54.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219' 2021-01-15T17:08:54.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:54.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2021-01-15T17:08:54.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=343597383697 2021-01-15T17:08:54.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 343597383697 2021-01-15T17:08:54.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219 15-343597383697' 2021-01-15T17:08:54.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:54.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2021-01-15T17:08:54.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=365072220175 2021-01-15T17:08:54.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 365072220175 2021-01-15T17:08:54.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219 15-343597383697 16-365072220175' 2021-01-15T17:08:54.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:54.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2021-01-15T17:08:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=386547056653 2021-01-15T17:08:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 386547056653 2021-01-15T17:08:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219 15-343597383697 16-365072220175 17-386547056653' 2021-01-15T17:08:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:54.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2021-01-15T17:08:54.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=408021893131 2021-01-15T17:08:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 408021893131 2021-01-15T17:08:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219 15-343597383697 16-365072220175 17-386547056653 18-408021893131' 2021-01-15T17:08:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:08:54.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2021-01-15T17:08:54.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729610 2021-01-15T17:08:54.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729610 2021-01-15T17:08:54.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836524 1-42949673002 2-64424509480 3-85899345959 4-107374182437 5-128849018915 6-150323855394 7-171798691872 8-193273528351 9-214748364828 10-236223201306 11-257698037784 12-279172874264 13-300647710741 14-322122547219 15-343597383697 16-365072220175 17-386547056653 18-408021893131 19-429496729610' 2021-01-15T17:08:54.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:54.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836524 2021-01-15T17:08:54.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:54.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:08:54.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836524 2021-01-15T17:08:54.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836524 2021-01-15T17:08:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2021-01-15T17:08:54.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:08:54.720 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836524 2021-01-15T17:08:55.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836524 -lt 21474836524 2021-01-15T17:08:55.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:55.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673002 2021-01-15T17:08:55.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:55.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:08:55.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673002 2021-01-15T17:08:55.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:55.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673002 2021-01-15T17:08:55.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2021-01-15T17:08:55.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:08:55.124 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673002 2021-01-15T17:08:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673002 -lt 42949673002 2021-01-15T17:08:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:55.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509480 2021-01-15T17:08:55.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:55.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:08:55.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509480 2021-01-15T17:08:55.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:55.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509480 2021-01-15T17:08:55.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2021-01-15T17:08:55.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:08:55.546 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509480 2021-01-15T17:08:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509481 -lt 64424509480 2021-01-15T17:08:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:55.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345959 2021-01-15T17:08:55.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:08:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345959 2021-01-15T17:08:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345959 2021-01-15T17:08:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2021-01-15T17:08:55.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:08:55.954 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345959 2021-01-15T17:08:56.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345960 -lt 85899345959 2021-01-15T17:08:56.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:56.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182437 2021-01-15T17:08:56.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:08:56.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182437 2021-01-15T17:08:56.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:56.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182437 2021-01-15T17:08:56.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2021-01-15T17:08:56.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:08:56.528 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182437 2021-01-15T17:08:56.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182438 -lt 107374182437 2021-01-15T17:08:56.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018915 2021-01-15T17:08:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:56.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:08:56.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018915 2021-01-15T17:08:56.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:56.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018915 2021-01-15T17:08:56.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018915' 2021-01-15T17:08:56.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:08:56.929 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018915 2021-01-15T17:08:57.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018916 -lt 128849018915 2021-01-15T17:08:57.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:57.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-150323855394 2021-01-15T17:08:57.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:57.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2021-01-15T17:08:57.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-150323855394 2021-01-15T17:08:57.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:57.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855394 2021-01-15T17:08:57.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2021-01-15T17:08:57.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2021-01-15T17:08:57.339 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.6 seq 150323855394 2021-01-15T17:08:57.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855394 -lt 150323855394 2021-01-15T17:08:57.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:57.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-171798691872 2021-01-15T17:08:57.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:57.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2021-01-15T17:08:57.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-171798691872 2021-01-15T17:08:57.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:57.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691872 2021-01-15T17:08:57.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 171798691872' 2021-01-15T17:08:57.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2021-01-15T17:08:57.752 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.7 seq 171798691872 2021-01-15T17:08:58.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691873 -lt 171798691872 2021-01-15T17:08:58.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:58.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-193273528351 2021-01-15T17:08:58.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:58.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2021-01-15T17:08:58.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-193273528351 2021-01-15T17:08:58.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:58.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528351 2021-01-15T17:08:58.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 193273528351' 2021-01-15T17:08:58.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2021-01-15T17:08:58.197 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.8 seq 193273528351 2021-01-15T17:08:58.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528352 -lt 193273528351 2021-01-15T17:08:58.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:58.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-214748364828 2021-01-15T17:08:58.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:58.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2021-01-15T17:08:58.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-214748364828 2021-01-15T17:08:58.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:58.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=214748364828 2021-01-15T17:08:58.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 214748364828' 2021-01-15T17:08:58.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2021-01-15T17:08:58.601 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.9 seq 214748364828 2021-01-15T17:08:59.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 214748364829 -lt 214748364828 2021-01-15T17:08:59.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:59.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-236223201306 2021-01-15T17:08:59.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:59.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2021-01-15T17:08:59.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-236223201306 2021-01-15T17:08:59.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:59.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=236223201306 2021-01-15T17:08:59.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 236223201306' 2021-01-15T17:08:59.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2021-01-15T17:08:59.017 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.10 seq 236223201306 2021-01-15T17:08:59.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 236223201307 -lt 236223201306 2021-01-15T17:08:59.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:59.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-257698037784 2021-01-15T17:08:59.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:59.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2021-01-15T17:08:59.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-257698037784 2021-01-15T17:08:59.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:59.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=257698037784 2021-01-15T17:08:59.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 257698037784' 2021-01-15T17:08:59.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2021-01-15T17:08:59.429 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.11 seq 257698037784 2021-01-15T17:08:59.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 257698037785 -lt 257698037784 2021-01-15T17:08:59.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:08:59.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-279172874264 2021-01-15T17:08:59.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:08:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2021-01-15T17:08:59.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-279172874264 2021-01-15T17:08:59.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:08:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=279172874264 2021-01-15T17:08:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 279172874264' 2021-01-15T17:08:59.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2021-01-15T17:08:59.856 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.12 seq 279172874264 2021-01-15T17:09:00.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 279172874265 -lt 279172874264 2021-01-15T17:09:00.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:00.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-300647710741 2021-01-15T17:09:00.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:00.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2021-01-15T17:09:00.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-300647710741 2021-01-15T17:09:00.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:00.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=300647710741 2021-01-15T17:09:00.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 300647710741' 2021-01-15T17:09:00.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2021-01-15T17:09:00.269 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.13 seq 300647710741 2021-01-15T17:09:00.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 300647710742 -lt 300647710741 2021-01-15T17:09:00.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:00.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-322122547219 2021-01-15T17:09:00.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2021-01-15T17:09:00.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-322122547219 2021-01-15T17:09:00.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:00.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=322122547219 2021-01-15T17:09:00.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 322122547219' 2021-01-15T17:09:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2021-01-15T17:09:00.671 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.14 seq 322122547219 2021-01-15T17:09:01.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 322122547220 -lt 322122547219 2021-01-15T17:09:01.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:01.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-343597383697 2021-01-15T17:09:01.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:01.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2021-01-15T17:09:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-343597383697 2021-01-15T17:09:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=343597383697 2021-01-15T17:09:01.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 343597383697' 2021-01-15T17:09:01.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2021-01-15T17:09:01.082 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.15 seq 343597383697 2021-01-15T17:09:01.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 343597383698 -lt 343597383697 2021-01-15T17:09:01.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:01.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-365072220175 2021-01-15T17:09:01.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:01.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2021-01-15T17:09:01.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-365072220175 2021-01-15T17:09:01.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:01.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=365072220175 2021-01-15T17:09:01.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 365072220175' 2021-01-15T17:09:01.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2021-01-15T17:09:01.525 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.16 seq 365072220175 2021-01-15T17:09:01.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 365072220176 -lt 365072220175 2021-01-15T17:09:01.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:01.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-386547056653 2021-01-15T17:09:01.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:01.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2021-01-15T17:09:01.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-386547056653 2021-01-15T17:09:01.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:01.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=386547056653 2021-01-15T17:09:01.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 386547056653' 2021-01-15T17:09:01.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2021-01-15T17:09:01.954 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.17 seq 386547056653 2021-01-15T17:09:02.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 386547056654 -lt 386547056653 2021-01-15T17:09:02.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:02.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-408021893131 2021-01-15T17:09:02.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:02.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2021-01-15T17:09:02.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-408021893131 2021-01-15T17:09:02.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:02.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=408021893131 2021-01-15T17:09:02.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 408021893131' 2021-01-15T17:09:02.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2021-01-15T17:09:02.357 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.18 seq 408021893131 2021-01-15T17:09:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 408021893132 -lt 408021893131 2021-01-15T17:09:02.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:09:02.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-429496729610 2021-01-15T17:09:02.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:09:02.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2021-01-15T17:09:02.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-429496729610 2021-01-15T17:09:02.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:09:02.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729610 2021-01-15T17:09:02.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 429496729610' 2021-01-15T17:09:02.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2021-01-15T17:09:02.812 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.19 seq 429496729610 2021-01-15T17:09:03.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729611 -lt 429496729610 2021-01-15T17:09:03.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:09:03.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:09:03.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:09:03.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T17:09:03.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:09:03.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:09:03.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:09:03.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:09:03.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:09:03.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:09:03.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:09:04.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T17:09:04.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:09:04.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:09:04.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:09:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T17:09:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:09:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:09:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:214: TEST_backfill_test_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2021-01-15T17:09:04.664 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T17:09:04.664 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T17:09:04.664 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.3879e-05 s, 64.1 MB/s 2021-01-15T17:09:04.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: seq 1 600 2021-01-15T17:09:04.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:04.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:04.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:04.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:04.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:04.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:04.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:04.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:04.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:05.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:05.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:05.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:09:05.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:05.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:05.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:05.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:05.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:09:06.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:06.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:09:06.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:06.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:06.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:06.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:06.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:06.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:09:07.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:07.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:07.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:07.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:07.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:08.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:09:08.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:08.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:08.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:09:08.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:08.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:08.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:08.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:09:09.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:09.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:09.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:09.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:09.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:09.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:09.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:09.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:10.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:10.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:09:10.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:10.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:10.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:09:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:10.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:10.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:10.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:09:11.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:11.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:11.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:11.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:11.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:11.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:11.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:12.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:12.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:09:12.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:12.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:12.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:12.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:12.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:09:13.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:13.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:13.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:09:13.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:13.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:13.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:13.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:13.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:09:14.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:14.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:14.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:14.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:14.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:15.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:09:15.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:15.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:15.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:09:15.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:15.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:15.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:15.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:15.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:09:16.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:16.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:16.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:16.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:16.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:09:17.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:17.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:17.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:09:17.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:17.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:17.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:09:18.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:18.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:18.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:18.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:18.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:18.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:18.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:18.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:19.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:19.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:09:19.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:19.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:19.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:19.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:19.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:09:20.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:20.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:20.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:09:20.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:20.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:20.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:20.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:20.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:20.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:21.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:21.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:09:21.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:21.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:21.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:21.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:21.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:22.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:09:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:22.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:09:22.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:22.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:22.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:22.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:22.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:09:23.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:23.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:23.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:23.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:23.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:23.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:23.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:23.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:24.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:24.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:09:24.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:24.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:24.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:09:24.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:24.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:24.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:24.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:09:25.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:25.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:25.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:25.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:25.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:25.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:25.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:26.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:26.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:26.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:09:26.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:26.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:26.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:26.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:26.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:09:27.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:27.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:27.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:27.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:27.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:09:28.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:28.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:28.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:09:28.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:28.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:28.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:28.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:28.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:09:29.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:29.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:29.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:29.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:29.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:29.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:29.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:29.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:30.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:30.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:09:30.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:30.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:30.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:09:30.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:30.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:30.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:30.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:09:31.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:31.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:31.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:31.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:31.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:31.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:31.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:31.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:32.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:32.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:09:32.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:32.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:32.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:32.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:09:33.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:33.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:33.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:09:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:33.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:33.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:33.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:09:34.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:34.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:34.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:34.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:34.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:34.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:34.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:34.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:34.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:35.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:35.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:09:35.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:35.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:35.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:09:35.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:35.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:35.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:35.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:09:36.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:36.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:36.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:36.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:36.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:36.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:36.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:37.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:09:37.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:37.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:37.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:09:37.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:37.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:37.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:37.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:09:38.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:38.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:38.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:38.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:38.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:38.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:38.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:39.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:39.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:39.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:39.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:09:39.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:39.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:39.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:39.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:39.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:09:40.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:40.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:40.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:40.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:40.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:40.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:40.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:41.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:41.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:41.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:41.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:09:41.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:41.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:41.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:41.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:41.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:09:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:42.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:42.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:09:42.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:42.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:42.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:42.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:42.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:43.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:43.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:43.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:09:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:43.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:43.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:43.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:44.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:09:44.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:44.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:09:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:44.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:44.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:44.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:44.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:09:45.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:45.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:45.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:45.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:45.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:45.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:45.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:45.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:45.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:46.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:09:46.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:46.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:46.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:09:46.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:46.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:46.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:46.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:46.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:09:47.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:47.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:47.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:47.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:47.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:47.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:47.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:47.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:48.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:48.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:09:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:48.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:48.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:48.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:48.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:09:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:49.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:49.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:09:49.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:49.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:49.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:49.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:49.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:49.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:49.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:50.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:50.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:50.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:09:50.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:50.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:50.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:50.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:09:51.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:51.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:51.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:09:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:51.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:51.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:51.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:51.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:51.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:52.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:52.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:52.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:09:52.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:52.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:52.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:52.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:52.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:09:53.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:53.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:53.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:09:53.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:53.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:53.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:53.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:53.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:53.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:09:54.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:54.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:54.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:54.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:54.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:54.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:54.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:55.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:55.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:09:55.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:55.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:09:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:55.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:55.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:55.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:55.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:09:56.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:56.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:56.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:56.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:56.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:56.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:56.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:57.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:57.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:09:57.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:57.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:57.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:09:57.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:57.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:57.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:57.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:09:58.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:58.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:58.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:58.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:58.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:09:59.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:09:59.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:09:59.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:09:59.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:09:59.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:09:59.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:09:59.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:10:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:10:00.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:10:00.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:10:00.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:10:00.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:00.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:00.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:00.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:00.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:01.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:10:01.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:01.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:01.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:10:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:01.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:01.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:01.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:01.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:10:02.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:02.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:02.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:02.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:03.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:10:03.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:03.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:03.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:10:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:03.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:03.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:03.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:03.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:10:04.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:04.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:04.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:04.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:04.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:04.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:04.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:04.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:10:05.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:05.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:10:05.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:05.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:05.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:10:06.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:06.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:06.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:06.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:06.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:06.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:06.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:06.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:06.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:07.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:07.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:10:07.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:07.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:10:07.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:07.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:07.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:07.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:10:08.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:08.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:08.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:08.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:08.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:08.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:08.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:08.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:09.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:09.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:10:09.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:09.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:09.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:10:09.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:09.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:09.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:09.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:10:10.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:10.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:10.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:10.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:10.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:10.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:10.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:11.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:11.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:10:11.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:11.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:11.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:10:11.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:11.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:11.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:11.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:10:12.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:12.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:12.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:12.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:12.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:12.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:12.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:12.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:13.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:13.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:13.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:10:13.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:13.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:13.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:10:13.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:13.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:13.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:13.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:10:14.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:14.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:14.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:14.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:14.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:14.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:14.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:15.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:15.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:15.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:15.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:10:15.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:15.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:15.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:15.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:15.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:10:16.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:16.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:16.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:10:16.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:16.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:16.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:16.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:16.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:16.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:16.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:17.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:17.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:17.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:10:17.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:17.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:17.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:10:17.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:17.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:17.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:10:18.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:18.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:18.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:18.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:18.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:18.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:19.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:19.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:19.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:19.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:10:19.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:19.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:19.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:19.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:19.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:19.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:19.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:19.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:19.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:20.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:10:20.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:20.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:20.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:10:20.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:20.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:20.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:20.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:20.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:10:21.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:21.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:21.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:21.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:21.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:21.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:22.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:22.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:22.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:10:22.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:22.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:22.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:10:22.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:22.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:22.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:22.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:22.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:10:23.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:23.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:23.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:23.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:23.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:23.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:23.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:24.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:24.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:10:24.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:24.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:10:24.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:24.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:24.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:24.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:10:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:25.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:25.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:25.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:25.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:25.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:26.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:26.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:26.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:10:26.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:26.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:26.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:26.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:26.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:10:27.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:27.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:27.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:10:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:27.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:27.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:27.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:27.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:27.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:27.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:27.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:28.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:28.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:28.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:10:28.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:28.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:28.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:28.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:28.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:10:29.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:29.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:29.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:10:29.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:29.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:29.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:29.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:29.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:30.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:30.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:10:31.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:31.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:31.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:10:31.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:31.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:31.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:31.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:31.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:31.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:31.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:32.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:32.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:32.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:10:32.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:32.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:32.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:32.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:32.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:33.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:10:33.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:33.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:33.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:10:33.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:33.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:33.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:33.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:33.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:33.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:10:34.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:34.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:34.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:34.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:34.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:34.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:34.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:35.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:35.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:10:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:35.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:35.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:10:35.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:35.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:35.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:35.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:35.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:10:36.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:36.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:36.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:36.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:36.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:36.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:36.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:36.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:37.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:37.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:10:37.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:37.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:37.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:10:37.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:37.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:37.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:37.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:37.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:10:38.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:38.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:38.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:38.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:38.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:38.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:39.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:10:39.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:39.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:39.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:10:39.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:39.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:39.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:39.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:10:40.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:40.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:40.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:40.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:40.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:40.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:40.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:40.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:41.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:41.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:41.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:10:41.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:41.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:41.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:10:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:41.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:41.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:41.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:10:42.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:42.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:42.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:42.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:42.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:42.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:42.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:42.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:43.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:43.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:43.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:10:43.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:43.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:43.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:10:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:43.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:43.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:43.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:10:44.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:44.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:44.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:44.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:44.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:45.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:45.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:45.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:45.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:10:45.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:45.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:45.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:45.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:45.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:10:46.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:46.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:46.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:10:46.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:46.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:46.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:46.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:46.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:46.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:10:47.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:47.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:47.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:47.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:47.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:10:48.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:48.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:48.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:10:48.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:48.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:48.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:48.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:48.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:48.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:10:49.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:49.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:49.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:49.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:49.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:49.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:49.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:49.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:49.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:50.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:10:50.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:50.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:50.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:10:50.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:50.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:50.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:10:51.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:51.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:51.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:51.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:51.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:51.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:51.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:51.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:52.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:52.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:52.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:10:52.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:52.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:52.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:10:52.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:52.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:52.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:52.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:10:53.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:53.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:53.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:53.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:53.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:53.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:53.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:54.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:54.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:54.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:54.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:10:54.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:54.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:54.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:54.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:54.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:10:55.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:55.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:55.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:10:55.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:55.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:55.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:55.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:55.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:55.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:10:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:56.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:56.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:56.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:57.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:10:57.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:57.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:57.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:10:57.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:57.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:57.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:57.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:10:58.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:58.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:58.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:58.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:58.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:58.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:58.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:58.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:59.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:10:59.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:59.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:59.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:10:59.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:10:59.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:10:59.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:10:59.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:11:00.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:00.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:00.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:00.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:00.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:01.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:01.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:01.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:11:01.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:01.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:01.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:01.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:11:02.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:02.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:02.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:02.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:02.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:02.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:02.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:02.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:03.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:03.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:03.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:11:03.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:03.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:03.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:11:03.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:03.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:03.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:03.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:11:04.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:04.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:04.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:04.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:04.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:04.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:04.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:04.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:05.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:05.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:05.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:11:05.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:05.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:05.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:11:05.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:05.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:05.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:05.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:11:06.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:06.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:06.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:06.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:06.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:06.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:07.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:07.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:07.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:07.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:11:07.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:07.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:07.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:07.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:07.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:11:08.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:08.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:08.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:11:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:08.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:08.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:08.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:08.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:11:09.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:09.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:09.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:09.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:09.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:09.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:09.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:09.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:09.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:10.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:10.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:11:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:10.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:10.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:11:10.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:10.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:10.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:10.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:10.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:10.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:11:11.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:11.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:11.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:11.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:11.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:11.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:11.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:11.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:11.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:12.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:12.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:11:12.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:12.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:12.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:11:12.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:12.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:12.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:11:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:13.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:13.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:13.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:13.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:13.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:13.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:13.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:13.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:14.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:14.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:11:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:14.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:11:14.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:14.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:14.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:14.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:11:15.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:15.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:15.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:15.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:15.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:15.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:16.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:16.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:16.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:11:16.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:16.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:16.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:11:16.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:16.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:16.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:16.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:11:17.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:17.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:17.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:17.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:17.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:17.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:17.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:18.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:18.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:18.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:18.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:11:18.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:18.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:18.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:19.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:11:19.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:19.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:19.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:11:19.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:19.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:19.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:19.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:19.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:11:20.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:20.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:20.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:20.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:20.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:20.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:21.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:21.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:21.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:11:21.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:21.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:21.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:11:21.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:21.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:21.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:21.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:11:22.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:22.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:22.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:22.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:22.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:22.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:22.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:23.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:23.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:11:23.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:23.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:23.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:11:23.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:23.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:23.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:23.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:11:24.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:24.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:24.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:24.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:24.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:24.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:24.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:25.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:25.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:11:25.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:25.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:25.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:25.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:25.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:11:26.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:26.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:11:26.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:26.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:26.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:26.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:26.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:26.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:11:27.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:27.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:27.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:27.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:27.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:27.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:27.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:27.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:28.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:28.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:11:28.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:28.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:28.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:11:28.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:28.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:28.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:28.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:28.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:11:29.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:29.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:29.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:29.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:29.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:29.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:29.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:29.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:30.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:11:30.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:30.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:30.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:11:30.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:30.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:30.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:30.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:11:31.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:31.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:31.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:31.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:31.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:31.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:31.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:32.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:32.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:32.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:11:32.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:32.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:32.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:32.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:32.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:32.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:11:33.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:33.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:33.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:33.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:33.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:33.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:34.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:34.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:11:34.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:34.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:34.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:11:34.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:34.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:34.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:34.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:34.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:11:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:35.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:35.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:35.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:35.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:35.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:35.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:35.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:36.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:36.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:11:36.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:36.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:11:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:36.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:11:37.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:37.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:37.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:37.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:37.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:37.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:37.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:38.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:38.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:38.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:11:38.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:38.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:11:38.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:38.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:38.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:38.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:11:39.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:39.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:39.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:39.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:39.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:40.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:40.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:40.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:40.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:11:40.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:40.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:40.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:40.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:40.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:11:41.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:41.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:41.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:11:41.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:41.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:41.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:41.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:41.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:41.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:41.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:42.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:42.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:42.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:11:42.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:42.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:42.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:11:43.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:43.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:43.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:11:43.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:43.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:43.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:43.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:43.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:11:44.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:44.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:44.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:44.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:44.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:44.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:44.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:44.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:44.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:45.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:11:45.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:45.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:45.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:11:45.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:45.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:45.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:45.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:11:46.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:46.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:46.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:46.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:46.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:46.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:46.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:46.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:47.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:47.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:11:47.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:47.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:11:47.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:47.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:47.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:47.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:11:48.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:48.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:48.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:48.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:49.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:49.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:49.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:11:49.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:49.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:49.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:49.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:49.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:50.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:11:50.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:50.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:50.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:11:50.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:50.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:50.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:11:51.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:51.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:51.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:51.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:51.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:51.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:52.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:52.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:11:52.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:52.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:52.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:11:52.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:52.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:52.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:52.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:52.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:11:53.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:53.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:53.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:53.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:53.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:53.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:53.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:54.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:54.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:54.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:11:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:54.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:54.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:54.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:54.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:11:55.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:55.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:55.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:11:55.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:55.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:55.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:55.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:56.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:11:56.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:56.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:11:57.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:57.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:57.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:11:57.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:57.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:57.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:57.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:57.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:57.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:58.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:58.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:58.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:58.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:11:58.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:58.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:58.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:58.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:58.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:58.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:58.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:58.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:58.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:59.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:59.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:11:59.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:59.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:59.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:11:59.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:11:59.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:11:59.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:11:59.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:12:00.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:00.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:00.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:00.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:00.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:00.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:00.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:01.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:12:01.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:01.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:01.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:12:01.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:01.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:01.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:01.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:01.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:12:02.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:02.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:02.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:02.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:02.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:02.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:02.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:02.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:03.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:03.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:12:03.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:03.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:03.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:03.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:03.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:03.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:04.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:04.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:04.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:12:04.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:04.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:04.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:04.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:04.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:12:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:05.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:05.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:12:05.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:05.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:05.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:05.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:05.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:12:06.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:06.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:06.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:06.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:06.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:06.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:06.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:06.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:07.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:07.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:07.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:12:07.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:07.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:07.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:12:07.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:07.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:07.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:07.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:12:08.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:08.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:08.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:08.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:08.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:08.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:09.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:09.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:09.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:12:09.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:09.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:09.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:09.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:09.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:12:10.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:10.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:12:10.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:10.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:10.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:10.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:10.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:10.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:10.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:11.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:11.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:11.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:11.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:12:11.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:11.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:11.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:12:12.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:12.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:12.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:12:12.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:12.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:12.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:12.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:12.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:12.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:12:13.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:13.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:13.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:13.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:14.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:12:14.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:14.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:14.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:12:14.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:14.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:14.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:14.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:14.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:12:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:15.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:15.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:15.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:15.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:15.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:15.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:16.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:16.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:12:16.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:16.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:16.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:12:16.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:16.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:16.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:12:17.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:17.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:17.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:17.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:17.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:17.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:18.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:18.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:18.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:12:18.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:18.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:18.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:18.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:18.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:18.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:18.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:18.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:19.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:12:19.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:19.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:19.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:19.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:19.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:12:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:20.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:12:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:20.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:20.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:20.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:20.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:12:21.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:21.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:21.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:21.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:21.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:21.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:21.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:22.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:22.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:12:22.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:22.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:12:22.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:22.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:22.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:22.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:22.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:12:23.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:23.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:23.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:23.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:23.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:23.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:23.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:23.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:24.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:24.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:12:24.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:24.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:24.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:12:24.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:24.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:24.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:24.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:12:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:25.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:25.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:25.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:25.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:25.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:26.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:26.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:12:26.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:26.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:26.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:26.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:26.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:12:27.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:27.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:27.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:12:27.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:27.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:27.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:27.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:27.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:27.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:27.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:28.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:28.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:28.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:12:28.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:28.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:28.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:28.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:12:29.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:29.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:29.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:12:29.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:29.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:29.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:12:30.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:30.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:30.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:30.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:30.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:30.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:30.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:30.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:30.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:31.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:12:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:31.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:31.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:12:31.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:31.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:31.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:31.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:12:32.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:32.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:32.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:32.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:32.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:32.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:32.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:32.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:33.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:33.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:33.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:12:33.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:33.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:33.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:12:33.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:33.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:33.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:33.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:12:34.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:34.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:34.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:34.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:34.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:12:35.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:35.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:35.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:35.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:12:36.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:36.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:36.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:36.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:36.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:36.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:37.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:37.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:12:37.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:37.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:12:37.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:37.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:37.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:37.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:12:38.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:38.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:38.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:38.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:38.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:38.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:38.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:39.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:39.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:39.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:39.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:12:39.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:39.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:39.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:39.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:12:40.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:40.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:40.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:12:40.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:40.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:40.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:40.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:40.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:12:41.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:41.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:41.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:41.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:42.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:12:42.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:42.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:42.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:12:42.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:42.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:42.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:42.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:42.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:12:43.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:43.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:43.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:43.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:43.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:43.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:43.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:43.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:43.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:44.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:44.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:12:44.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:44.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:12:44.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:44.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:44.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:44.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:12:45.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:45.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:45.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:45.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:45.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:45.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:46.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:46.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:46.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:12:46.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:46.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:46.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:46.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:46.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:12:47.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:47.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:47.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:12:47.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:47.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:47.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:47.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:47.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:47.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:47.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:48.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:48.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:48.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:48.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:12:48.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:48.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:48.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:48.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:48.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:12:49.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:49.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:12:49.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:49.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:49.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:49.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:49.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:49.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:12:50.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:50.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:50.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:50.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:50.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:51.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:12:51.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:51.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:51.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:12:51.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:51.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:51.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:51.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:51.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:51.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:12:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:52.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:52.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:52.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:52.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:53.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:12:53.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:53.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:53.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:12:53.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:53.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:53.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:53.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:12:54.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:54.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:54.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:54.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:54.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:54.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:55.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:55.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:12:55.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:55.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:55.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:12:55.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:55.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:55.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:12:56.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:56.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:56.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:56.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:56.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:56.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:56.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:57.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:57.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:57.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:12:57.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:57.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:57.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:12:57.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:57.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:57.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:57.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:12:58.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:58.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:58.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:58.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:58.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:58.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:58.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:59.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:59.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:59.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:12:59.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:12:59.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:12:59.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:12:59.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:12:59.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:13:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:00.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:13:00.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:00.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:00.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:00.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:00.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:13:01.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:01.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:01.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:01.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:01.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:01.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:01.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:01.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:01.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:02.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:02.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:13:02.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:02.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:02.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:13:02.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:02.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:02.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:02.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:02.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:13:03.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:03.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:03.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:03.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:03.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:03.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:03.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:03.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:03.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:04.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:13:04.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:04.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:04.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:13:04.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:04.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:04.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:04.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:04.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:13:05.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:05.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:05.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:05.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:05.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:05.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:05.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:06.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:06.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:06.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:13:06.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:06.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:06.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:06.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:13:07.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:07.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:07.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:07.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:07.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:07.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:07.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:08.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:08.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:13:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:08.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:08.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:13:08.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:08.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:08.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:13:09.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:09.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:09.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:09.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:09.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:09.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:09.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:09.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:10.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:10.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:13:10.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:10.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:10.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:13:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:10.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:10.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:10.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:13:11.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:11.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:11.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:11.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:11.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:11.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:11.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:12.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:12.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:13:12.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:12.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:12.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:12.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:12.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:13:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:13.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:13.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:13:13.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:13.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:13.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:13.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:13:14.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:14.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:14.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:14.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:13:15.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:15.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:13:15.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:15.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:15.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:15.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:15.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:13:16.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:16.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:16.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:16.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:16.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:16.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:16.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:16.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:16.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:17.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:17.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:13:17.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:17.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:17.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:13:17.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:17.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:13:18.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:18.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:18.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:18.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:18.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:18.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:18.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:18.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:19.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:19.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:19.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:13:19.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:19.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:19.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:13:19.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:20.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:20.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:13:20.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:20.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:20.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:20.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:20.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:20.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:21.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:21.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:21.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:13:21.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:21.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:21.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:22.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:13:22.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:22.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:13:22.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:22.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:22.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:22.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:22.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:22.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:13:23.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:23.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:23.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:23.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:23.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:23.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:23.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:24.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:24.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:13:24.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:24.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:24.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:13:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:24.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:24.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:24.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:13:25.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:25.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:25.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:25.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:25.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:25.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:25.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:25.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:26.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:26.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:13:26.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:26.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:26.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:13:26.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:26.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:26.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:13:27.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:27.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:27.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:27.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:27.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:27.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:28.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:28.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:13:28.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:28.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:28.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:28.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:28.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:13:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:29.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:29.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:13:29.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:29.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:29.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:29.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:29.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:29.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:30.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:30.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:13:30.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:30.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:30.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:30.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:30.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:31.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:13:31.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:31.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:13:31.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:31.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:31.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:31.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:31.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:13:32.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:32.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:32.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:32.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:32.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:32.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:32.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:32.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:32.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:33.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:13:33.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:33.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:13:33.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:33.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:33.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:33.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:13:34.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:34.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:34.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:34.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:34.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:34.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:35.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:35.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:35.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:13:35.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:35.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:35.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:35.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:35.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:13:36.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:36.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:36.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:13:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:36.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:36.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:36.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:36.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:36.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:36.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:37.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:37.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:13:37.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:37.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:37.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:37.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:37.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:13:38.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:38.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:38.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:38.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:38.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:38.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:38.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:38.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:38.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:39.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:39.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:13:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:39.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:13:39.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:39.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:39.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:39.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:13:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:40.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:40.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:40.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:40.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:40.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:40.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:41.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:41.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:41.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:13:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:41.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:41.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:41.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:41.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:13:42.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:42.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:13:42.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:42.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:42.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:42.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:42.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:42.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:42.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:43.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:43.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:43.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:43.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:13:43.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:43.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:43.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:43.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:13:44.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:44.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:44.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:13:44.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:44.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:44.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:44.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:44.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:44.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:13:45.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:45.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:45.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:45.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:46.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:13:46.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:46.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:13:46.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:46.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:46.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:46.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:46.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:46.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:13:47.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:47.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:48.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:13:48.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:48.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:48.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:13:48.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:48.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:48.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:48.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:48.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:13:49.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:49.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:49.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:49.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:49.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:49.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:49.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:50.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:50.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:13:50.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:50.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:50.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:13:50.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:50.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:50.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:50.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:13:51.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:51.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:51.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:51.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:52.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:52.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:52.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:13:52.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:52.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:13:52.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:52.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:13:53.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:53.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:53.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:53.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:53.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:53.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:13:54.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:54.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:54.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:54.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:13:55.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:55.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:13:55.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:55.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:55.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:55.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:55.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:55.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:13:56.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:56.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:56.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:56.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:56.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:13:57.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:57.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:57.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:57.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:57.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:57.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:57.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:57.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:58.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:58.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:13:58.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:58.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:58.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:13:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:58.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:58.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:58.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:13:59.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:13:59.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:13:59.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:13:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:13:59.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:13:59.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:13:59.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:13:59.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:14:00.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:14:00.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:14:00.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:14:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:00.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:14:00.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:00.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:00.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:00.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:14:01.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:01.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:01.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:01.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:01.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:01.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:01.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:02.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:02.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:02.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:02.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:14:02.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:02.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:14:03.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:03.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:03.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:14:03.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:03.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:03.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:03.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:03.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:03.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:14:04.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:04.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:04.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:04.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:04.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:04.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:04.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:05.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:14:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:05.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:05.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:14:05.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:05.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:14:06.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:06.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:06.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:06.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:06.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:06.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:06.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:07.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:14:07.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:07.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:07.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:07.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:07.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:14:08.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:08.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:08.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:08.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:08.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:09.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:14:09.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:09.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:09.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:14:09.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:09.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:09.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:14:10.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:10.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:10.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:10.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:10.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:10.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:10.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:11.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:11.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:11.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:14:11.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:11.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:11.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:14:12.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:12.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:12.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:14:12.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:12.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:12.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:12.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:12.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:12.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:12.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:13.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:13.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:13.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:13.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:14:13.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:13.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:13.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:13.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:14.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:14:14.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:14.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:14.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:14:14.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:14.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:14.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:14.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:14.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:14:15.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:15.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:15.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:15.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:15.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:15.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:15.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:15.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:16.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:14:16.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:16.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:14:16.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:16.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:16.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:16.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:14:17.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:17.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:17.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:17.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:17.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:18.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:18.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:14:18.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:18.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:18.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:14:18.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:18.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:18.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:18.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:14:19.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:19.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:19.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:19.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:19.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:20.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:20.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:20.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:20.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:14:20.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:20.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:20.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:20.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:20.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:14:21.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:21.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:21.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:14:21.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:21.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:21.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:21.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:21.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:21.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:14:22.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:22.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:22.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:22.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:22.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:23.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:14:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:23.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:23.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:14:23.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:23.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:23.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:23.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:23.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:14:24.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:24.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:24.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:24.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:24.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:24.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:24.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:24.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:24.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:25.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:14:25.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:25.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:14:25.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:25.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:25.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:25.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:14:26.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:26.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:26.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:26.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:26.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:26.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:26.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:26.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:27.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:27.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:14:27.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:27.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:27.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:14:28.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:28.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:28.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:14:28.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:28.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:28.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:28.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:28.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:28.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:14:29.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:29.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:29.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:29.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:29.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:29.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:30.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:30.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:14:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:30.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:30.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:14:30.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:30.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:30.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:30.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:14:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:31.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:31.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:31.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:31.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:31.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:31.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:31.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:32.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:32.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:14:32.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:32.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:32.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:14:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:32.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:32.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:32.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:14:33.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:33.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:33.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:33.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:33.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:33.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:33.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:34.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:34.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:14:34.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:34.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:34.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:34.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:34.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:14:35.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:35.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:35.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:14:35.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:35.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:35.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:35.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:35.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:35.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:36.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:36.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:36.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:36.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:14:36.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:36.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:36.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:36.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:36.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:37.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:14:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:37.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:37.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:14:37.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:37.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:37.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:14:38.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:38.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:38.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:38.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:38.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:38.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:38.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:38.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:38.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:39.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:39.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:14:39.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:39.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:14:40.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:40.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:40.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:40.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:40.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:41.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:14:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:41.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:14:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:41.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:41.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:41.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:41.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:14:42.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:42.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:42.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:42.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:42.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:42.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:42.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:42.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:43.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:43.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:14:43.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:43.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:43.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:14:43.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:43.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:43.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:43.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:14:44.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:44.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:44.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:44.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:44.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:45.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:45.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:45.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:14:45.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:45.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:45.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:45.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:45.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:14:46.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:46.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:46.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:14:46.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:46.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:46.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:46.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:46.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:46.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:46.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:47.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:47.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:47.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:47.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:14:47.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:47.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:47.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:47.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:47.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:48.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:14:48.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:48.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:48.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:14:48.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:48.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:48.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:48.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:48.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:14:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:49.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:49.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:49.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:49.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:49.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:50.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:50.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:14:50.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:50.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:50.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:14:50.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:50.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:50.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:50.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:14:51.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:51.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:51.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:51.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:51.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:52.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:52.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:52.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:52.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:14:52.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:52.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:52.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:52.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:52.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:53.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:14:53.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:53.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:53.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:14:53.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:53.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:53.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:53.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:53.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:14:54.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:54.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:54.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:54.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:54.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:55.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:14:55.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:55.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:55.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:14:55.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:55.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:55.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:55.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:55.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:55.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:14:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:56.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:56.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:56.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:56.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:56.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:56.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:57.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:57.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:14:57.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:57.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:57.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:14:57.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:57.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:57.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:57.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:57.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:14:58.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:58.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:58.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:58.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:58.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:58.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:58.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:59.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:59.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:59.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:14:59.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:59.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:59.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:14:59.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:14:59.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:14:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:14:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:15:00.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:00.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:00.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:00.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:00.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:00.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:00.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:01.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:01.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:15:01.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:01.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:01.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:01.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:01.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:01.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:01.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:02.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:15:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:02.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:15:02.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:02.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:02.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:02.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:02.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:15:03.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:03.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:03.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:03.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:03.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:03.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:03.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:04.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:15:04.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:04.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:04.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:15:04.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:04.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:04.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:04.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:04.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:15:05.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:05.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:05.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:05.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:05.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:05.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:05.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:05.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:06.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:06.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:06.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:15:06.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:06.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:06.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:06.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:06.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:15:07.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:07.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:07.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:15:07.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:07.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:07.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:07.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:07.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:08.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:08.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:08.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:08.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:15:08.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:08.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:08.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:08.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:08.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:09.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:15:09.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:09.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:09.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:15:09.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:09.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:09.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:09.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:09.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:15:10.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:10.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:10.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:10.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:10.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:10.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:10.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:10.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:10.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:11.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:15:11.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:11.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:11.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:11.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:11.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:11.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:12.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:12.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:12.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:12.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:15:12.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:12.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:12.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:12.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:12.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:13.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:15:13.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:13.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:13.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:15:13.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:13.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:13.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:13.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:13.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:15:14.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:14.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:14.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:14.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:14.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:15.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:15:15.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:15.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:15.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:15:15.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:15.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:15.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:15.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:15.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:15:16.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:16.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:16.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:16.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:16.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:16.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:17.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:17.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:15:17.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:17.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:17.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:17.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:17.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:15:18.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:18.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:18.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:15:18.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:18.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:18.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:18.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:18.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:18.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:18.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:19.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:19.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:15:19.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:19.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:19.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:19.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:19.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:20.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:15:20.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:20.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:15:20.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:20.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:20.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:20.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:15:21.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:21.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:21.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:21.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:21.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:22.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:22.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:15:22.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:22.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:22.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:15:22.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:22.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:22.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:22.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:15:23.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:23.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:23.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:23.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:23.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:23.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:23.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:23.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:24.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:24.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:24.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:15:24.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:24.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:24.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:15:24.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:24.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:24.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:24.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:15:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:25.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:25.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:25.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:25.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:25.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:25.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:26.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:15:26.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:26.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:26.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:15:27.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:27.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:27.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:15:27.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:27.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:27.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:27.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:27.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:27.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:27.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:28.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:28.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:28.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:28.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:15:28.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:28.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:28.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:29.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:15:29.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:29.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:29.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:15:29.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:29.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:29.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:15:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:30.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:30.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:30.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:30.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:30.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:30.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:30.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:31.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:15:31.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:31.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:31.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:15:31.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:31.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:31.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:31.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:15:32.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:32.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:32.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:32.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:32.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:32.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:33.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:33.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:33.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:33.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:15:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:33.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:33.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:33.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:33.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:15:34.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:34.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:34.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:15:34.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:34.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:34.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:34.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:34.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:34.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:34.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:35.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:15:35.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:35.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:35.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:35.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:15:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:36.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:36.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:15:36.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:36.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:36.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:36.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:15:37.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:37.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:37.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:37.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:37.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:37.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:38.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:38.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:38.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:15:38.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:38.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:38.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:15:38.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:38.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:38.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:38.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:15:39.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:39.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:39.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:39.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:39.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:39.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:40.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:40.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:40.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:15:40.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:40.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:40.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:40.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:40.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:15:41.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:41.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:41.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:15:41.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:41.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:41.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:41.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:41.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:41.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:42.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:42.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:42.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:42.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:42.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:43.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:15:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:43.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:43.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:15:43.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:43.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:43.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:43.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:15:44.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:44.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:44.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:44.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:44.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:45.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:45.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:45.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:15:45.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:45.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:45.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:15:45.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:45.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:45.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:45.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:15:46.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:46.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:46.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:46.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:46.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:46.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:46.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:46.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:47.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:47.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:47.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:15:47.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:47.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:47.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:48.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:15:48.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:48.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:48.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:15:48.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:48.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:48.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:48.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:48.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:48.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:49.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:49.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:49.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:49.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:15:49.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:49.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:49.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:49.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:49.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:50.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:15:50.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:50.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:50.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:15:50.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:50.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:50.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:50.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:50.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:50.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:15:51.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:51.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:51.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:51.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:51.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:51.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:51.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:51.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:51.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:52.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:15:52.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:52.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:52.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:15:52.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:52.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:52.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:52.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:15:53.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:53.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:53.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:53.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:53.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:53.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:53.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:53.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:54.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:54.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:15:54.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:54.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:54.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:54.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:54.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:15:55.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:55.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:55.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:15:55.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:55.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:55.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:55.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:55.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:55.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:56.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:56.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:56.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:56.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:15:56.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:56.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:56.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:57.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:15:57.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:57.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:57.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:15:57.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:57.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:57.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:57.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:57.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:15:58.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:58.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:58.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:58.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:58.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:58.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:59.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:59.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:59.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:15:59.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:59.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:59.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:15:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:15:59.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:15:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:15:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:16:00.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:00.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:00.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:00.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:00.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:00.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:01.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:01.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:16:01.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:01.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:01.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:01.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:16:02.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:02.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:02.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:16:02.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:02.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:02.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:02.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:16:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:03.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:03.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:04.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:04.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:16:04.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:04.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:16:04.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:04.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:04.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:04.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:16:05.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:05.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:05.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:05.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:05.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:05.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:05.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:06.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:06.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:06.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:16:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:06.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:06.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:06.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:06.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:16:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:07.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:07.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:16:07.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:215: TEST_backfill_test_multi: for o in $(seq 1 $objects) 2021-01-15T17:16:07.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:07.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test1 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:07.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test2 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:07.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:07.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test3 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test4 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test5 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test6 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test7 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:217: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:219: TEST_backfill_test_multi: rados -p test8 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:16:08.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:223: TEST_backfill_test_multi: ceph pg dump pgs 2021-01-15T17:16:08.937 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:16:08.938 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 596 0 0 0 0 2441216 0 0 96 96 active+clean 2021-01-15T17:08:52.082021+0000 148'596 148:615 [3] 3 [3] 3 0'0 2021-01-15T17:08:48.486677+0000 0'0 2021-01-15T17:08:48.486677+0000 0 2021-01-15T17:16:08.938 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 593 0 0 0 0 2428928 0 0 93 93 active+clean 2021-01-15T17:08:48.712201+0000 148'593 148:610 [5] 5 [5] 5 0'0 2021-01-15T17:08:45.307362+0000 0'0 2021-01-15T17:08:45.307362+0000 0 2021-01-15T17:16:08.939 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 593 0 0 0 0 2428928 0 0 93 93 active+clean 2021-01-15T17:08:32.712886+0000 148'593 148:630 [17] 17 [17] 17 0'0 2021-01-15T17:08:29.823784+0000 0'0 2021-01-15T17:08:29.823784+0000 0 2021-01-15T17:16:08.939 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 595 0 0 0 0 2437120 0 0 95 95 active+clean 2021-01-15T17:08:29.957710+0000 148'595 148:641 [8] 8 [8] 8 0'0 2021-01-15T17:08:26.766428+0000 0'0 2021-01-15T17:08:26.766428+0000 0 2021-01-15T17:16:08.940 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 594 0 0 0 0 2433024 0 0 94 94 active+clean 2021-01-15T17:08:36.349876+0000 148'594 148:624 [1] 1 [1] 1 0'0 2021-01-15T17:08:33.006872+0000 0'0 2021-01-15T17:08:33.006872+0000 0 2021-01-15T17:16:08.940 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 596 0 0 0 0 2441216 0 0 96 96 active+clean 2021-01-15T17:08:39.788633+0000 148'596 148:629 [4] 4 [4] 4 0'0 2021-01-15T17:08:36.021761+0000 0'0 2021-01-15T17:08:36.021761+0000 0 2021-01-15T17:16:08.940 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 593 0 0 0 0 2428928 0 0 93 93 active+clean 2021-01-15T17:08:44.084271+0000 148'593 148:624 [12] 12 [12] 12 0'0 2021-01-15T17:08:39.553212+0000 0'0 2021-01-15T17:08:39.553212+0000 0 2021-01-15T17:16:08.941 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 596 0 0 0 0 2441216 0 0 96 96 active+clean 2021-01-15T17:08:45.345394+0000 148'596 148:616 [7] 7 [7] 7 0'0 2021-01-15T17:08:42.554595+0000 0'0 2021-01-15T17:08:42.554595+0000 0 2021-01-15T17:16:08.941 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:16:08.941 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:16:08.942 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:16:08.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:16:08.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:08.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test1 size 2 2021-01-15T17:16:09.855 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:16:09.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:09.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test2 size 2 2021-01-15T17:16:10.644 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T17:16:10.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:10.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test3 size 2 2021-01-15T17:16:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T17:16:11.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:11.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test4 size 2 2021-01-15T17:16:12.802 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2021-01-15T17:16:12.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:12.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test5 size 2 2021-01-15T17:16:14.840 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 2 2021-01-15T17:16:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test6 size 2 2021-01-15T17:16:16.945 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 2 2021-01-15T17:16:16.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:16.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test7 size 2 2021-01-15T17:16:18.812 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T17:16:18.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:225: TEST_backfill_test_multi: for p in $(seq 1 $pools) 2021-01-15T17:16:18.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:227: TEST_backfill_test_multi: ceph osd pool set test8 size 2 2021-01-15T17:16:19.973 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 2 2021-01-15T17:16:19.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:229: TEST_backfill_test_multi: sleep 30 2021-01-15T17:16:49.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:231: TEST_backfill_test_multi: wait_for_not_backfilling 240 2021-01-15T17:16:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:16:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:16:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:16:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:16:49.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:16:49.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:16:49.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:16:49.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:16:49.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:16:49.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:16:49.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:16:49.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:16:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:16:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:16:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:16:50.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:16:50.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:16:50.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:16:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:16:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:16:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:16:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:16:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:5 2021-01-15T17:16:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:6 2021-01-15T17:16:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:7 2021-01-15T17:16:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:8 2021-01-15T17:16:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:9 2021-01-15T17:16:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:10 2021-01-15T17:16:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:11 2021-01-15T17:16:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:12 2021-01-15T17:16:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:13 2021-01-15T17:16:50.532 INFO:tasks.workunit.client.0.smithi145.stderr:14 2021-01-15T17:16:50.532 INFO:tasks.workunit.client.0.smithi145.stderr:15 2021-01-15T17:16:50.532 INFO:tasks.workunit.client.0.smithi145.stderr:16 2021-01-15T17:16:50.532 INFO:tasks.workunit.client.0.smithi145.stderr:17 2021-01-15T17:16:50.532 INFO:tasks.workunit.client.0.smithi145.stderr:18 2021-01-15T17:16:50.533 INFO:tasks.workunit.client.0.smithi145.stderr:19' 2021-01-15T17:16:50.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:16:50.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:50.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:16:50.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836621 2021-01-15T17:16:50.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836621 2021-01-15T17:16:50.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621' 2021-01-15T17:16:50.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:50.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:16:50.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673100 2021-01-15T17:16:50.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673100 2021-01-15T17:16:50.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100' 2021-01-15T17:16:50.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:50.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:16:50.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509578 2021-01-15T17:16:50.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509578 2021-01-15T17:16:50.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578' 2021-01-15T17:16:51.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:16:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899346057 2021-01-15T17:16:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899346057 2021-01-15T17:16:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057' 2021-01-15T17:16:51.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:16:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182535 2021-01-15T17:16:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182535 2021-01-15T17:16:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535' 2021-01-15T17:16:51.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:16:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849019013 2021-01-15T17:16:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849019013 2021-01-15T17:16:51.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013' 2021-01-15T17:16:51.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-01-15T17:16:51.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855491 2021-01-15T17:16:51.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855491 2021-01-15T17:16:51.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491' 2021-01-15T17:16:51.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-01-15T17:16:51.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691970 2021-01-15T17:16:51.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691970 2021-01-15T17:16:51.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970' 2021-01-15T17:16:51.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-01-15T17:16:51.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528449 2021-01-15T17:16:51.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528449 2021-01-15T17:16:51.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449' 2021-01-15T17:16:51.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-01-15T17:16:51.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=214748364926 2021-01-15T17:16:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 214748364926 2021-01-15T17:16:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926' 2021-01-15T17:16:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:51.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2021-01-15T17:16:52.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=236223201404 2021-01-15T17:16:52.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 236223201404 2021-01-15T17:16:52.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404' 2021-01-15T17:16:52.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:52.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2021-01-15T17:16:52.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=257698037882 2021-01-15T17:16:52.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 257698037882 2021-01-15T17:16:52.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882' 2021-01-15T17:16:52.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:52.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2021-01-15T17:16:52.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=279172874361 2021-01-15T17:16:52.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 279172874361 2021-01-15T17:16:52.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361' 2021-01-15T17:16:52.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:52.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2021-01-15T17:16:52.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=300647710838 2021-01-15T17:16:52.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 300647710838 2021-01-15T17:16:52.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838' 2021-01-15T17:16:52.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:52.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2021-01-15T17:16:52.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=322122547316 2021-01-15T17:16:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 322122547316 2021-01-15T17:16:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316' 2021-01-15T17:16:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:52.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2021-01-15T17:16:53.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=343597383794 2021-01-15T17:16:53.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 343597383794 2021-01-15T17:16:53.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316 15-343597383794' 2021-01-15T17:16:53.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:53.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2021-01-15T17:16:53.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=365072220272 2021-01-15T17:16:53.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 365072220272 2021-01-15T17:16:53.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316 15-343597383794 16-365072220272' 2021-01-15T17:16:53.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:53.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2021-01-15T17:16:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=386547056750 2021-01-15T17:16:53.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 386547056750 2021-01-15T17:16:53.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316 15-343597383794 16-365072220272 17-386547056750' 2021-01-15T17:16:53.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:53.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2021-01-15T17:16:53.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=408021893228 2021-01-15T17:16:53.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 408021893228 2021-01-15T17:16:53.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316 15-343597383794 16-365072220272 17-386547056750 18-408021893228' 2021-01-15T17:16:53.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:16:53.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2021-01-15T17:16:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729707 2021-01-15T17:16:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729707 2021-01-15T17:16:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836621 1-42949673100 2-64424509578 3-85899346057 4-107374182535 5-128849019013 6-150323855491 7-171798691970 8-193273528449 9-214748364926 10-236223201404 11-257698037882 12-279172874361 13-300647710838 14-322122547316 15-343597383794 16-365072220272 17-386547056750 18-408021893228 19-429496729707' 2021-01-15T17:16:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:53.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836621 2021-01-15T17:16:53.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:53.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:16:53.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836621 2021-01-15T17:16:53.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836621 2021-01-15T17:16:53.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836621' 2021-01-15T17:16:53.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:16:53.591 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836621 2021-01-15T17:16:54.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836622 -lt 21474836621 2021-01-15T17:16:54.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:54.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673100 2021-01-15T17:16:54.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:54.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:16:54.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673100 2021-01-15T17:16:54.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:54.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673100 2021-01-15T17:16:54.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673100' 2021-01-15T17:16:54.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:16:54.033 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673100 2021-01-15T17:16:54.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673100 -lt 42949673100 2021-01-15T17:16:54.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:54.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509578 2021-01-15T17:16:54.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:54.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:16:54.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509578 2021-01-15T17:16:54.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:54.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509578 2021-01-15T17:16:54.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509578' 2021-01-15T17:16:54.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:16:54.539 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509578 2021-01-15T17:16:54.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509579 -lt 64424509578 2021-01-15T17:16:54.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:54.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899346057 2021-01-15T17:16:54.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:54.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:16:54.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899346057 2021-01-15T17:16:54.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899346057 2021-01-15T17:16:54.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899346057' 2021-01-15T17:16:54.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:16:54.947 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899346057 2021-01-15T17:16:55.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899346057 -lt 85899346057 2021-01-15T17:16:55.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:55.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182535 2021-01-15T17:16:55.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:55.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:16:55.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182535 2021-01-15T17:16:55.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:55.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182535 2021-01-15T17:16:55.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182535' 2021-01-15T17:16:55.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:16:55.399 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182535 2021-01-15T17:16:55.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182535 -lt 107374182535 2021-01-15T17:16:55.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:55.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849019013 2021-01-15T17:16:55.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:16:55.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849019013 2021-01-15T17:16:55.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:55.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849019013 2021-01-15T17:16:55.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849019013' 2021-01-15T17:16:55.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:16:55.803 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849019013 2021-01-15T17:16:56.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849019014 -lt 128849019013 2021-01-15T17:16:56.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:56.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-150323855491 2021-01-15T17:16:56.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:56.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2021-01-15T17:16:56.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-150323855491 2021-01-15T17:16:56.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:56.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855491 2021-01-15T17:16:56.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 150323855491' 2021-01-15T17:16:56.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2021-01-15T17:16:56.236 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.6 seq 150323855491 2021-01-15T17:16:56.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855492 -lt 150323855491 2021-01-15T17:16:56.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:56.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-171798691970 2021-01-15T17:16:56.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:56.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2021-01-15T17:16:56.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-171798691970 2021-01-15T17:16:56.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:56.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691970 2021-01-15T17:16:56.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 171798691970' 2021-01-15T17:16:56.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2021-01-15T17:16:56.640 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.7 seq 171798691970 2021-01-15T17:16:57.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691971 -lt 171798691970 2021-01-15T17:16:57.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:57.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-193273528449 2021-01-15T17:16:57.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:57.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2021-01-15T17:16:57.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-193273528449 2021-01-15T17:16:57.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:57.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528449 2021-01-15T17:16:57.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 193273528449' 2021-01-15T17:16:57.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2021-01-15T17:16:57.047 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.8 seq 193273528449 2021-01-15T17:16:57.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528450 -lt 193273528449 2021-01-15T17:16:57.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:57.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-214748364926 2021-01-15T17:16:57.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:57.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2021-01-15T17:16:57.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-214748364926 2021-01-15T17:16:57.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=214748364926 2021-01-15T17:16:57.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 214748364926' 2021-01-15T17:16:57.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2021-01-15T17:16:57.480 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.9 seq 214748364926 2021-01-15T17:16:57.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 214748364927 -lt 214748364926 2021-01-15T17:16:57.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:57.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-236223201404 2021-01-15T17:16:57.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:57.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2021-01-15T17:16:57.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-236223201404 2021-01-15T17:16:57.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:57.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=236223201404 2021-01-15T17:16:57.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 236223201404' 2021-01-15T17:16:57.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2021-01-15T17:16:57.887 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.10 seq 236223201404 2021-01-15T17:16:58.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 236223201405 -lt 236223201404 2021-01-15T17:16:58.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:58.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-257698037882 2021-01-15T17:16:58.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:58.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2021-01-15T17:16:58.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-257698037882 2021-01-15T17:16:58.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:58.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=257698037882 2021-01-15T17:16:58.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 257698037882' 2021-01-15T17:16:58.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2021-01-15T17:16:58.301 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.11 seq 257698037882 2021-01-15T17:16:58.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 257698037883 -lt 257698037882 2021-01-15T17:16:58.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:58.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-279172874361 2021-01-15T17:16:58.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2021-01-15T17:16:58.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-279172874361 2021-01-15T17:16:58.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:58.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=279172874361 2021-01-15T17:16:58.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 279172874361' 2021-01-15T17:16:58.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2021-01-15T17:16:58.744 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.12 seq 279172874361 2021-01-15T17:16:59.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 279172874362 -lt 279172874361 2021-01-15T17:16:59.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:59.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-300647710838 2021-01-15T17:16:59.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:59.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2021-01-15T17:16:59.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-300647710838 2021-01-15T17:16:59.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=300647710838 2021-01-15T17:16:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 300647710838' 2021-01-15T17:16:59.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2021-01-15T17:16:59.150 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.13 seq 300647710838 2021-01-15T17:16:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 300647710839 -lt 300647710838 2021-01-15T17:16:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:16:59.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-322122547316 2021-01-15T17:16:59.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:16:59.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2021-01-15T17:16:59.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-322122547316 2021-01-15T17:16:59.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:16:59.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=322122547316 2021-01-15T17:16:59.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 322122547316' 2021-01-15T17:16:59.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2021-01-15T17:16:59.561 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.14 seq 322122547316 2021-01-15T17:17:00.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 322122547317 -lt 322122547316 2021-01-15T17:17:00.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:17:00.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-343597383794 2021-01-15T17:17:00.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:17:00.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2021-01-15T17:17:00.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-343597383794 2021-01-15T17:17:00.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:17:00.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=343597383794 2021-01-15T17:17:00.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 343597383794' 2021-01-15T17:17:00.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2021-01-15T17:17:00.019 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.15 seq 343597383794 2021-01-15T17:17:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 343597383796 -lt 343597383794 2021-01-15T17:17:00.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:17:00.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-365072220272 2021-01-15T17:17:00.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:17:00.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2021-01-15T17:17:00.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-365072220272 2021-01-15T17:17:00.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:17:00.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=365072220272 2021-01-15T17:17:00.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 365072220272' 2021-01-15T17:17:00.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2021-01-15T17:17:00.515 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.16 seq 365072220272 2021-01-15T17:17:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 365072220274 -lt 365072220272 2021-01-15T17:17:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:17:01.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-386547056750 2021-01-15T17:17:01.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:17:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2021-01-15T17:17:01.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-386547056750 2021-01-15T17:17:01.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:17:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=386547056750 2021-01-15T17:17:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 386547056750' 2021-01-15T17:17:01.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2021-01-15T17:17:01.035 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.17 seq 386547056750 2021-01-15T17:17:01.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 386547056752 -lt 386547056750 2021-01-15T17:17:01.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:17:01.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-408021893228 2021-01-15T17:17:01.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:17:01.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2021-01-15T17:17:01.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-408021893228 2021-01-15T17:17:01.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:17:01.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=408021893228 2021-01-15T17:17:01.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 408021893228' 2021-01-15T17:17:01.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2021-01-15T17:17:01.478 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.18 seq 408021893228 2021-01-15T17:17:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 408021893230 -lt 408021893228 2021-01-15T17:17:01.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:17:01.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-429496729707 2021-01-15T17:17:01.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:17:01.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2021-01-15T17:17:01.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-429496729707 2021-01-15T17:17:01.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:17:01.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729707 2021-01-15T17:17:01.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 429496729707' 2021-01-15T17:17:01.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2021-01-15T17:17:01.881 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.19 seq 429496729707 2021-01-15T17:17:02.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729709 -lt 429496729707 2021-01-15T17:17:02.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:17:02.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:17:02.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:17:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 8 == 0 2021-01-15T17:17:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:17:02.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:17:02.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:17:02.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:17:02.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:17:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:17:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:17:03.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:17:03.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:17:03.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' -1 2021-01-15T17:17:03.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:17:03.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=4 2021-01-15T17:17:03.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:17:08.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:17:08.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:17:08.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:17:08.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:17:08.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:17:08.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:17:08.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:17:08.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:17:08.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:17:08.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:17:08.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:17:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2021-01-15T17:17:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2021-01-15T17:17:18.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:17:18.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:17:18.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:17:18.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:17:18.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:17:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:17:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:17:18.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:17:19.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:17:19.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:17:19.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:17:19.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2021-01-15T17:17:19.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:17:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:17:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:17:34.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:17:34.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:17:34.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:17:34.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:17:34.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:17:34.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:17:34.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:17:34.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:17:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:17:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2021-01-15T17:17:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:17:49.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:17:49.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:17:49.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:17:49.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:17:49.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:17:49.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:17:49.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:17:49.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:17:49.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:17:49.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:17:49.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:17:49.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2021-01-15T17:17:49.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:18:04.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:18:04.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:18:04.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:18:04.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:18:04.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:18:04.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:18:04.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:18:04.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:18:05.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:18:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:18:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:18:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 5 >= 17 )) 2021-01-15T17:18:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:18:20.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:18:20.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:18:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:18:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:18:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:18:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:18:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:18:20.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:18:20.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:18:20.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:18:20.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:18:20.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 6 >= 17 )) 2021-01-15T17:18:20.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:18:35.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:18:35.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:18:35.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:18:35.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:18:35.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:18:35.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:18:35.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:18:35.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:18:36.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:18:36.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:18:36.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:18:36.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 7 >= 17 )) 2021-01-15T17:18:36.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:18:51.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:18:51.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:18:51.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:18:51.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:18:51.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:18:51.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:18:51.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:18:51.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:18:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=4 2021-01-15T17:18:51.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 4 = 0 2021-01-15T17:18:51.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 4 '!=' 4 2021-01-15T17:18:51.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 8 >= 17 )) 2021-01-15T17:18:51.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:19:06.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:19:06.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:19:06.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:19:06.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:19:06.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:19:06.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:19:06.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:19:06.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:19:07.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=2 2021-01-15T17:19:07.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 2 = 0 2021-01-15T17:19:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 2 '!=' 4 2021-01-15T17:19:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:19:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=2 2021-01-15T17:19:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:19:12.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:19:12.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:19:12.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:19:12.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:19:12.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:19:12.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:19:12.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:19:12.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:19:12.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:19:12.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:19:12.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:19:12.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:19:12.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:232: TEST_backfill_test_multi: wait_for_not_activating 60 2021-01-15T17:19:12.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:19:12.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:19:12.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:19:12.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:19:12.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:19:12.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:19:12.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:19:12.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:19:12.478 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:19:12.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:19:12.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:19:12.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:19:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:19:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:19:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:19:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:19:12.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:19:12.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:19:12.927 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:19:12.928 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:19:12.928 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:19:12.928 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:19:12.928 INFO:tasks.workunit.client.0.smithi145.stderr:5 2021-01-15T17:19:12.929 INFO:tasks.workunit.client.0.smithi145.stderr:6 2021-01-15T17:19:12.929 INFO:tasks.workunit.client.0.smithi145.stderr:7 2021-01-15T17:19:12.929 INFO:tasks.workunit.client.0.smithi145.stderr:8 2021-01-15T17:19:12.929 INFO:tasks.workunit.client.0.smithi145.stderr:9 2021-01-15T17:19:12.929 INFO:tasks.workunit.client.0.smithi145.stderr:10 2021-01-15T17:19:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:11 2021-01-15T17:19:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:12 2021-01-15T17:19:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:13 2021-01-15T17:19:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:14 2021-01-15T17:19:12.931 INFO:tasks.workunit.client.0.smithi145.stderr:15 2021-01-15T17:19:12.931 INFO:tasks.workunit.client.0.smithi145.stderr:16 2021-01-15T17:19:12.931 INFO:tasks.workunit.client.0.smithi145.stderr:17 2021-01-15T17:19:12.931 INFO:tasks.workunit.client.0.smithi145.stderr:18 2021-01-15T17:19:12.931 INFO:tasks.workunit.client.0.smithi145.stderr:19' 2021-01-15T17:19:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:19:12.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:12.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:19:13.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836652 2021-01-15T17:19:13.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836652 2021-01-15T17:19:13.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652' 2021-01-15T17:19:13.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:19:13.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673130 2021-01-15T17:19:13.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673130 2021-01-15T17:19:13.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130' 2021-01-15T17:19:13.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:19:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509609 2021-01-15T17:19:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509609 2021-01-15T17:19:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609' 2021-01-15T17:19:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:19:13.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899346087 2021-01-15T17:19:13.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899346087 2021-01-15T17:19:13.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087' 2021-01-15T17:19:13.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:19:13.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182565 2021-01-15T17:19:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182565 2021-01-15T17:19:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565' 2021-01-15T17:19:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:19:13.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849019043 2021-01-15T17:19:13.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849019043 2021-01-15T17:19:13.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043' 2021-01-15T17:19:13.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2021-01-15T17:19:13.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855522 2021-01-15T17:19:13.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855522 2021-01-15T17:19:13.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522' 2021-01-15T17:19:13.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:13.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2021-01-15T17:19:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798692000 2021-01-15T17:19:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798692000 2021-01-15T17:19:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000' 2021-01-15T17:19:14.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2021-01-15T17:19:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528479 2021-01-15T17:19:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528479 2021-01-15T17:19:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479' 2021-01-15T17:19:14.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2021-01-15T17:19:14.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=214748364957 2021-01-15T17:19:14.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 214748364957 2021-01-15T17:19:14.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957' 2021-01-15T17:19:14.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.10 flush_pg_stats 2021-01-15T17:19:14.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=236223201435 2021-01-15T17:19:14.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 236223201435 2021-01-15T17:19:14.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435' 2021-01-15T17:19:14.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.11 flush_pg_stats 2021-01-15T17:19:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=257698037913 2021-01-15T17:19:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 257698037913 2021-01-15T17:19:14.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913' 2021-01-15T17:19:14.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.12 flush_pg_stats 2021-01-15T17:19:14.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=279172874392 2021-01-15T17:19:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 279172874392 2021-01-15T17:19:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392' 2021-01-15T17:19:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.13 flush_pg_stats 2021-01-15T17:19:14.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=300647710869 2021-01-15T17:19:14.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 300647710869 2021-01-15T17:19:14.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869' 2021-01-15T17:19:14.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:14.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.14 flush_pg_stats 2021-01-15T17:19:15.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=322122547347 2021-01-15T17:19:15.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 322122547347 2021-01-15T17:19:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347' 2021-01-15T17:19:15.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:15.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.15 flush_pg_stats 2021-01-15T17:19:15.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=343597383825 2021-01-15T17:19:15.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 343597383825 2021-01-15T17:19:15.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347 15-343597383825' 2021-01-15T17:19:15.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:15.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.16 flush_pg_stats 2021-01-15T17:19:15.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=365072220303 2021-01-15T17:19:15.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 365072220303 2021-01-15T17:19:15.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347 15-343597383825 16-365072220303' 2021-01-15T17:19:15.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:15.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.17 flush_pg_stats 2021-01-15T17:19:15.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=386547056781 2021-01-15T17:19:15.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 386547056781 2021-01-15T17:19:15.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347 15-343597383825 16-365072220303 17-386547056781' 2021-01-15T17:19:15.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:15.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.18 flush_pg_stats 2021-01-15T17:19:15.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=408021893259 2021-01-15T17:19:15.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 408021893259 2021-01-15T17:19:15.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347 15-343597383825 16-365072220303 17-386547056781 18-408021893259' 2021-01-15T17:19:15.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:19:15.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.19 flush_pg_stats 2021-01-15T17:19:15.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=429496729738 2021-01-15T17:19:15.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 429496729738 2021-01-15T17:19:15.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836652 1-42949673130 2-64424509609 3-85899346087 4-107374182565 5-128849019043 6-150323855522 7-171798692000 8-193273528479 9-214748364957 10-236223201435 11-257698037913 12-279172874392 13-300647710869 14-322122547347 15-343597383825 16-365072220303 17-386547056781 18-408021893259 19-429496729738' 2021-01-15T17:19:15.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:15.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836652 2021-01-15T17:19:15.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:15.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:19:15.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836652 2021-01-15T17:19:15.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:15.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836652 2021-01-15T17:19:15.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836652' 2021-01-15T17:19:15.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:19:15.775 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836652 2021-01-15T17:19:16.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836652 -lt 21474836652 2021-01-15T17:19:16.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:16.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673130 2021-01-15T17:19:16.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:16.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:19:16.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673130 2021-01-15T17:19:16.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:16.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673130 2021-01-15T17:19:16.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673130' 2021-01-15T17:19:16.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:19:16.172 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673130 2021-01-15T17:19:16.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673131 -lt 42949673130 2021-01-15T17:19:16.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:16.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509609 2021-01-15T17:19:16.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:16.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:19:16.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509609 2021-01-15T17:19:16.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:16.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509609 2021-01-15T17:19:16.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509609' 2021-01-15T17:19:16.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:19:16.588 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509609 2021-01-15T17:19:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509609 -lt 64424509609 2021-01-15T17:19:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:17.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899346087 2021-01-15T17:19:17.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:17.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:19:17.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899346087 2021-01-15T17:19:17.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:17.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899346087 2021-01-15T17:19:17.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899346087' 2021-01-15T17:19:17.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:19:17.054 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899346087 2021-01-15T17:19:17.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899346088 -lt 85899346087 2021-01-15T17:19:17.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:17.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182565 2021-01-15T17:19:17.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:17.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:19:17.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182565 2021-01-15T17:19:17.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:17.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182565 2021-01-15T17:19:17.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182565' 2021-01-15T17:19:17.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:19:17.462 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182565 2021-01-15T17:19:17.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182566 -lt 107374182565 2021-01-15T17:19:17.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:17.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849019043 2021-01-15T17:19:17.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:19:17.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849019043 2021-01-15T17:19:17.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:17.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849019043 2021-01-15T17:19:17.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849019043' 2021-01-15T17:19:17.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:19:17.871 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849019043 2021-01-15T17:19:18.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849019044 -lt 128849019043 2021-01-15T17:19:18.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:18.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 6-150323855522 2021-01-15T17:19:18.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:18.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=6 2021-01-15T17:19:18.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 6-150323855522 2021-01-15T17:19:18.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:18.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855522 2021-01-15T17:19:18.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.6 seq 150323855522' 2021-01-15T17:19:18.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 6 2021-01-15T17:19:18.341 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.6 seq 150323855522 2021-01-15T17:19:18.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855523 -lt 150323855522 2021-01-15T17:19:18.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:18.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 7-171798692000 2021-01-15T17:19:18.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:18.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=7 2021-01-15T17:19:18.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 7-171798692000 2021-01-15T17:19:18.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:18.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798692000 2021-01-15T17:19:18.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.7 seq 171798692000' 2021-01-15T17:19:18.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 7 2021-01-15T17:19:18.742 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.7 seq 171798692000 2021-01-15T17:19:19.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798692001 -lt 171798692000 2021-01-15T17:19:19.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:19.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 8-193273528479 2021-01-15T17:19:19.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:19.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=8 2021-01-15T17:19:19.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 8-193273528479 2021-01-15T17:19:19.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:19.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528479 2021-01-15T17:19:19.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.8 seq 193273528479' 2021-01-15T17:19:19.149 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.8 seq 193273528479 2021-01-15T17:19:19.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 8 2021-01-15T17:19:19.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528480 -lt 193273528479 2021-01-15T17:19:19.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:19.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 9-214748364957 2021-01-15T17:19:19.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:19.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=9 2021-01-15T17:19:19.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 9-214748364957 2021-01-15T17:19:19.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=214748364957 2021-01-15T17:19:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.9 seq 214748364957' 2021-01-15T17:19:19.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 9 2021-01-15T17:19:19.595 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.9 seq 214748364957 2021-01-15T17:19:20.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 214748364957 -lt 214748364957 2021-01-15T17:19:20.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:20.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 10-236223201435 2021-01-15T17:19:20.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:20.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=10 2021-01-15T17:19:20.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 10-236223201435 2021-01-15T17:19:20.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:20.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=236223201435 2021-01-15T17:19:20.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.10 seq 236223201435' 2021-01-15T17:19:20.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 10 2021-01-15T17:19:20.017 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.10 seq 236223201435 2021-01-15T17:19:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 236223201436 -lt 236223201435 2021-01-15T17:19:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:20.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 11-257698037913 2021-01-15T17:19:20.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:20.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=11 2021-01-15T17:19:20.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 11-257698037913 2021-01-15T17:19:20.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:20.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=257698037913 2021-01-15T17:19:20.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.11 seq 257698037913' 2021-01-15T17:19:20.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 11 2021-01-15T17:19:20.444 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.11 seq 257698037913 2021-01-15T17:19:20.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 257698037914 -lt 257698037913 2021-01-15T17:19:20.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:20.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 12-279172874392 2021-01-15T17:19:20.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:20.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=12 2021-01-15T17:19:20.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 12-279172874392 2021-01-15T17:19:20.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:20.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=279172874392 2021-01-15T17:19:20.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.12 seq 279172874392' 2021-01-15T17:19:20.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 12 2021-01-15T17:19:20.843 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.12 seq 279172874392 2021-01-15T17:19:21.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 279172874393 -lt 279172874392 2021-01-15T17:19:21.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:21.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 13-300647710869 2021-01-15T17:19:21.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:21.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=13 2021-01-15T17:19:21.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 13-300647710869 2021-01-15T17:19:21.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:21.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=300647710869 2021-01-15T17:19:21.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.13 seq 300647710869' 2021-01-15T17:19:21.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 13 2021-01-15T17:19:21.348 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.13 seq 300647710869 2021-01-15T17:19:21.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 300647710870 -lt 300647710869 2021-01-15T17:19:21.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:21.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 14-322122547347 2021-01-15T17:19:21.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:21.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=14 2021-01-15T17:19:21.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 14-322122547347 2021-01-15T17:19:21.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:21.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=322122547347 2021-01-15T17:19:21.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.14 seq 322122547347' 2021-01-15T17:19:21.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 14 2021-01-15T17:19:21.758 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.14 seq 322122547347 2021-01-15T17:19:22.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 322122547348 -lt 322122547347 2021-01-15T17:19:22.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:22.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 15-343597383825 2021-01-15T17:19:22.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:22.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=15 2021-01-15T17:19:22.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 15-343597383825 2021-01-15T17:19:22.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=343597383825 2021-01-15T17:19:22.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.15 seq 343597383825' 2021-01-15T17:19:22.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 15 2021-01-15T17:19:22.195 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.15 seq 343597383825 2021-01-15T17:19:22.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 343597383826 -lt 343597383825 2021-01-15T17:19:22.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:22.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 16-365072220303 2021-01-15T17:19:22.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:22.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=16 2021-01-15T17:19:22.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 16-365072220303 2021-01-15T17:19:22.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:22.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=365072220303 2021-01-15T17:19:22.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.16 seq 365072220303' 2021-01-15T17:19:22.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 16 2021-01-15T17:19:22.597 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.16 seq 365072220303 2021-01-15T17:19:23.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 365072220304 -lt 365072220303 2021-01-15T17:19:23.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 17-386547056781 2021-01-15T17:19:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:23.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=17 2021-01-15T17:19:23.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 17-386547056781 2021-01-15T17:19:23.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:23.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=386547056781 2021-01-15T17:19:23.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.17 seq 386547056781' 2021-01-15T17:19:23.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 17 2021-01-15T17:19:23.019 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.17 seq 386547056781 2021-01-15T17:19:23.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 386547056782 -lt 386547056781 2021-01-15T17:19:23.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:23.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 18-408021893259 2021-01-15T17:19:23.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=18 2021-01-15T17:19:23.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 18-408021893259 2021-01-15T17:19:23.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=408021893259 2021-01-15T17:19:23.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.18 seq 408021893259' 2021-01-15T17:19:23.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 18 2021-01-15T17:19:23.447 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.18 seq 408021893259 2021-01-15T17:19:23.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 408021893260 -lt 408021893259 2021-01-15T17:19:23.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:19:23.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 19-429496729738 2021-01-15T17:19:23.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:19:23.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=19 2021-01-15T17:19:23.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 19-429496729738 2021-01-15T17:19:23.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:19:23.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=429496729738 2021-01-15T17:19:23.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.19 seq 429496729738' 2021-01-15T17:19:23.857 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.19 seq 429496729738 2021-01-15T17:19:23.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 19 2021-01-15T17:19:24.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 429496729739 -lt 429496729738 2021-01-15T17:19:24.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:19:24.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:19:24.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:19:24.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 8 == 0 2021-01-15T17:19:24.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:19:24.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:19:24.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:19:24.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:19:24.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:19:24.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:19:24.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:19:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:19:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:19:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:19:25.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:19:25.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:234: TEST_backfill_test_multi: ERRORS=0 2021-01-15T17:19:25.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: ceph pg dump pgs 2021-01-15T17:19:25.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: grep +backfill_toofull 2021-01-15T17:19:25.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: wc -l 2021-01-15T17:19:25.603 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:19:25.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:235: TEST_backfill_test_multi: full=4 2021-01-15T17:19:25.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:236: TEST_backfill_test_multi: '[' 4 -lt 1 ']' 2021-01-15T17:19:25.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:242: TEST_backfill_test_multi: expr 8 - 4 2021-01-15T17:19:25.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:242: TEST_backfill_test_multi: expected=4 2021-01-15T17:19:25.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: ceph pg dump pgs 2021-01-15T17:19:25.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: grep active+clean 2021-01-15T17:19:25.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: wc -l 2021-01-15T17:19:25.992 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:19:26.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:243: TEST_backfill_test_multi: '[' 4 '!=' 4 ']' 2021-01-15T17:19:26.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:249: TEST_backfill_test_multi: ceph pg dump pgs 2021-01-15T17:19:26.379 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:19:26.379 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:19:09.927200+0000 148'600 183:1860 [3,15] 3 [3,15] 3 0'0 2021-01-15T17:08:48.486677+0000 0'0 2021-01-15T17:08:48.486677+0000 0 2021-01-15T17:19:26.379 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:19:21.680720+0000 148'600 183:674 [5,1] 5 [5] 5 0'0 2021-01-15T17:08:45.307362+0000 0'0 2021-01-15T17:08:45.307362+0000 0 2021-01-15T17:19:26.380 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:19:16.753402+0000 148'600 183:688 [17,5] 17 [17] 17 0'0 2021-01-15T17:08:29.823784+0000 0'0 2021-01-15T17:08:29.823784+0000 0 2021-01-15T17:19:26.380 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:19:00.773631+0000 148'600 181:1888 [8,0] 8 [8,0] 8 0'0 2021-01-15T17:08:26.766428+0000 0'0 2021-01-15T17:08:26.766428+0000 0 2021-01-15T17:19:26.380 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:19:17.156116+0000 148'600 183:689 [1,7] 1 [1] 1 0'0 2021-01-15T17:08:33.006872+0000 0'0 2021-01-15T17:08:33.006872+0000 0 2021-01-15T17:19:26.380 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:19:05.701126+0000 148'600 181:1876 [4,10] 4 [4,10] 4 0'0 2021-01-15T17:08:36.021761+0000 0'0 2021-01-15T17:08:36.021761+0000 0 2021-01-15T17:19:26.380 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:19:07.165884+0000 148'600 182:1872 [12,19] 12 [12,19] 12 0'0 2021-01-15T17:08:39.553212+0000 0'0 2021-01-15T17:08:39.553212+0000 0 2021-01-15T17:19:26.381 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:19:19.772966+0000 148'600 183:677 [7,4] 7 [7] 7 0'0 2021-01-15T17:08:42.554595+0000 0'0 2021-01-15T17:08:42.554595+0000 0 2021-01-15T17:19:26.381 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:19:26.381 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:19:26.381 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:19:26.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:250: TEST_backfill_test_multi: ceph status 2021-01-15T17:19:26.897 INFO:tasks.workunit.client.0.smithi145.stdout: cluster: 2021-01-15T17:19:26.897 INFO:tasks.workunit.client.0.smithi145.stdout: id: 83bc673c-2d0b-48f5-bd28-415cc33826c5 2021-01-15T17:19:26.897 INFO:tasks.workunit.client.0.smithi145.stdout: health: HEALTH_WARN 2021-01-15T17:19:26.898 INFO:tasks.workunit.client.0.smithi145.stdout: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull 2021-01-15T17:19:26.898 INFO:tasks.workunit.client.0.smithi145.stdout: Degraded data redundancy: 2400/9600 objects degraded (25.000%), 4 pgs degraded, 4 pgs undersized 2021-01-15T17:19:26.898 INFO:tasks.workunit.client.0.smithi145.stdout: 8 pool(s) do not have an application enabled 2021-01-15T17:19:26.898 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:19:26.899 INFO:tasks.workunit.client.0.smithi145.stdout: services: 2021-01-15T17:19:26.899 INFO:tasks.workunit.client.0.smithi145.stdout: mon: 1 daemons, quorum a (age 14m) 2021-01-15T17:19:26.899 INFO:tasks.workunit.client.0.smithi145.stdout: mgr: x(active, since 14m) 2021-01-15T17:19:26.899 INFO:tasks.workunit.client.0.smithi145.stdout: osd: 20 osds: 20 up (since 11m), 20 in (since 11m); 4 remapped pgs 2021-01-15T17:19:26.899 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:19:26.900 INFO:tasks.workunit.client.0.smithi145.stdout: data: 2021-01-15T17:19:26.900 INFO:tasks.workunit.client.0.smithi145.stdout: pools: 8 pools, 8 pgs 2021-01-15T17:19:26.900 INFO:tasks.workunit.client.0.smithi145.stdout: objects: 4.80k objects, 19 MiB 2021-01-15T17:19:26.900 INFO:tasks.workunit.client.0.smithi145.stdout: usage: 28 MiB used, 42 MiB / 70 MiB avail 2021-01-15T17:19:26.900 INFO:tasks.workunit.client.0.smithi145.stdout: pgs: 2400/9600 objects degraded (25.000%) 2021-01-15T17:19:26.901 INFO:tasks.workunit.client.0.smithi145.stdout: 4 active+clean 2021-01-15T17:19:26.901 INFO:tasks.workunit.client.0.smithi145.stdout: 4 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:19:26.901 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:19:26.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:252: TEST_backfill_test_multi: ceph status --format=json-pretty 2021-01-15T17:19:27.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: jq .health.checks.PG_BACKFILL_FULL.severity td/osd-backfill-space/stat.json 2021-01-15T17:19:27.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: eval 'SEV="HEALTH_WARN"' 2021-01-15T17:19:27.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:254: TEST_backfill_test_multi: SEV=HEALTH_WARN 2021-01-15T17:19:27.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:255: TEST_backfill_test_multi: '[' HEALTH_WARN '!=' HEALTH_WARN ']' 2021-01-15T17:19:27.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: jq .health.checks.PG_BACKFILL_FULL.summary.message td/osd-backfill-space/stat.json 2021-01-15T17:19:27.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: eval 'MSG="Low' space hindering backfill '(add' storage if this 'doesn'\''t' resolve 'itself):' 4 pgs 'backfill_toofull"' 2021-01-15T17:19:27.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:259: TEST_backfill_test_multi: MSG='Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' 2021-01-15T17:19:27.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:260: TEST_backfill_test_multi: '[' 'Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' '!=' 'Low space hindering backfill (add storage if this doesn'\''t resolve itself): 4 pgs backfill_toofull' ']' 2021-01-15T17:19:27.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:264: TEST_backfill_test_multi: rm -f td/osd-backfill-space/stat.json 2021-01-15T17:19:27.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:266: TEST_backfill_test_multi: '[' 0 '!=' 0 ']' 2021-01-15T17:19:27.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: seq 1 8 2021-01-15T17:19:27.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:27.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test1 2021-01-15T17:19:27.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:19:27.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:19:27.982 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:19:27.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:27.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test2 2021-01-15T17:19:27.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T17:19:27.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T17:19:28.508 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T17:19:28.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:28.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test3 2021-01-15T17:19:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T17:19:28.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T17:19:29.146 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T17:19:29.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:29.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test4 2021-01-15T17:19:29.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2021-01-15T17:19:29.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-01-15T17:19:29.809 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' does not exist 2021-01-15T17:19:29.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:29.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test5 2021-01-15T17:19:29.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2021-01-15T17:19:29.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-01-15T17:19:30.545 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' does not exist 2021-01-15T17:19:30.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:30.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test6 2021-01-15T17:19:30.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2021-01-15T17:19:30.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-01-15T17:19:31.509 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' does not exist 2021-01-15T17:19:31.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:31.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test7 2021-01-15T17:19:31.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2021-01-15T17:19:31.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-01-15T17:19:32.427 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' does not exist 2021-01-15T17:19:32.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:271: TEST_backfill_test_multi: for i in $(seq 1 $pools) 2021-01-15T17:19:32.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:273: TEST_backfill_test_multi: delete_pool test8 2021-01-15T17:19:32.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2021-01-15T17:19:32.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-01-15T17:19:32.972 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' does not exist 2021-01-15T17:19:32.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:276: TEST_backfill_test_multi: kill_daemons td/osd-backfill-space 2021-01-15T17:19:32.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:19:32.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:19:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:19:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:19:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:19:33.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:19:33.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:277: TEST_backfill_test_multi: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.10.log td/osd-backfill-space/osd.11.log td/osd-backfill-space/osd.12.log td/osd-backfill-space/osd.13.log td/osd-backfill-space/osd.14.log td/osd-backfill-space/osd.15.log td/osd-backfill-space/osd.16.log td/osd-backfill-space/osd.17.log td/osd-backfill-space/osd.18.log td/osd-backfill-space/osd.19.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log td/osd-backfill-space/osd.3.log td/osd-backfill-space/osd.4.log td/osd-backfill-space/osd.5.log td/osd-backfill-space/osd.6.log td/osd-backfill-space/osd.7.log td/osd-backfill-space/osd.8.log td/osd-backfill-space/osd.9.log 2021-01-15T17:19:33.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:19:33.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:19:33.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:19:33.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:19:33.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:19:33.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:19:33.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:19:33.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:19:33.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:19:33.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:19:33.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:19:33.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:19:33.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:19:33.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:19:33.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:19:33.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:19:33.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:19:33.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:19:33.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:19:33.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:19:33.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:19:33.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:19:33.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:19:33.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:19:33.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:33.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:33.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:19:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:19:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:19:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:19:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:19:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:19:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:19:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:19:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:19:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:19:33.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:19:33.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:19:33.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:19:33.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:19:33.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:19:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:19:33.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:19:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:19:33.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:19:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:19:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:19:33.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:19:33.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:19:33.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:19:33.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:19:33.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:19:33.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:19:33.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:19:33.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:19:33.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:19:33.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:33.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:33.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:19:33.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:19:33.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:19:33.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:19:33.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:19:33.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:33.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:33.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:19:33.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:19:33.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:19:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:19:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:19:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_sametarget td/osd-backfill-space 2021-01-15T17:19:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:294: TEST_backfill_test_sametarget: local dir=td/osd-backfill-space 2021-01-15T17:19:33.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:295: TEST_backfill_test_sametarget: local pools=10 2021-01-15T17:19:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:296: TEST_backfill_test_sametarget: local OSDS=5 2021-01-15T17:19:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:298: TEST_backfill_test_sametarget: run_mon td/osd-backfill-space a 2021-01-15T17:19:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:19:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:19:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:19:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:19:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:19:33.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:19:33.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:19:33.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:33.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:33.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:33.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:33.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:33.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:33.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:19:34.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:19:34.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:19:34.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:19:34.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:19:34.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:19:34.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:19:34.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:19:34.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:19:34.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:19:34.144 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:19:34.144 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:34.144 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:34.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:19:34.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:19:34.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:19:34.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:19:34.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:19:34.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:19:34.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:19:34.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:19:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:19:34.244 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:19:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:19:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:19:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:19:34.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:19:34.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:19:34.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:299: TEST_backfill_test_sametarget: run_mgr td/osd-backfill-space x 2021-01-15T17:19:34.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:19:34.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:19:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:19:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:19:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:19:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:19:35.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:19:35.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:35.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:35.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:35.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:35.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:35.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:35.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:19:35.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:19:35.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:300: TEST_backfill_test_sametarget: export CEPH_ARGS 2021-01-15T17:19:35.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: expr 5 - 1 2021-01-15T17:19:35.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: seq 0 4 2021-01-15T17:19:35.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:19:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 0 2021-01-15T17:19:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:19:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:19:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:19:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:19:35.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:19:35.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:19:35.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:19:35.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:19:35.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:19:35.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:19:35.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:19:35.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:19:35.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:19:35.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:19:35.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:19:35.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:35.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:35.288 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:35.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:35.289 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:35.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:35.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:19:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:19:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:19:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:19:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:19:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:19:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:19:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:19:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:19:35.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:19:35.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:19:35.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f0665f91-2897-4106-9821-5c50cb7ae513 2021-01-15T17:19:35.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 f0665f91-2897-4106-9821-5c50cb7ae513' 2021-01-15T17:19:35.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:19:35.293 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 f0665f91-2897-4106-9821-5c50cb7ae513 2021-01-15T17:19:35.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCnzgFg8GD3ERAAuC3cme7JD8cbfctHAyPVRA== 2021-01-15T17:19:35.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCnzgFg8GD3ERAAuC3cme7JD8cbfctHAyPVRA=="}' 2021-01-15T17:19:35.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f0665f91-2897-4106-9821-5c50cb7ae513 -i td/osd-backfill-space/0/new.json 2021-01-15T17:19:35.658 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:19:35.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:19:35.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCnzgFg8GD3ERAAuC3cme7JD8cbfctHAyPVRA== --osd-uuid f0665f91-2897-4106-9821-5c50cb7ae513 2021-01-15T17:19:35.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.696+0000 7fc91b6c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:35.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.698+0000 7fc91b6c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:35.700 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.700+0000 7fc91b6c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:35.701 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.701+0000 7fc91b6c9080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:19:35.701 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.701+0000 7fc91b6c9080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:19:35.702 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.701+0000 7fc91b6c9080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:19:35.740 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:35.739+0000 7fc91b6c9080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:19:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:19:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:19:40.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:19:40.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:19:40.362 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:19:40.891 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:19:40.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:19:40.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:40.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:19:40.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:19:40.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:19:40.916 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:40.916+0000 7eff0e447080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:40.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:40.918+0000 7eff0e447080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:40.919 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:40.919+0000 7eff0e447080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:41.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:19:41.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:19:41.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:19:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:19:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:19:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:19:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:41.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:19:41.298 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:19:41.439 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:41.438+0000 7eff0e447080 -1 Falling back to public interface 2021-01-15T17:19:41.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:19:42.225 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:42.224+0000 7eff0e447080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:19:42.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:19:42.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:42.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:19:42.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:42.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:19:42.688 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:19:43.123 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/393844,v1:127.0.0.1:6803/393844] [v2:127.0.0.1:6804/393844,v1:127.0.0.1:6805/393844] exists,up f0665f91-2897-4106-9821-5c50cb7ae513 2021-01-15T17:19:43.123 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 824cdb7d-cbb5-41ca-bd98-ff20f38afd87 2021-01-15T17:19:43.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:19:43.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:19:43.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:19:43.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:19:43.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 1 2021-01-15T17:19:43.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:19:43.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:19:43.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:19:43.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:19:43.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:19:43.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:19:43.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:19:43.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:19:43.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:19:43.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:19:43.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:19:43.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:19:43.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:19:43.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:19:43.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:19:43.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:43.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:43.128 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:43.128 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:43.128 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:43.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:43.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:43.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:19:43.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:19:43.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:19:43.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:19:43.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:19:43.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:19:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:19:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:43.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:19:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:19:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:19:43.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:19:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=824cdb7d-cbb5-41ca-bd98-ff20f38afd87 2021-01-15T17:19:43.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 824cdb7d-cbb5-41ca-bd98-ff20f38afd87' 2021-01-15T17:19:43.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:19:43.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCvzgFgejdPCBAAAviZ1cO9k2XJ/F54FUmRrA== 2021-01-15T17:19:43.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCvzgFgejdPCBAAAviZ1cO9k2XJ/F54FUmRrA=="}' 2021-01-15T17:19:43.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 824cdb7d-cbb5-41ca-bd98-ff20f38afd87 -i td/osd-backfill-space/1/new.json 2021-01-15T17:19:43.630 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:19:43.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:19:43.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCvzgFgejdPCBAAAviZ1cO9k2XJ/F54FUmRrA== --osd-uuid 824cdb7d-cbb5-41ca-bd98-ff20f38afd87 2021-01-15T17:19:43.670 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.669+0000 7f67797f7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.687+0000 7f67797f7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:43.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.688+0000 7f67797f7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:43.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.689+0000 7f67797f7080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:19:43.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.689+0000 7f67797f7080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:19:43.690 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.689+0000 7f67797f7080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:19:43.746 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:43.745+0000 7f67797f7080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:19:48.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:19:48.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:19:48.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:19:48.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:19:48.551 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:19:49.079 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:19:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:19:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:19:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:19:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:19:49.104 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:49.103+0000 7f552a71f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:49.119 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:49.118+0000 7f552a71f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:49.121 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:49.120+0000 7f552a71f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:49.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:19:49.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:19:49.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:19:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:19:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:19:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:19:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:49.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:19:49.506 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:19:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:49.635+0000 7f552a71f080 -1 Falling back to public interface 2021-01-15T17:19:49.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:19:50.419 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:50.419+0000 7f552a71f080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:19:50.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:19:50.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:19:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:19:50.897 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:19:51.296 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/395108,v1:127.0.0.1:6811/395108] [v2:127.0.0.1:6812/395108,v1:127.0.0.1:6813/395108] exists,up 824cdb7d-cbb5-41ca-bd98-ff20f38afd87 2021-01-15T17:19:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:19:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:19:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:19:51.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:19:51.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 2 2021-01-15T17:19:51.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:19:51.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:19:51.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:19:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:19:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:19:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:19:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:19:51.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:19:51.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:19:51.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:19:51.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:19:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:19:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:19:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:19:51.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:19:51.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:51.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:51.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:19:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:19:51.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:19:51.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:19:51.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:19:51.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:19:51.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:19:51.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:51.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:19:51.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:19:51.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:19:51.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:19:51.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=365bb17c-ad54-409a-a282-f4b7fd95509e 2021-01-15T17:19:51.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 365bb17c-ad54-409a-a282-f4b7fd95509e' 2021-01-15T17:19:51.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:19:51.307 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 365bb17c-ad54-409a-a282-f4b7fd95509e 2021-01-15T17:19:51.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC3zgFgqLu7EhAACRjJm3XllsNkZxaiYht6lg== 2021-01-15T17:19:51.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC3zgFgqLu7EhAACRjJm3XllsNkZxaiYht6lg=="}' 2021-01-15T17:19:51.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 365bb17c-ad54-409a-a282-f4b7fd95509e -i td/osd-backfill-space/2/new.json 2021-01-15T17:19:51.906 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:19:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:19:51.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC3zgFgqLu7EhAACRjJm3XllsNkZxaiYht6lg== --osd-uuid 365bb17c-ad54-409a-a282-f4b7fd95509e 2021-01-15T17:19:51.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.945+0000 7f65ff271080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.946+0000 7f65ff271080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.947+0000 7f65ff271080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:51.948 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.948+0000 7f65ff271080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:19:51.949 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.948+0000 7f65ff271080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:19:51.949 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.948+0000 7f65ff271080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:19:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:51.993+0000 7f65ff271080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:19:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:19:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:19:56.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:19:56.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:19:56.827 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:19:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:19:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:19:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:19:57.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:19:57.359 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:19:57.383 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:57.382+0000 7f9256416080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:57.384 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:57.384+0000 7f9256416080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:57.386 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:57.386+0000 7f9256416080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:19:57.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:19:57.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:19:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:19:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:19:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:19:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:19:57.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:57.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:19:57.756 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:19:57.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:57.901+0000 7f9256416080 -1 Falling back to public interface 2021-01-15T17:19:58.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:19:58.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:19:58.677+0000 7f9256416080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:19:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:19:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:19:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:19:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:19:59.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:19:59.156 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:19:59.569 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/396372,v1:127.0.0.1:6819/396372] [v2:127.0.0.1:6820/396372,v1:127.0.0.1:6821/396372] exists,up 365bb17c-ad54-409a-a282-f4b7fd95509e 2021-01-15T17:19:59.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:19:59.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:19:59.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:19:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:19:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 3 2021-01-15T17:19:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:19:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:19:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:19:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:19:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2021-01-15T17:19:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:19:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:19:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:19:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:19:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2021-01-15T17:19:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2021-01-15T17:19:59.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:19:59.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:19:59.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:19:59.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:19:59.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:19:59.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:19:59.575 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:19:59.575 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:19:59.575 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:19:59.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:19:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:19:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:19:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:19:59.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:19:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:19:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:19:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:19:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:19:59.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:19:59.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:19:59.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2021-01-15T17:19:59.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:19:59.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a884379b-9439-49f7-815a-c4ab09195b8a 2021-01-15T17:19:59.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a884379b-9439-49f7-815a-c4ab09195b8a' 2021-01-15T17:19:59.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:19:59.580 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 a884379b-9439-49f7-815a-c4ab09195b8a 2021-01-15T17:19:59.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC/zgFgnOwMIxAAKRHihI20sf95Iu/fQyHeWw== 2021-01-15T17:19:59.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC/zgFgnOwMIxAAKRHihI20sf95Iu/fQyHeWw=="}' 2021-01-15T17:19:59.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a884379b-9439-49f7-815a-c4ab09195b8a -i td/osd-backfill-space/3/new.json 2021-01-15T17:20:00.102 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:20:00.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2021-01-15T17:20:00.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC/zgFgnOwMIxAAKRHihI20sf95Iu/fQyHeWw== --osd-uuid a884379b-9439-49f7-815a-c4ab09195b8a 2021-01-15T17:20:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.141+0000 7f5f8d815080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:00.143 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.143+0000 7f5f8d815080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.144+0000 7f5f8d815080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:00.145 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.145+0000 7f5f8d815080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:20:00.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.145+0000 7f5f8d815080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:20:00.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.145+0000 7f5f8d815080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:20:00.194 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:00.194+0000 7f5f8d815080 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2021-01-15T17:20:04.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2021-01-15T17:20:04.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:20:04.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:20:04.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:20:04.964 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:20:05.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:20:05.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:20:05.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:20:05.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:20:05.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:20:05.506 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:20:05.531 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:05.530+0000 7ff2fc6c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:05.532 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:05.532+0000 7ff2fc6c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:05.534 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:05.533+0000 7ff2fc6c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:20:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:20:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:20:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:20:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:20:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:20:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:20:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:20:05.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:20:05.907 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:20:06.054 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:06.053+0000 7ff2fc6c3080 -1 Falling back to public interface 2021-01-15T17:20:06.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:20:06.861 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:06.860+0000 7ff2fc6c3080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:20:07.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:20:07.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:20:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:20:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:20:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:20:07.327 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:20:07.763 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/397636,v1:127.0.0.1:6827/397636] [v2:127.0.0.1:6828/397636,v1:127.0.0.1:6829/397636] exists,up a884379b-9439-49f7-815a-c4ab09195b8a 2021-01-15T17:20:07.763 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 345b56b7-0600-4751-9897-5141c3581b2b 2021-01-15T17:20:07.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:20:07.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:20:07.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:20:07.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:302: TEST_backfill_test_sametarget: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:20:07.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:304: TEST_backfill_test_sametarget: run_osd td/osd-backfill-space 4 2021-01-15T17:20:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:20:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:20:07.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:20:07.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:20:07.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2021-01-15T17:20:07.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:20:07.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:20:07.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:20:07.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:20:07.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2021-01-15T17:20:07.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2021-01-15T17:20:07.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:20:07.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:20:07.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:20:07.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:20:07.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:20:07.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:20:07.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:20:07.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:20:07.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:20:07.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:20:07.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:20:07.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:20:07.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:20:07.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:20:07.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:20:07.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:20:07.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:20:07.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:20:07.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:20:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:20:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:20:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2021-01-15T17:20:07.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:20:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=345b56b7-0600-4751-9897-5141c3581b2b 2021-01-15T17:20:07.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 345b56b7-0600-4751-9897-5141c3581b2b' 2021-01-15T17:20:07.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:20:07.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDHzgFgoMlGLhAABJvybQJJB6rhkXXbgct5QA== 2021-01-15T17:20:07.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDHzgFgoMlGLhAABJvybQJJB6rhkXXbgct5QA=="}' 2021-01-15T17:20:07.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 345b56b7-0600-4751-9897-5141c3581b2b -i td/osd-backfill-space/4/new.json 2021-01-15T17:20:08.268 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:20:08.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2021-01-15T17:20:08.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDHzgFgoMlGLhAABJvybQJJB6rhkXXbgct5QA== --osd-uuid 345b56b7-0600-4751-9897-5141c3581b2b 2021-01-15T17:20:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.306+0000 7f9ff2938080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.308+0000 7f9ff2938080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.310+0000 7f9ff2938080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:08.311 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.311+0000 7f9ff2938080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:20:08.311 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.311+0000 7f9ff2938080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:20:08.312 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.311+0000 7f9ff2938080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:20:08.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:08.358+0000 7f9ff2938080 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2021-01-15T17:20:13.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2021-01-15T17:20:13.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:20:13.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:20:13.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:20:13.081 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:20:13.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:20:13.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:20:13.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:20:13.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:20:13.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:20:13.614 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:20:13.638 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:13.637+0000 7fdf75d42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:13.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:13.676+0000 7fdf75d42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:13.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:13.678+0000 7fdf75d42080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:20:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:20:14.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:20:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:20:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:20:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:20:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:20:14.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:20:14.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:20:14.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:20:14.012 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:20:14.193 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:14.193+0000 7fdf75d42080 -1 Falling back to public interface 2021-01-15T17:20:14.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:20:15.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:20:15.004+0000 7fdf75d42080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:20:15.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:20:15.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:20:15.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:20:15.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:20:15.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:20:15.426 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:20:15.844 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/398900,v1:127.0.0.1:6835/398900] [v2:127.0.0.1:6836/398900,v1:127.0.0.1:6837/398900] exists,up 345b56b7-0600-4751-9897-5141c3581b2b 2021-01-15T17:20:15.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:20:15.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:20:15.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:20:15.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:307: TEST_backfill_test_sametarget: ceph osd set-backfillfull-ratio .85 2021-01-15T17:20:16.572 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:20:16.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: seq 1 10 2021-01-15T17:20:16.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:16.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test1 1 1 2021-01-15T17:20:16.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T17:20:17.181 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T17:20:17.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:18.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test1 size 2 2021-01-15T17:20:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:20:19.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:19.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test2 1 1 2021-01-15T17:20:19.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T17:20:20.035 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T17:20:20.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:21.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test2 size 2 2021-01-15T17:20:21.887 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T17:20:21.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:21.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test3 1 1 2021-01-15T17:20:21.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T17:20:22.635 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T17:20:22.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:23.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test3 size 2 2021-01-15T17:20:24.687 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T17:20:24.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:24.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test4 1 1 2021-01-15T17:20:24.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2021-01-15T17:20:25.509 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' already exists 2021-01-15T17:20:25.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:26.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test4 size 2 2021-01-15T17:20:27.308 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2021-01-15T17:20:27.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:27.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test5 1 1 2021-01-15T17:20:27.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2021-01-15T17:20:27.956 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' already exists 2021-01-15T17:20:27.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:28.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test5 size 2 2021-01-15T17:20:29.890 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 2 2021-01-15T17:20:29.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:29.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test6 1 1 2021-01-15T17:20:29.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2021-01-15T17:20:30.516 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' already exists 2021-01-15T17:20:30.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:31.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test6 size 2 2021-01-15T17:20:32.450 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 2 2021-01-15T17:20:32.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:32.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test7 1 1 2021-01-15T17:20:32.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2021-01-15T17:20:33.110 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' already exists 2021-01-15T17:20:33.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:34.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test7 size 2 2021-01-15T17:20:35.189 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T17:20:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test8 1 1 2021-01-15T17:20:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2021-01-15T17:20:35.759 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' already exists 2021-01-15T17:20:35.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:36.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test8 size 2 2021-01-15T17:20:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 2 2021-01-15T17:20:37.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:37.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test9 1 1 2021-01-15T17:20:37.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2021-01-15T17:20:38.180 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' already exists 2021-01-15T17:20:38.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:39.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test9 size 2 2021-01-15T17:20:40.333 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 9 size to 2 2021-01-15T17:20:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:309: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:311: TEST_backfill_test_sametarget: create_pool test10 1 1 2021-01-15T17:20:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2021-01-15T17:20:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' already exists 2021-01-15T17:20:41.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:20:42.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:312: TEST_backfill_test_sametarget: ceph osd pool set test10 size 2 2021-01-15T17:20:42.907 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 10 size to 2 2021-01-15T17:20:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:314: TEST_backfill_test_sametarget: sleep 5 2021-01-15T17:20:47.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:316: TEST_backfill_test_sametarget: wait_for_clean 2021-01-15T17:20:47.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:20:47.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:20:47.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:20:47.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:20:47.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:20:47.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:20:47.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:20:47.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:20:47.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:20:47.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:20:48.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:20:48.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:20:48.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:20:48.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:20:48.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:20:48.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:20:48.425 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:20:48.426 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:20:48.426 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:20:48.426 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:20:48.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:20:48.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:20:48.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:20:48.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836496 2021-01-15T17:20:48.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836496 2021-01-15T17:20:48.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496' 2021-01-15T17:20:48.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:20:48.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:20:48.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T17:20:48.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T17:20:48.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974' 2021-01-15T17:20:48.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:20:48.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:20:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T17:20:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T17:20:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452' 2021-01-15T17:20:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:20:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:20:48.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345932 2021-01-15T17:20:48.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345932 2021-01-15T17:20:48.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452 3-85899345932' 2021-01-15T17:20:48.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:20:48.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:20:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182409 2021-01-15T17:20:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182409 2021-01-15T17:20:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836496 1-42949672974 2-64424509452 3-85899345932 4-107374182409' 2021-01-15T17:20:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:20:49.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2021-01-15T17:20:49.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:20:49.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:20:49.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836496 2021-01-15T17:20:49.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:20:49.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836496 2021-01-15T17:20:49.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2021-01-15T17:20:49.140 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836496 2021-01-15T17:20:49.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:20:49.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836496 2021-01-15T17:20:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:20:49.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T17:20:49.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:20:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:20:49.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T17:20:49.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:20:49.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T17:20:49.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T17:20:49.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:20:49.538 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T17:20:49.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T17:20:49.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:20:49.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T17:20:49.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:20:49.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:20:49.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T17:20:49.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:20:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T17:20:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T17:20:49.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:20:49.936 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T17:20:50.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T17:20:50.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:20:50.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345932 2021-01-15T17:20:50.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:20:50.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:20:50.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345932 2021-01-15T17:20:50.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:20:50.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345932 2021-01-15T17:20:50.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2021-01-15T17:20:50.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:20:50.336 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345932 2021-01-15T17:20:50.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345932 -lt 85899345932 2021-01-15T17:20:50.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:20:50.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182409 2021-01-15T17:20:50.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:20:50.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:20:50.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182409 2021-01-15T17:20:50.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:20:50.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182409 2021-01-15T17:20:50.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2021-01-15T17:20:50.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:20:50.745 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182409 2021-01-15T17:20:51.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182409 -lt 107374182409 2021-01-15T17:20:51.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:20:51.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:20:51.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:20:51.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2021-01-15T17:20:51.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:20:52.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2021-01-15T17:20:52.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:20:52.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:20:52.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:20:52.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2021-01-15T17:20:52.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:20:52.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:20:52.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:318: TEST_backfill_test_sametarget: ceph pg dump pgs 2021-01-15T17:20:52.950 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:20:52.951 INFO:tasks.workunit.client.0.smithi145.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:43.698692+0000 0'0 81:14 [2,1] 2 [2,1] 2 0'0 2021-01-15T17:20:40.812305+0000 0'0 2021-01-15T17:20:40.812305+0000 0 2021-01-15T17:20:52.951 INFO:tasks.workunit.client.0.smithi145.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:40.720735+0000 0'0 81:25 [1,2] 1 [1,2] 1 0'0 2021-01-15T17:20:37.964909+0000 0'0 2021-01-15T17:20:37.964909+0000 0 2021-01-15T17:20:52.951 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:38.259010+0000 0'0 77:24 [3,4] 3 [3,4] 3 0'0 2021-01-15T17:20:35.545007+0000 0'0 2021-01-15T17:20:35.545007+0000 0 2021-01-15T17:20:52.951 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:36.002755+0000 0'0 81:33 [1,4] 1 [1,4] 1 0'0 2021-01-15T17:20:32.885896+0000 0'0 2021-01-15T17:20:32.885896+0000 0 2021-01-15T17:20:52.952 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:23.074931+0000 0'0 77:51 [3,1] 3 [3,1] 3 0'0 2021-01-15T17:20:19.674522+0000 0'0 2021-01-15T17:20:19.674522+0000 0 2021-01-15T17:20:52.952 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:19.674867+0000 0'0 81:59 [1,0] 1 [1,0] 1 0'0 2021-01-15T17:20:16.928220+0000 0'0 2021-01-15T17:20:16.928220+0000 0 2021-01-15T17:20:52.952 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:24.885689+0000 0'0 81:50 [1,2] 1 [1,2] 1 0'0 2021-01-15T17:20:22.320659+0000 0'0 2021-01-15T17:20:22.320659+0000 0 2021-01-15T17:20:52.952 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:27.532068+0000 0'0 81:42 [4,0] 4 [4,0] 4 0'0 2021-01-15T17:20:25.061183+0000 0'0 2021-01-15T17:20:25.061183+0000 0 2021-01-15T17:20:52.953 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:30.972965+0000 0'0 77:37 [3,4] 3 [3,4] 3 0'0 2021-01-15T17:20:27.695229+0000 0'0 2021-01-15T17:20:27.695229+0000 0 2021-01-15T17:20:52.953 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:20:33.235175+0000 0'0 81:33 [4,2] 4 [4,2] 4 0'0 2021-01-15T17:20:30.269224+0000 0'0 2021-01-15T17:20:30.269224+0000 0 2021-01-15T17:20:52.953 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:20:52.953 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:20:52.954 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:20:52.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:322: TEST_backfill_test_sametarget: local PG1 2021-01-15T17:20:52.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:323: TEST_backfill_test_sametarget: local POOLNUM1 2021-01-15T17:20:52.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:324: TEST_backfill_test_sametarget: local pool1 2021-01-15T17:20:52.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:325: TEST_backfill_test_sametarget: local chk_osd1 2021-01-15T17:20:52.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:326: TEST_backfill_test_sametarget: local chk_osd2 2021-01-15T17:20:52.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:328: TEST_backfill_test_sametarget: local PG2 2021-01-15T17:20:52.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:329: TEST_backfill_test_sametarget: local POOLNUM2 2021-01-15T17:20:52.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:330: TEST_backfill_test_sametarget: local pool2 2021-01-15T17:20:52.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: seq 1 10 2021-01-15T17:20:52.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:52.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 1.0 --format=json 2021-01-15T17:20:52.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2021-01-15T17:20:53.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2021-01-15T17:20:53.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=1 2021-01-15T17:20:53.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2021-01-15T17:20:53.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=0 2021-01-15T17:20:53.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 1 = 1 ']' 2021-01-15T17:20:53.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:338: TEST_backfill_test_sametarget: PG1=1.0 2021-01-15T17:20:53.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:339: TEST_backfill_test_sametarget: POOLNUM1=1 2021-01-15T17:20:53.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:340: TEST_backfill_test_sametarget: pool1=test1 2021-01-15T17:20:53.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:341: TEST_backfill_test_sametarget: chk_osd1=1 2021-01-15T17:20:53.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:342: TEST_backfill_test_sametarget: chk_osd2=0 2021-01-15T17:20:53.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:53.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 2.0 --format=json 2021-01-15T17:20:53.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2021-01-15T17:20:53.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2021-01-15T17:20:53.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=3 2021-01-15T17:20:53.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2021-01-15T17:20:53.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=1 2021-01-15T17:20:53.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 2 = 1 ']' 2021-01-15T17:20:53.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 3 -a 0 = 1 ']' 2021-01-15T17:20:53.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:53.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 3.0 --format=json 2021-01-15T17:20:53.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2021-01-15T17:20:54.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2021-01-15T17:20:54.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=1 2021-01-15T17:20:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2021-01-15T17:20:54.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=2 2021-01-15T17:20:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 3 = 1 ']' 2021-01-15T17:20:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 1 -a 0 = 2 ']' 2021-01-15T17:20:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:331: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: ceph pg map 4.0 --format=json 2021-01-15T17:20:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:333: TEST_backfill_test_sametarget: jq '.acting[]' 2021-01-15T17:20:54.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: head -1 td/osd-backfill-space/acting 2021-01-15T17:20:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:334: TEST_backfill_test_sametarget: local test_osd1=4 2021-01-15T17:20:54.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: tail -1 td/osd-backfill-space/acting 2021-01-15T17:20:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:335: TEST_backfill_test_sametarget: local test_osd2=0 2021-01-15T17:20:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:336: TEST_backfill_test_sametarget: '[' 4 = 1 ']' 2021-01-15T17:20:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:343: TEST_backfill_test_sametarget: '[' 1 '!=' 4 -a 0 = 0 ']' 2021-01-15T17:20:54.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:345: TEST_backfill_test_sametarget: PG2=4.0 2021-01-15T17:20:54.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:346: TEST_backfill_test_sametarget: POOLNUM2=4 2021-01-15T17:20:54.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:347: TEST_backfill_test_sametarget: pool2=test4 2021-01-15T17:20:54.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:348: TEST_backfill_test_sametarget: break 2021-01-15T17:20:54.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:351: TEST_backfill_test_sametarget: rm -f td/osd-backfill-space/acting 2021-01-15T17:20:54.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:353: TEST_backfill_test_sametarget: '[' test4 = '' ']' 2021-01-15T17:20:54.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: seq 1 10 2021-01-15T17:20:54.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:54.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 1 '!=' 1 -a 1 '!=' 4 ']' 2021-01-15T17:20:54.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:54.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 2 '!=' 1 -a 2 '!=' 4 ']' 2021-01-15T17:20:54.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test2 2021-01-15T17:20:54.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T17:20:54.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T17:20:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T17:20:55.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:55.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 3 '!=' 1 -a 3 '!=' 4 ']' 2021-01-15T17:20:55.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test3 2021-01-15T17:20:55.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T17:20:55.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T17:20:55.702 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T17:20:55.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:55.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 4 '!=' 1 -a 4 '!=' 4 ']' 2021-01-15T17:20:55.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:55.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 5 '!=' 1 -a 5 '!=' 4 ']' 2021-01-15T17:20:55.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test5 2021-01-15T17:20:55.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2021-01-15T17:20:55.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-01-15T17:20:56.339 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' does not exist 2021-01-15T17:20:56.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:56.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 6 '!=' 1 -a 6 '!=' 4 ']' 2021-01-15T17:20:56.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test6 2021-01-15T17:20:56.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2021-01-15T17:20:56.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-01-15T17:20:56.857 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' does not exist 2021-01-15T17:20:56.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:56.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 7 '!=' 1 -a 7 '!=' 4 ']' 2021-01-15T17:20:56.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test7 2021-01-15T17:20:56.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2021-01-15T17:20:56.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-01-15T17:20:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' does not exist 2021-01-15T17:20:57.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:57.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 8 '!=' 1 -a 8 '!=' 4 ']' 2021-01-15T17:20:57.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test8 2021-01-15T17:20:57.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2021-01-15T17:20:57.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-01-15T17:20:58.200 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' does not exist 2021-01-15T17:20:58.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:58.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 9 '!=' 1 -a 9 '!=' 4 ']' 2021-01-15T17:20:58.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test9 2021-01-15T17:20:58.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2021-01-15T17:20:58.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-01-15T17:20:58.758 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' does not exist 2021-01-15T17:20:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:359: TEST_backfill_test_sametarget: for p in $(seq 1 $pools) 2021-01-15T17:20:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:361: TEST_backfill_test_sametarget: '[' 10 '!=' 1 -a 10 '!=' 4 ']' 2021-01-15T17:20:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:363: TEST_backfill_test_sametarget: delete_pool test10 2021-01-15T17:20:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2021-01-15T17:20:58.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-01-15T17:20:59.327 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' does not exist 2021-01-15T17:20:59.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:367: TEST_backfill_test_sametarget: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T17:21:00.118 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:21:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:368: TEST_backfill_test_sametarget: ceph osd pool set test4 size 1 --yes-i-really-mean-it 2021-01-15T17:21:01.133 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 1 2021-01-15T17:21:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:370: TEST_backfill_test_sametarget: wait_for_clean 2021-01-15T17:21:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:21:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:21:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:21:01.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:21:01.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:21:01.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:21:01.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:21:01.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:21:01.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:21:01.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:21:01.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:21:01.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:21:01.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:21:01.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:21:01.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:21:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:21:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:21:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:21:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:21:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:21:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:21:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:21:01.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:21:01.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836501 2021-01-15T17:21:01.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836501 2021-01-15T17:21:01.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501' 2021-01-15T17:21:01.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:21:01.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:21:01.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T17:21:01.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T17:21:01.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2021-01-15T17:21:01.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:21:01.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:21:02.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T17:21:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T17:21:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509457' 2021-01-15T17:21:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:21:02.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:21:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2021-01-15T17:21:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2021-01-15T17:21:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509457 3-85899345936' 2021-01-15T17:21:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:21:02.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:21:02.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T17:21:02.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T17:21:02.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509457 3-85899345936 4-107374182414' 2021-01-15T17:21:02.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:21:02.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836501 2021-01-15T17:21:02.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:21:02.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:21:02.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2021-01-15T17:21:02.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:21:02.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836501 2021-01-15T17:21:02.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2021-01-15T17:21:02.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:21:02.335 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836501 2021-01-15T17:21:02.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836501 2021-01-15T17:21:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:21:03.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:21:03.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:21:04.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836501 -lt 21474836501 2021-01-15T17:21:04.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:21:04.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T17:21:04.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:21:04.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:21:04.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T17:21:04.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:21:04.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T17:21:04.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T17:21:04.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:21:04.133 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T17:21:04.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672979 2021-01-15T17:21:04.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:21:04.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T17:21:04.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:21:04.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:21:04.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T17:21:04.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:21:04.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T17:21:04.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T17:21:04.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:21:04.541 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T17:21:04.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2021-01-15T17:21:04.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:21:04.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2021-01-15T17:21:04.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:21:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:21:04.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2021-01-15T17:21:04.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:21:04.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2021-01-15T17:21:04.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-01-15T17:21:04.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:21:04.942 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345936 2021-01-15T17:21:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345937 -lt 85899345936 2021-01-15T17:21:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:21:05.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T17:21:05.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:21:05.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:21:05.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T17:21:05.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:21:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T17:21:05.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T17:21:05.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:21:05.340 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T17:21:05.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2021-01-15T17:21:05.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:21:05.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:21:05.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:21:06.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T17:21:06.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:21:06.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:21:06.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:21:06.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:21:06.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:21:06.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:21:06.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:21:06.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T17:21:06.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:21:06.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:21:06.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:21:07.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T17:21:07.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:21:07.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:21:07.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:372: TEST_backfill_test_sametarget: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2021-01-15T17:21:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T17:21:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T17:21:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.3521e-05 s, 64.5 MB/s 2021-01-15T17:21:07.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: seq 1 600 2021-01-15T17:21:07.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:07.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:21:07.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:21:07.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:07.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:21:07.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:21:07.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:07.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:21:07.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:21:07.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:07.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:21:07.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:21:07.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:07.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:21:07.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:21:08.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:21:08.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:21:08.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:21:08.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:21:08.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:21:08.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:21:08.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:21:08.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:21:08.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:21:08.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:21:08.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:08.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:21:08.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:21:09.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:21:09.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:21:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:21:09.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:21:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:21:09.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:21:09.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:21:09.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:21:09.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:21:09.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:21:09.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:21:10.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:21:10.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:21:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:21:10.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:21:10.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:21:10.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:21:10.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:21:10.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:21:10.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:21:10.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:21:10.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:21:10.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:21:10.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:21:10.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:10.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:21:11.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:21:11.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:21:11.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:21:11.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:21:11.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:21:11.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:21:11.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:21:11.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:21:11.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:21:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:21:11.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:21:11.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:11.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:21:12.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:21:12.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:21:12.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:21:12.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:21:12.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:21:12.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:21:12.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:21:12.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:21:12.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:21:12.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:21:12.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:21:12.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:12.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:21:12.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:21:13.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:21:13.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:21:13.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:21:13.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:21:13.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:21:13.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:21:13.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:21:13.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:21:13.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:21:13.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:21:13.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:13.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:21:13.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:21:14.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:21:14.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:21:14.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:21:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:21:14.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:21:14.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:21:14.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:21:14.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:21:14.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:21:14.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:21:14.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:14.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:21:14.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:21:15.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:21:15.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:21:15.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:21:15.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:21:15.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:21:15.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:21:15.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:21:15.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:21:15.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:21:15.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:21:15.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:15.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:21:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:21:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:21:16.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:21:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:21:16.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:21:16.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:21:16.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:21:16.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:21:16.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:21:16.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:21:16.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:21:16.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:21:16.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:21:16.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:16.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:21:17.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:21:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:21:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:21:17.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:21:17.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:21:17.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:21:17.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:21:17.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:21:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:21:17.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:21:17.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:21:17.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:17.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:21:18.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:21:18.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:21:18.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:21:18.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:21:18.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:21:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:21:18.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:21:18.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:21:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:21:18.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:21:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:21:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:21:19.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:21:19.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:21:19.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:21:19.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:21:19.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:21:19.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:21:19.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:21:19.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:21:19.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:21:19.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:21:19.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:21:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:21:19.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:19.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:21:20.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:21:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:21:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:21:20.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:21:20.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:21:20.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:21:20.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:21:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:21:20.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:21:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:21:20.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:21:20.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:20.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:21:20.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:21:21.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:21:21.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:21:21.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:21:21.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:21:21.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:21:21.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:21:21.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:21:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:21:21.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:21:21.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:21:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:21:21.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:21:22.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:21:22.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:21:22.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:21:22.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:21:22.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:21:22.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:21:22.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:21:22.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:21:22.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:21:22.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:21:22.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:21:22.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:21:22.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:22.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:21:22.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:21:23.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:21:23.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:21:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:21:23.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:21:23.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:21:23.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:21:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:21:23.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:21:23.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:21:23.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:21:23.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:23.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:21:23.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:21:24.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:21:24.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:21:24.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:21:24.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:21:24.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:21:24.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:21:24.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:21:24.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:21:24.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:21:24.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:21:24.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:24.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:21:24.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:21:25.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:21:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:21:25.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:21:25.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:21:25.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:21:25.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:21:25.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:21:25.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:21:25.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:21:25.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:21:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:21:25.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:21:25.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:25.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:21:26.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:21:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:21:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:21:26.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:21:26.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:21:26.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:21:26.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:21:26.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:21:26.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:21:26.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:21:26.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:21:26.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:26.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:21:26.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:21:27.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:21:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:21:27.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:21:27.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:21:27.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:21:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:21:27.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:21:27.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:21:27.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:21:27.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:21:27.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:27.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:21:27.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:21:28.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:21:28.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:21:28.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:21:28.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:21:28.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:21:28.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:21:28.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:21:28.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:21:28.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:21:28.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:21:28.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:21:28.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:21:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:21:29.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:21:29.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:21:29.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:21:29.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:21:29.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:21:29.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:21:29.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:21:29.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:21:29.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:21:29.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:21:29.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:29.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:21:29.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:21:30.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:21:30.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:21:30.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:21:30.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:21:30.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:21:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:21:30.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:21:30.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:21:30.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:21:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:21:30.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:21:30.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:21:30.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:30.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:21:31.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:21:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:21:31.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:21:31.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:21:31.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:21:31.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:21:31.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:21:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:21:31.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:21:31.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:21:31.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:21:31.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:31.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:21:32.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:21:32.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:21:32.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:21:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:21:32.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:21:32.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:21:32.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:21:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:21:32.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:21:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:21:32.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:21:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:21:32.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:21:33.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:21:33.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:21:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:21:33.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:21:33.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:21:33.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:21:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:21:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:21:33.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:21:33.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:21:33.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:33.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:21:33.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:21:34.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:21:34.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:21:34.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:21:34.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:21:34.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:21:34.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:21:34.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:21:34.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:21:34.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:21:34.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:21:34.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:21:34.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:21:34.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:34.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:21:35.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:21:35.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:21:35.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:21:35.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:21:35.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:21:35.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:21:35.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:21:35.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:21:35.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:21:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:21:35.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:21:35.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:35.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:21:36.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:21:36.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:21:36.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:21:36.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:21:36.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:21:36.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:21:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:21:36.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:21:36.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:21:36.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:21:36.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:21:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:21:36.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:21:36.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:36.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:21:37.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:21:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:21:37.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:21:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:21:37.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:21:37.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:21:37.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:21:37.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:21:37.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:21:37.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:21:37.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:21:37.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:37.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:21:38.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:21:38.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:21:38.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:21:38.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:21:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:21:38.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:21:38.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:21:38.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:21:38.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:21:38.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:21:38.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:21:38.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:38.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:21:38.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:21:39.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:39.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:21:39.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:21:39.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:39.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:21:39.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:21:39.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:39.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:21:40.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:21:40.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:21:40.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:21:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:21:40.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:21:40.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:21:40.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:21:40.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:21:40.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:21:40.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:21:40.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:21:40.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:40.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:21:40.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:21:41.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:21:41.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:21:41.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:21:41.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:21:41.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:21:41.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:21:41.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:21:41.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:21:41.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:21:41.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:21:41.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:41.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:21:41.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:21:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:21:42.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:21:42.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:21:42.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:21:42.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:21:42.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:21:42.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:21:42.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:21:42.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:21:42.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:21:42.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:21:42.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:21:42.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:42.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:21:43.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:21:43.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:21:43.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:21:43.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:21:43.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:21:43.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:21:43.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:21:43.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:21:43.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:21:43.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:21:43.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:21:43.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:43.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:21:43.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:21:44.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:21:44.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:21:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:21:44.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:21:44.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:21:44.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:21:44.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:21:44.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:21:44.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:21:44.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:21:44.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:44.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:21:44.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:21:45.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:21:45.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:21:45.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:21:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:21:45.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:21:45.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:21:45.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:21:45.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:21:45.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:21:45.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:21:45.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:45.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:21:45.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:21:46.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:21:46.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:21:46.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:21:46.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:21:46.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:21:46.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:21:46.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:21:46.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:21:46.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:21:46.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:21:46.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:21:46.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:21:46.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:46.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:21:47.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:21:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:21:47.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:21:47.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:21:47.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:21:47.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:21:47.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:21:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:21:47.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:21:47.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:21:47.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:21:47.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:47.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:21:48.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:21:48.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:21:48.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:21:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:21:48.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:21:48.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:21:48.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:21:48.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:21:48.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:21:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:21:48.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:21:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:21:48.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:21:49.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:21:49.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:21:49.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:21:49.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:21:49.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:21:49.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:21:49.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:21:49.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:21:49.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:21:49.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:21:49.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:49.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:21:49.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:21:50.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:21:50.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:21:50.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:21:50.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:21:50.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:21:50.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:21:50.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:21:50.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:21:50.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:21:50.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:21:50.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:21:50.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:21:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:21:51.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:21:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:21:51.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:21:51.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:21:51.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:21:51.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:21:51.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:21:51.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:21:51.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:21:51.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:21:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:21:51.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:51.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:21:52.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:21:52.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:21:52.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:21:52.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:21:52.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:21:52.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:21:52.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:21:52.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:21:52.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:21:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:21:52.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:21:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:21:52.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:21:53.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:21:53.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:21:53.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:21:53.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:21:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:21:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:21:53.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:21:53.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:21:53.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:21:53.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:21:53.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:53.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:21:53.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:21:54.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:21:54.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:21:54.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:21:54.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:21:54.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:21:54.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:21:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:21:54.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:21:54.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:21:54.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:21:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:21:54.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:21:54.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:54.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:21:55.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:21:55.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:21:55.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:21:55.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:21:55.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:21:55.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:21:55.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:21:55.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:21:55.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:21:55.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:21:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:21:55.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:55.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:21:56.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:21:56.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:21:56.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:21:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:21:56.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:21:56.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:21:56.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:21:56.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:21:56.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:21:56.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:21:56.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:21:56.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:56.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:21:56.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:21:57.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:21:57.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:21:57.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:21:57.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:21:57.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:21:57.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:21:57.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:21:57.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:21:57.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:21:57.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:21:57.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:57.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:21:57.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:21:58.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:21:58.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:21:58.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:21:58.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:21:58.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:21:58.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:21:58.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:21:58.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:21:58.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:21:58.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:21:58.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:21:58.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:21:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:58.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:21:59.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:21:59.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:21:59.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:21:59.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:21:59.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:21:59.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:21:59.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:21:59.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:21:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:21:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:21:59.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:21:59.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:21:59.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:22:00.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:22:00.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:22:00.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:22:00.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:22:00.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:22:00.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:22:00.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:22:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:22:00.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:22:00.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:22:00.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:22:00.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:00.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:22:00.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:22:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:22:01.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:22:01.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:22:01.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:22:01.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:22:01.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:22:01.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:22:01.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:22:01.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:22:01.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:22:01.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:22:01.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:22:01.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:01.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:22:02.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:22:02.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:22:02.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:22:02.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:22:02.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:22:02.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:22:02.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:22:02.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:22:02.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:22:02.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:22:02.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:22:02.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:02.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:22:02.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:22:03.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:22:03.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:22:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:22:03.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:22:03.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:22:03.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:22:03.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:22:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:22:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:22:03.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:22:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:22:03.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:22:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:22:04.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:22:04.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:22:04.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:22:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:22:04.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:22:04.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:22:04.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:22:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:22:04.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:22:04.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:22:04.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:22:04.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:04.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:22:05.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:22:05.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:05.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:22:05.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:22:05.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:05.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:22:05.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:22:05.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:05.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:22:05.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:22:05.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:05.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:22:05.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:22:05.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:05.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:22:05.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:22:06.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:22:06.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:22:06.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:22:06.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:22:06.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:22:06.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:22:06.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:22:06.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:22:06.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:22:06.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:22:06.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:22:06.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:22:06.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:06.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:22:07.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:22:07.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:22:07.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:22:07.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:22:07.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:22:07.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:22:07.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:22:07.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:22:07.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:22:07.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:22:07.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:22:07.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:07.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:22:07.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:22:08.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:22:08.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:22:08.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:22:08.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:22:08.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:22:08.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:22:08.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:22:08.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:22:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:22:08.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:22:08.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:22:08.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:22:08.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:08.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:22:09.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:22:09.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:22:09.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:22:09.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:22:09.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:22:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:22:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:22:09.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:22:09.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:22:09.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:22:09.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:22:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:09.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:22:10.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:22:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:22:10.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:22:10.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:10.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:22:10.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:22:10.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:10.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:22:10.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:22:11.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:22:11.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:22:11.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:22:11.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:22:11.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:22:11.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:22:11.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:22:11.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:22:11.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:22:11.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:22:11.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:11.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:22:11.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:22:11.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:22:12.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:22:12.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:22:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:22:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:22:12.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:22:12.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:22:12.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:22:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:22:12.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:22:12.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:22:12.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:22:12.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:12.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:22:12.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:22:13.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:22:13.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:22:13.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:22:13.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:22:13.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:22:13.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:22:13.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:22:13.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:22:13.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:22:13.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:22:13.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:13.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:22:13.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:22:14.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:22:14.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:22:14.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:22:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:22:14.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:22:14.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:22:14.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:22:14.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:22:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:22:14.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:22:14.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:22:14.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:22:14.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:14.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:22:15.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:22:15.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:22:15.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:22:15.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:22:15.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:22:15.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:22:15.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:22:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:22:15.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:22:15.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:22:15.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:22:15.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:15.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:22:16.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:22:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:22:16.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:22:16.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:22:16.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:22:16.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:22:16.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:22:16.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:22:16.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:22:16.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:22:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:22:16.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:16.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:22:16.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:22:17.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:22:17.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:22:17.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:22:17.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:22:17.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:22:17.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:22:17.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:22:17.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:22:17.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:22:17.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:22:17.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:17.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:22:17.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:22:18.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:22:18.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:22:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:22:18.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:22:18.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:22:18.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:22:18.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:22:18.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:22:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:22:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:22:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:22:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:22:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:18.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:22:19.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:22:19.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:22:19.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:22:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:22:19.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:22:19.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:22:19.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:22:19.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:22:19.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:22:19.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:22:19.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:22:19.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:19.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:22:19.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:22:20.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:22:20.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:22:20.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:22:20.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:22:20.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:22:20.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:22:20.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:22:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:22:20.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:22:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:22:20.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:22:20.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:22:20.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:20.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:22:21.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:22:21.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:22:21.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:22:21.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:22:21.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:22:21.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:22:21.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:22:21.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:22:21.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:22:21.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:22:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:22:21.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:21.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:22:22.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:22:22.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:22:22.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:22:22.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:22:22.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:22:22.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:22:22.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:22:22.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:22:22.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:22:22.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:22:22.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:22:22.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:22.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:22:22.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:22:23.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:22:23.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:22:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:22:23.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:22:23.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:22:23.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:22:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:22:23.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:22:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:22:23.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:22:23.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:23.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:22:23.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:22:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:22:24.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:22:24.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:22:24.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:22:24.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:22:24.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:22:24.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:22:24.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:22:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:22:24.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:22:24.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:22:24.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:22:24.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:24.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:22:24.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:22:25.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:22:25.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:22:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:22:25.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:22:25.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:22:25.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:22:25.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:22:25.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:22:25.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:22:25.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:22:25.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:25.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:22:25.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:22:26.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:22:26.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:22:26.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:22:26.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:22:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:22:26.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:22:26.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:22:26.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:22:26.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:22:26.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:22:26.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:22:26.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:22:27.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:22:27.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:22:27.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:22:27.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:22:27.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:22:27.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:22:27.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:22:27.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:22:27.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:22:27.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:22:27.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:27.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:22:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:22:28.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:22:28.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:22:28.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:22:28.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:22:28.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:22:28.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:22:28.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:22:28.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:22:28.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:22:28.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:22:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:22:28.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:22:28.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:28.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:22:28.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:22:29.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:22:29.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:22:29.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:22:29.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:22:29.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:22:29.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:22:29.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:22:29.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:22:29.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:22:29.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:22:29.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:29.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:22:29.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:22:30.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:22:30.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:22:30.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:22:30.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:22:30.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:22:30.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:22:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:22:30.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:22:30.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:22:30.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:22:30.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:22:30.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:22:30.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:30.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:22:31.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:22:31.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:22:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:22:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:22:31.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:22:31.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:22:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:22:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:22:31.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:22:31.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:22:31.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:22:31.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:31.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:22:32.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:22:32.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:22:32.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:22:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:22:32.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:22:32.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:22:32.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:22:32.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:22:32.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:22:32.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:22:32.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:22:32.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:32.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:22:32.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:22:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:22:33.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:22:33.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:22:33.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:22:33.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:22:33.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:22:33.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:22:33.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:22:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:22:33.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:22:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:22:33.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:22:34.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:22:34.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:22:34.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:22:34.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:22:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:22:34.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:22:34.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:22:34.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:22:34.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:22:34.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:22:34.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:22:34.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:22:34.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:34.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:22:35.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:22:35.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:35.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:22:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:22:35.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:35.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:22:35.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:22:35.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:35.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:22:35.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:22:35.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:35.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:22:35.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:22:35.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:35.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:22:35.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:22:36.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:22:36.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:22:36.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:22:36.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:22:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:22:36.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:22:36.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:22:36.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:22:36.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:22:36.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:22:36.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:22:36.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:22:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:22:37.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:22:37.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:22:37.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:22:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:22:37.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:22:37.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:22:37.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:22:37.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:22:37.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:22:37.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:22:37.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:22:37.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:22:37.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:22:38.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:22:38.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:22:38.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:22:38.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:22:38.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:22:38.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:22:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:22:38.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:22:38.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:22:38.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:22:38.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:22:38.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:22:38.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:38.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:22:39.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:22:39.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:22:39.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:22:39.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:22:39.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:22:39.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:22:39.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:22:39.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:22:39.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:22:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:22:39.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:22:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:22:40.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:22:40.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:22:40.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:22:40.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:22:40.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:22:40.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:22:40.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:22:40.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:22:40.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:22:40.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:22:40.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:22:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:22:40.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:22:41.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:41.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:22:41.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:22:41.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:41.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:22:41.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:22:41.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:41.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:22:41.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:22:41.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:41.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:22:42.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:22:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:22:42.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:22:42.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:22:42.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:22:42.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:22:42.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:22:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:22:42.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:22:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:22:42.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:22:42.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:42.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:22:42.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:22:43.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:22:43.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:22:43.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:22:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:22:43.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:22:43.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:22:43.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:22:43.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:22:43.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:22:43.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:22:43.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:43.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:22:43.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:22:44.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:373: TEST_backfill_test_sametarget: for i in $(seq 1 $objects) 2021-01-15T17:22:44.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:375: TEST_backfill_test_sametarget: rados -p test1 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:22:44.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:376: TEST_backfill_test_sametarget: rados -p test4 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:22:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:379: TEST_backfill_test_sametarget: ceph osd pool set test1 size 2 2021-01-15T17:22:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:22:44.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:380: TEST_backfill_test_sametarget: ceph osd pool set test4 size 2 2021-01-15T17:22:45.743 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2021-01-15T17:22:45.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:381: TEST_backfill_test_sametarget: sleep 30 2021-01-15T17:23:15.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:383: TEST_backfill_test_sametarget: wait_for_not_backfilling 240 2021-01-15T17:23:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:23:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:23:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:23:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:23:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:23:15.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:23:15.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:23:15.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:23:15.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:23:15.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:23:15.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:23:15.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:23:15.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:23:15.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:23:15.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:23:15.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:23:15.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:23:16.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:23:16.293 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:23:16.293 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:23:16.293 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:23:16.293 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:23:16.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:23:16.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:23:16.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:23:16.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836530 2021-01-15T17:23:16.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836530 2021-01-15T17:23:16.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530' 2021-01-15T17:23:16.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:23:16.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:23:16.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673007 2021-01-15T17:23:16.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673007 2021-01-15T17:23:16.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673007' 2021-01-15T17:23:16.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:23:16.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:23:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509486 2021-01-15T17:23:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509486 2021-01-15T17:23:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673007 2-64424509486' 2021-01-15T17:23:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:23:16.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:23:16.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345965 2021-01-15T17:23:16.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345965 2021-01-15T17:23:16.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673007 2-64424509486 3-85899345965' 2021-01-15T17:23:16.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:23:16.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:23:16.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182443 2021-01-15T17:23:16.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182443 2021-01-15T17:23:16.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836530 1-42949673007 2-64424509486 3-85899345965 4-107374182443' 2021-01-15T17:23:16.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:23:16.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836530 2021-01-15T17:23:16.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:23:17.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:23:17.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836530 2021-01-15T17:23:17.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:23:17.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836530 2021-01-15T17:23:17.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2021-01-15T17:23:17.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:23:17.001 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836530 2021-01-15T17:23:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836530 -lt 21474836530 2021-01-15T17:23:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:23:17.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673007 2021-01-15T17:23:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:23:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:23:17.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673007 2021-01-15T17:23:17.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:23:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673007 2021-01-15T17:23:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2021-01-15T17:23:17.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:23:17.412 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673007 2021-01-15T17:23:17.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673008 -lt 42949673007 2021-01-15T17:23:17.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:23:17.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509486 2021-01-15T17:23:17.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:23:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:23:17.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509486 2021-01-15T17:23:17.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:23:17.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509486 2021-01-15T17:23:17.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509486' 2021-01-15T17:23:17.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:23:17.812 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509486 2021-01-15T17:23:18.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509486 -lt 64424509486 2021-01-15T17:23:18.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:23:18.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345965 2021-01-15T17:23:18.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:23:18.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:23:18.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345965 2021-01-15T17:23:18.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:23:18.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345965 2021-01-15T17:23:18.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2021-01-15T17:23:18.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:23:18.251 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345965 2021-01-15T17:23:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345965 -lt 85899345965 2021-01-15T17:23:18.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:23:18.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182443 2021-01-15T17:23:18.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:23:18.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:23:18.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182443 2021-01-15T17:23:18.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:23:18.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182443 2021-01-15T17:23:18.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182443' 2021-01-15T17:23:18.650 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182443 2021-01-15T17:23:18.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:23:19.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182443 -lt 107374182443 2021-01-15T17:23:19.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:23:19.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:23:19.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:23:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:23:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:23:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:23:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:23:19.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:23:19.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:23:19.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:23:19.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:23:20.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:23:20.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:23:20.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2021-01-15T17:23:20.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:23:20.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2021-01-15T17:23:20.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:23:25.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:23:25.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:23:25.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:23:25.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:23:25.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:23:25.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:23:25.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:23:25.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:23:25.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:23:25.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:23:25.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:23:25.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2021-01-15T17:23:25.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2021-01-15T17:23:35.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:23:35.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:23:35.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:23:35.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:23:35.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:23:35.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:23:35.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:23:35.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:23:35.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:23:35.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:23:35.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:23:35.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2021-01-15T17:23:35.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:23:50.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:23:50.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:23:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:23:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:23:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:23:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:23:50.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:23:50.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:23:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:23:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:23:51.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:23:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2021-01-15T17:23:51.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:24:06.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:24:06.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:24:06.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:24:06.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:24:06.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:24:06.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:24:06.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:24:06.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:24:06.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:24:06.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:24:06.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:24:06.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2021-01-15T17:24:06.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:24:21.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:24:21.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:24:21.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:24:21.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:24:21.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:24:21.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:24:21.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:24:21.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:24:22.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:24:22.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:24:22.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:24:22.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 5 >= 17 )) 2021-01-15T17:24:22.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:24:37.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:24:37.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:24:37.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:24:37.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:24:37.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:24:37.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:24:37.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:24:37.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:24:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:24:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:24:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:24:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:24:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:384: TEST_backfill_test_sametarget: wait_for_not_activating 60 2021-01-15T17:24:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:24:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:24:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:24:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:24:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:24:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:24:37.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:24:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:24:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:24:37.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:24:37.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:24:37.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:24:37.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:24:37.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:24:37.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:24:37.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:24:37.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:24:37.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:24:37.903 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:24:37.903 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:24:37.903 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:24:37.904 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:24:37.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:24:37.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:24:37.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:24:38.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836548 2021-01-15T17:24:38.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836548 2021-01-15T17:24:38.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548' 2021-01-15T17:24:38.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:24:38.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:24:38.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673026 2021-01-15T17:24:38.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673026 2021-01-15T17:24:38.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673026' 2021-01-15T17:24:38.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:24:38.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:24:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509504 2021-01-15T17:24:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509504 2021-01-15T17:24:38.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673026 2-64424509504' 2021-01-15T17:24:38.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:24:38.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:24:38.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345984 2021-01-15T17:24:38.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345984 2021-01-15T17:24:38.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673026 2-64424509504 3-85899345984' 2021-01-15T17:24:38.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:24:38.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:24:38.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182461 2021-01-15T17:24:38.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182461 2021-01-15T17:24:38.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836548 1-42949673026 2-64424509504 3-85899345984 4-107374182461' 2021-01-15T17:24:38.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:24:38.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836548 2021-01-15T17:24:38.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:24:38.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:24:38.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836548 2021-01-15T17:24:38.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:24:38.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836548 2021-01-15T17:24:38.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836548' 2021-01-15T17:24:38.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:24:38.627 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836548 2021-01-15T17:24:39.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836546 -lt 21474836548 2021-01-15T17:24:39.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:24:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:24:40.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:24:40.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836549 -lt 21474836548 2021-01-15T17:24:40.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:24:40.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673026 2021-01-15T17:24:40.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:24:40.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:24:40.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673026 2021-01-15T17:24:40.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:24:40.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673026 2021-01-15T17:24:40.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673026' 2021-01-15T17:24:40.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:24:40.425 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673026 2021-01-15T17:24:40.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673026 -lt 42949673026 2021-01-15T17:24:40.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:24:40.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509504 2021-01-15T17:24:40.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:24:40.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:24:40.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509504 2021-01-15T17:24:40.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:24:40.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509504 2021-01-15T17:24:40.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509504' 2021-01-15T17:24:40.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:24:40.825 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509504 2021-01-15T17:24:41.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509505 -lt 64424509504 2021-01-15T17:24:41.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:24:41.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345984 2021-01-15T17:24:41.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:24:41.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:24:41.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345984 2021-01-15T17:24:41.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:24:41.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345984 2021-01-15T17:24:41.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345984' 2021-01-15T17:24:41.227 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345984 2021-01-15T17:24:41.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:24:41.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345984 -lt 85899345984 2021-01-15T17:24:41.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:24:41.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182461 2021-01-15T17:24:41.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:24:41.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:24:41.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182461 2021-01-15T17:24:41.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:24:41.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182461 2021-01-15T17:24:41.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182461' 2021-01-15T17:24:41.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:24:41.658 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182461 2021-01-15T17:24:42.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182461 -lt 107374182461 2021-01-15T17:24:42.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:24:42.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:24:42.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:24:42.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:24:42.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:24:42.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:24:42.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:24:42.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:24:42.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:24:42.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:24:42.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:24:42.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:24:42.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:24:42.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:24:42.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:24:42.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:386: TEST_backfill_test_sametarget: ERRORS=0 2021-01-15T17:24:42.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: ceph pg dump pgs 2021-01-15T17:24:42.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: grep +backfill_toofull 2021-01-15T17:24:42.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: wc -l 2021-01-15T17:24:43.327 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:24:43.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:387: TEST_backfill_test_sametarget: '[' 1 '!=' 1 ']' 2021-01-15T17:24:43.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: ceph pg dump pgs 2021-01-15T17:24:43.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: grep active+clean 2021-01-15T17:24:43.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: wc -l 2021-01-15T17:24:43.710 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:24:43.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:393: TEST_backfill_test_sametarget: '[' 1 '!=' 1 ']' 2021-01-15T17:24:43.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:399: TEST_backfill_test_sametarget: ceph pg dump pgs 2021-01-15T17:24:44.089 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:24:44.089 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:24:24.294509+0000 95'600 104:1901 [1,0] 1 [1,0] 1 0'0 2021-01-15T17:20:16.928220+0000 0'0 2021-01-15T17:20:16.928220+0000 0 2021-01-15T17:24:44.090 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:24:16.745868+0000 95'600 103:685 [4,0] 4 [4] 4 0'0 2021-01-15T17:20:25.061183+0000 0'0 2021-01-15T17:20:25.061183+0000 0 2021-01-15T17:24:44.090 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:24:44.090 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:24:44.091 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:24:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:401: TEST_backfill_test_sametarget: '[' 0 '!=' 0 ']' 2021-01-15T17:24:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:406: TEST_backfill_test_sametarget: delete_pool test1 2021-01-15T17:24:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:24:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:24:44.633 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:24:44.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:407: TEST_backfill_test_sametarget: delete_pool test4 2021-01-15T17:24:44.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2021-01-15T17:24:44.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-01-15T17:24:45.275 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' does not exist 2021-01-15T17:24:45.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:408: TEST_backfill_test_sametarget: kill_daemons td/osd-backfill-space 2021-01-15T17:24:45.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:24:45.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:24:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:24:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:24:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:24:45.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:24:45.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:409: TEST_backfill_test_sametarget: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log td/osd-backfill-space/osd.3.log td/osd-backfill-space/osd.4.log 2021-01-15T17:24:45.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:24:45.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:24:45.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:24:45.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:24:45.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:24:45.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:24:45.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:24:45.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:24:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:24:45.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:24:45.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:24:45.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:24:45.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:24:45.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:24:45.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:24:45.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:24:45.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:24:45.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:24:45.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:24:45.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:24:45.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:24:45.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:24:45.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:24:45.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:24:45.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:45.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:45.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:24:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:24:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:24:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:24:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:24:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:24:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:24:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:24:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:24:45.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:24:45.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:24:45.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:24:45.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:24:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:24:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:24:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:24:45.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:24:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:24:45.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:24:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:24:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:24:45.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:24:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:24:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:24:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:24:45.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:24:45.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:24:45.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:24:45.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:24:45.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:24:45.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:45.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:45.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:24:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:24:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:24:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:24:45.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:24:45.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:45.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:45.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:24:45.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:24:45.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:24:45.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:24:45.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:24:45.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_backfill_test_simple td/osd-backfill-space 2021-01-15T17:24:45.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:109: TEST_backfill_test_simple: local dir=td/osd-backfill-space 2021-01-15T17:24:45.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:110: TEST_backfill_test_simple: local pools=2 2021-01-15T17:24:45.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:111: TEST_backfill_test_simple: local OSDS=3 2021-01-15T17:24:45.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:113: TEST_backfill_test_simple: run_mon td/osd-backfill-space a 2021-01-15T17:24:45.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:24:45.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:24:45.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:24:45.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:24:45.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:24:45.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:24:45.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:24:45.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:24:45.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:24:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:24:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:45.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:24:45.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:24:45.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:24:45.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:24:45.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:24:45.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:24:45.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:24:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:24:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:24:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:24:45.951 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:24:45.952 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:45.952 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:45.952 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:24:45.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:24:45.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:24:46.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:24:46.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:24:46.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:24:46.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:24:46.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:24:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:24:46.052 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:24:46.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:24:46.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:24:46.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:114: TEST_backfill_test_simple: run_mgr td/osd-backfill-space x 2021-01-15T17:24:46.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:24:46.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:24:46.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:24:46.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:24:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:24:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:24:46.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:24:46.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:24:46.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:24:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:24:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:46.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:46.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:24:46.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:24:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:115: TEST_backfill_test_simple: export CEPH_ARGS 2021-01-15T17:24:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: expr 3 - 1 2021-01-15T17:24:46.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: seq 0 2 2021-01-15T17:24:46.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:24:46.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 0 2021-01-15T17:24:46.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:24:46.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:24:46.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:24:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:24:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:24:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:24:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:24:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:24:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:24:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:24:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:24:46.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:24:46.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:24:46.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:24:46.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:24:46.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:24:46.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:24:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:24:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:46.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:24:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:24:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:24:46.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:24:46.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:24:46.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:24:46.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:24:46.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:24:46.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:24:46.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:24:46.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:24:46.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:24:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d954cf10-0115-4345-b816-41565f57797c 2021-01-15T17:24:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d954cf10-0115-4345-b816-41565f57797c' 2021-01-15T17:24:46.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:24:46.691 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d954cf10-0115-4345-b816-41565f57797c 2021-01-15T17:24:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDezwFgwt2qKRAAEPyJWfGWSVyC2YQxvANnSA== 2021-01-15T17:24:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDezwFgwt2qKRAAEPyJWfGWSVyC2YQxvANnSA=="}' 2021-01-15T17:24:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d954cf10-0115-4345-b816-41565f57797c -i td/osd-backfill-space/0/new.json 2021-01-15T17:24:47.042 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:24:47.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:24:47.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDezwFgwt2qKRAAEPyJWfGWSVyC2YQxvANnSA== --osd-uuid d954cf10-0115-4345-b816-41565f57797c 2021-01-15T17:24:47.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.081+0000 7f7e37cce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:47.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.083+0000 7f7e37cce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:47.084 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.084+0000 7f7e37cce080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.084+0000 7f7e37cce080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:24:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.084+0000 7f7e37cce080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:24:47.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.084+0000 7f7e37cce080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:24:47.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:47.113+0000 7f7e37cce080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:24:51.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:24:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:24:51.673 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:24:51.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:24:51.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:24:52.198 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:24:52.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:24:52.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:24:52.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:24:52.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:24:52.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:24:52.223 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:52.222+0000 7fe29d88f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:52.246 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:52.244+0000 7fe29d88f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:52.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:52.246+0000 7fe29d88f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:52.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:24:52.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:24:52.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:24:52.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:24:52.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:24:52.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:24:52.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:24:52.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:24:52.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:24:52.604 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:24:52.767 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:52.766+0000 7fe29d88f080 -1 Falling back to public interface 2021-01-15T17:24:52.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:24:53.555 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:53.554+0000 7fe29d88f080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:24:53.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:24:53.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:24:53.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:24:53.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:24:53.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:24:53.997 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:24:54.427 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/427657,v1:127.0.0.1:6803/427657] [v2:127.0.0.1:6804/427657,v1:127.0.0.1:6805/427657] exists,up d954cf10-0115-4345-b816-41565f57797c 2021-01-15T17:24:54.428 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 3f638894-1d08-460d-ba96-57937e22152d 2021-01-15T17:24:54.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:24:54.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:24:54.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:24:54.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:24:54.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 1 2021-01-15T17:24:54.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:24:54.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:24:54.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:24:54.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:24:54.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:24:54.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:24:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:24:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:24:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:24:54.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:24:54.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:24:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:24:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:24:54.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:24:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:24:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:24:54.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:24:54.437 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:24:54.438 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:24:54.438 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:24:54.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:54.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:24:54.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:24:54.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:24:54.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:24:54.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:24:54.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:24:54.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:24:54.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:24:54.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:24:54.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:24:54.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:24:54.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:24:54.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:24:54.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3f638894-1d08-460d-ba96-57937e22152d 2021-01-15T17:24:54.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 3f638894-1d08-460d-ba96-57937e22152d' 2021-01-15T17:24:54.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:24:54.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDmzwFgfAtEGhAAcAdqmsn2Re6LtYy4ypCLhg== 2021-01-15T17:24:54.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDmzwFgfAtEGhAAcAdqmsn2Re6LtYy4ypCLhg=="}' 2021-01-15T17:24:54.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3f638894-1d08-460d-ba96-57937e22152d -i td/osd-backfill-space/1/new.json 2021-01-15T17:24:54.957 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:24:54.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:24:54.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDmzwFgfAtEGhAAcAdqmsn2Re6LtYy4ypCLhg== --osd-uuid 3f638894-1d08-460d-ba96-57937e22152d 2021-01-15T17:24:54.995 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.995+0000 7f779c6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:54.997 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.997+0000 7f779c6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:54.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.998+0000 7f779c6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:24:55.000 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.999+0000 7f779c6f2080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:24:55.000 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.999+0000 7f779c6f2080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:24:55.001 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:54.999+0000 7f779c6f2080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:24:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:24:55.041+0000 7f779c6f2080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:24:59.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:24:59.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:24:59.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:24:59.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:24:59.587 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:25:00.113 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:25:00.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:25:00.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:25:00.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:25:00.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:25:00.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:25:00.138 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:00.137+0000 7efd82f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:00.139 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:00.139+0000 7efd82f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:00.141 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:00.141+0000 7efd82f97080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:25:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:25:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:25:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:25:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:25:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:25:00.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:25:00.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:25:00.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:25:00.512 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:25:00.662 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:00.662+0000 7efd82f97080 -1 Falling back to public interface 2021-01-15T17:25:00.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:25:01.477 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:01.477+0000 7efd82f97080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:25:01.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:25:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:25:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:25:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:25:01.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:25:01.933 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:25:02.336 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/428921,v1:127.0.0.1:6811/428921] [v2:127.0.0.1:6812/428921,v1:127.0.0.1:6813/428921] exists,up 3f638894-1d08-460d-ba96-57937e22152d 2021-01-15T17:25:02.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:25:02.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:25:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:25:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:117: TEST_backfill_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:25:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:119: TEST_backfill_test_simple: run_osd td/osd-backfill-space 2 2021-01-15T17:25:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:25:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:25:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:25:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:25:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:25:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:25:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:25:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:25:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:25:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:25:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:25:02.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:25:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:25:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:25:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:25:02.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:25:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:25:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:25:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:25:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:25:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:25:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:25:02.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:25:02.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:25:02.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:25:02.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:25:02.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:25:02.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:25:02.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:25:02.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f983ab61-986c-4d8d-8e21-daf4318bb3ea 2021-01-15T17:25:02.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 f983ab61-986c-4d8d-8e21-daf4318bb3ea' 2021-01-15T17:25:02.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:25:02.346 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 f983ab61-986c-4d8d-8e21-daf4318bb3ea 2021-01-15T17:25:02.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDuzwFgcx8SFRAAzLeiSffIyhdFwOCNT57MdQ== 2021-01-15T17:25:02.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDuzwFgcx8SFRAAzLeiSffIyhdFwOCNT57MdQ=="}' 2021-01-15T17:25:02.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f983ab61-986c-4d8d-8e21-daf4318bb3ea -i td/osd-backfill-space/2/new.json 2021-01-15T17:25:02.880 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:25:02.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:25:02.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDuzwFgcx8SFRAAzLeiSffIyhdFwOCNT57MdQ== --osd-uuid f983ab61-986c-4d8d-8e21-daf4318bb3ea 2021-01-15T17:25:02.920 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.920+0000 7f9c77519080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:02.922 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.922+0000 7f9c77519080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:02.923 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.922+0000 7f9c77519080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:02.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.923+0000 7f9c77519080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:25:02.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.923+0000 7f9c77519080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:25:02.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.923+0000 7f9c77519080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:25:02.984 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:02.983+0000 7f9c77519080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:25:07.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:25:07.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:25:07.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:25:07.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:25:07.619 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:25:08.150 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:25:08.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:25:08.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:25:08.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:25:08.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:25:08.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:25:08.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:08.175+0000 7fbe75f6b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:08.199 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:08.199+0000 7fbe75f6b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:08.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:08.200+0000 7fbe75f6b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:25:08.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:25:08.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:25:08.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:25:08.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:25:08.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:25:08.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:25:08.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:25:08.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:25:08.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:25:08.574 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:25:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:08.717+0000 7fbe75f6b080 -1 Falling back to public interface 2021-01-15T17:25:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:25:09.512 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:25:09.511+0000 7fbe75f6b080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:25:09.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:25:09.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:25:09.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:25:09.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:25:09.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:25:09.974 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:25:10.370 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/430185,v1:127.0.0.1:6819/430185] [v2:127.0.0.1:6820/430185,v1:127.0.0.1:6821/430185] exists,up f983ab61-986c-4d8d-8e21-daf4318bb3ea 2021-01-15T17:25:10.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:25:10.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:25:10.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:25:10.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:122: TEST_backfill_test_simple: ceph osd set-backfillfull-ratio .85 2021-01-15T17:25:11.112 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:25:11.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:11.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:11.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:126: TEST_backfill_test_simple: create_pool test1 1 1 2021-01-15T17:25:11.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T17:25:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T17:25:11.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:25:12.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:127: TEST_backfill_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T17:25:13.488 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:25:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:124: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:13.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:126: TEST_backfill_test_simple: create_pool test2 1 1 2021-01-15T17:25:13.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T17:25:14.091 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T17:25:14.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:25:15.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:127: TEST_backfill_test_simple: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-01-15T17:25:15.871 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 1 2021-01-15T17:25:15.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:130: TEST_backfill_test_simple: wait_for_clean 2021-01-15T17:25:15.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:25:15.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:25:15.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:25:15.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:25:15.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:25:15.882 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:25:15.883 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:25:15.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:25:15.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:25:15.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:25:15.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:25:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:25:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:25:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:25:15.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:25:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:25:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:25:16.409 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:25:16.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:25:16.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:25:16.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:25:16.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T17:25:16.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T17:25:16.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T17:25:16.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:25:16.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:25:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T17:25:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T17:25:16.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T17:25:16.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:25:16.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:25:16.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T17:25:16.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T17:25:16.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509444' 2021-01-15T17:25:16.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:25:16.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T17:25:16.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:25:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:25:16.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T17:25:16.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:25:16.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T17:25:16.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T17:25:16.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:25:16.831 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T17:25:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2021-01-15T17:25:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:25:18.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:25:18.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:25:18.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T17:25:18.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:25:18.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T17:25:18.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:25:18.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:25:18.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T17:25:18.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:25:18.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T17:25:18.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T17:25:18.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:25:18.638 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T17:25:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672965 2021-01-15T17:25:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:25:19.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T17:25:19.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:25:19.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:25:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T17:25:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:25:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T17:25:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T17:25:19.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:25:19.035 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T17:25:19.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T17:25:19.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:25:19.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:25:19.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:25:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T17:25:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:25:19.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:25:19.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:25:19.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:25:19.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:25:19.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:25:19.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:25:20.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T17:25:20.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:25:20.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:25:20.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:25:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T17:25:20.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:25:20.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:25:20.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:135: TEST_backfill_test_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=4 2021-01-15T17:25:20.879 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T17:25:20.879 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T17:25:20.879 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.7454e-05 s, 60.7 MB/s 2021-01-15T17:25:20.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: seq 1 600 2021-01-15T17:25:20.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:20.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:20.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:20.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:25:20.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:20.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:25:21.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:25:21.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:25:21.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:25:21.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:25:21.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:25:21.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:25:21.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:25:21.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:25:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:25:21.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:25:21.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:21.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:21.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:25:21.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:21.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:25:22.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:25:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:25:22.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:25:22.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:25:22.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:25:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:25:22.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:25:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:25:22.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:25:22.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:25:22.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:25:22.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:25:22.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:22.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:22.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:22.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:25:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:25:23.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:25:23.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:25:23.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:25:23.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:25:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:25:23.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:25:23.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:25:23.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:25:23.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:25:23.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:25:23.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:23.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:23.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:25:23.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:23.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:25:24.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:25:24.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:25:24.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:25:24.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:25:24.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:25:24.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:25:24.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:25:24.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:25:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:25:24.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:25:24.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:24.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:24.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:25:24.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:24.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:25:25.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:25:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:25:25.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:25:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:25:25.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:25:25.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:25:25.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:25:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:25:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:25:25.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:25:25.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:25.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:25:25.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:25.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:25:26.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:25:26.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:25:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:25:26.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:25:26.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:25:26.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:25:26.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:25:26.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:25:26.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:25:26.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:25:26.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:26.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:26.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:25:26.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:26.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:25:27.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:25:27.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:25:27.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:25:27.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:25:27.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:25:27.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:25:27.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:25:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:25:27.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:25:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:25:27.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:25:27.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:25:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:27.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:25:27.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:27.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:25:28.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:25:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:25:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:25:28.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:25:28.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:25:28.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:25:28.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:25:28.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:25:28.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:25:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:25:28.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:28.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:28.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:25:28.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:28.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:25:29.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:25:29.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:25:29.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:25:29.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:25:29.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:25:29.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:25:29.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:25:29.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:25:29.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:25:29.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:25:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:25:29.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:25:29.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:29.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:29.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:29.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:25:30.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:25:30.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:25:30.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:25:30.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:25:30.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:25:30.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:25:30.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:25:30.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:25:30.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:25:30.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:25:30.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:25:30.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:30.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:30.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:25:30.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:30.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:25:31.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:25:31.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:25:31.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:25:31.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:25:31.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:25:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:25:31.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:25:31.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:25:31.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:25:31.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:25:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:31.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:25:31.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:31.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:25:32.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:25:32.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:25:32.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:25:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:25:32.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:25:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:25:32.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:25:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:25:32.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:25:32.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:25:32.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:32.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:32.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:25:32.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:25:33.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:25:33.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:25:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:25:33.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:25:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:25:33.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:25:33.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:25:33.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:25:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:25:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:25:33.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:25:33.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:25:33.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:33.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:33.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:25:34.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:25:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:25:34.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:25:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:25:34.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:25:34.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:25:34.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:25:34.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:25:34.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:25:34.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:25:34.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:25:34.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:34.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:34.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:34.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:25:35.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:25:35.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:25:35.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:25:35.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:25:35.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:25:35.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:25:35.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:25:35.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:25:35.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:25:35.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:25:35.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:25:35.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:25:35.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:25:35.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:35.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:35.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:35.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:25:36.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:25:36.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:36.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:36.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:25:36.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:25:36.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:36.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:36.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:25:36.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:25:36.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:36.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:36.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:25:36.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:25:36.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:36.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:36.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:25:36.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:25:36.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:36.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:36.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:25:36.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:36.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:25:37.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:25:37.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:25:37.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:25:37.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:25:37.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:25:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:25:37.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:25:37.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:25:37.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:25:37.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:25:37.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:25:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:25:37.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:37.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:25:38.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:25:38.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:25:38.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:25:38.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:25:38.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:25:38.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:25:38.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:25:38.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:25:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:25:38.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:25:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:25:38.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:38.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:38.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:25:38.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:38.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:25:39.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:25:39.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:25:39.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:25:39.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:25:39.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:25:39.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:25:39.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:25:39.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:25:39.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:25:39.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:25:39.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:39.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:39.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:25:39.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:39.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:25:40.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:25:40.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:25:40.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:25:40.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:25:40.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:25:40.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:25:40.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:25:40.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:25:40.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:25:40.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:40.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:40.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:25:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:25:41.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:25:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:25:41.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:25:41.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:25:41.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:25:41.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:25:41.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:25:41.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:25:41.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:25:41.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:25:41.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:25:41.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:25:41.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:41.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:41.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:41.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:25:42.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:25:42.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:25:42.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:25:42.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:25:42.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:25:42.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:25:42.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:25:42.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:25:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:25:42.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:25:42.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:25:42.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:42.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:42.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:42.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:25:43.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:25:43.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:25:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:25:43.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:25:43.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:25:43.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:25:43.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:25:43.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:25:43.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:25:43.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:25:43.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:25:43.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:25:43.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:25:43.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:43.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:43.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:43.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:25:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:25:44.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:25:44.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:25:44.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:25:44.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:25:44.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:25:44.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:25:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:25:44.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:25:44.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:25:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:25:44.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:44.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:44.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:44.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:25:45.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:25:45.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:25:45.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:25:45.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:25:45.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:25:45.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:25:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:25:45.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:25:45.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:25:45.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:25:45.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:25:45.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:45.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:45.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:25:45.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:45.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:25:46.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:25:46.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:25:46.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:25:46.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:25:46.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:25:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:25:46.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:25:46.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:25:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:25:46.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:25:46.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:46.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:46.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:25:46.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:46.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:25:47.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:25:47.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:25:47.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:25:47.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:25:47.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:25:47.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:25:47.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:25:47.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:25:47.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:25:47.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:25:47.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:25:47.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:25:47.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:47.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:47.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:47.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:25:48.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:25:48.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:25:48.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:25:48.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:25:48.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:25:48.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:25:48.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:25:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:25:48.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:25:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:25:48.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:25:48.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:48.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:48.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:25:48.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:25:49.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:25:49.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:25:49.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:25:49.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:25:49.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:25:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:25:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:25:49.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:25:49.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:25:49.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:25:49.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:25:49.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:25:49.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:49.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:49.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:25:50.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:25:50.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:25:50.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:25:50.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:25:50.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:25:50.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:25:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:25:50.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:25:50.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:25:50.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:25:50.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:25:50.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:50.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:50.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:50.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:25:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:25:51.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:51.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:51.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:25:51.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:25:51.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:51.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:25:51.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:25:51.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:51.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:51.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:25:51.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:51.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:25:52.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:25:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:25:52.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:25:52.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:25:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:25:52.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:25:52.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:25:52.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:25:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:25:52.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:25:52.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:25:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:25:52.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:52.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:52.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:52.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:25:53.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:25:53.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:25:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj201 td/osd-backfill-space/datafile 2021-01-15T17:25:53.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:25:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj202 td/osd-backfill-space/datafile 2021-01-15T17:25:53.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:25:53.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj203 td/osd-backfill-space/datafile 2021-01-15T17:25:53.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:25:53.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj204 td/osd-backfill-space/datafile 2021-01-15T17:25:53.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:25:53.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj205 td/osd-backfill-space/datafile 2021-01-15T17:25:53.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:53.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:53.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:25:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj206 td/osd-backfill-space/datafile 2021-01-15T17:25:54.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:25:54.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj207 td/osd-backfill-space/datafile 2021-01-15T17:25:54.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:25:54.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj208 td/osd-backfill-space/datafile 2021-01-15T17:25:54.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:25:54.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj209 td/osd-backfill-space/datafile 2021-01-15T17:25:54.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:25:54.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj210 td/osd-backfill-space/datafile 2021-01-15T17:25:54.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:25:54.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj211 td/osd-backfill-space/datafile 2021-01-15T17:25:54.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:54.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:54.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:25:54.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:54.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj212 td/osd-backfill-space/datafile 2021-01-15T17:25:55.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:25:55.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj213 td/osd-backfill-space/datafile 2021-01-15T17:25:55.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:25:55.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj214 td/osd-backfill-space/datafile 2021-01-15T17:25:55.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:25:55.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj215 td/osd-backfill-space/datafile 2021-01-15T17:25:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:25:55.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj216 td/osd-backfill-space/datafile 2021-01-15T17:25:55.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:25:55.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj217 td/osd-backfill-space/datafile 2021-01-15T17:25:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:25:55.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj218 td/osd-backfill-space/datafile 2021-01-15T17:25:55.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:55.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:55.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:55.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:25:56.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj219 td/osd-backfill-space/datafile 2021-01-15T17:25:56.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:25:56.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj220 td/osd-backfill-space/datafile 2021-01-15T17:25:56.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:25:56.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj221 td/osd-backfill-space/datafile 2021-01-15T17:25:56.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:25:56.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj222 td/osd-backfill-space/datafile 2021-01-15T17:25:56.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:25:56.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj223 td/osd-backfill-space/datafile 2021-01-15T17:25:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:25:56.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj224 td/osd-backfill-space/datafile 2021-01-15T17:25:56.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:56.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:25:56.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj225 td/osd-backfill-space/datafile 2021-01-15T17:25:57.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:25:57.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj226 td/osd-backfill-space/datafile 2021-01-15T17:25:57.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:25:57.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj227 td/osd-backfill-space/datafile 2021-01-15T17:25:57.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:25:57.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj228 td/osd-backfill-space/datafile 2021-01-15T17:25:57.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:25:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj229 td/osd-backfill-space/datafile 2021-01-15T17:25:57.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:25:57.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj230 td/osd-backfill-space/datafile 2021-01-15T17:25:57.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:57.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:57.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:25:57.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj231 td/osd-backfill-space/datafile 2021-01-15T17:25:58.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:25:58.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj232 td/osd-backfill-space/datafile 2021-01-15T17:25:58.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:25:58.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj233 td/osd-backfill-space/datafile 2021-01-15T17:25:58.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:25:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj234 td/osd-backfill-space/datafile 2021-01-15T17:25:58.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:25:58.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj235 td/osd-backfill-space/datafile 2021-01-15T17:25:58.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:25:58.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj236 td/osd-backfill-space/datafile 2021-01-15T17:25:58.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:25:58.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj237 td/osd-backfill-space/datafile 2021-01-15T17:25:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:58.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:25:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj238 td/osd-backfill-space/datafile 2021-01-15T17:25:59.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:25:59.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj239 td/osd-backfill-space/datafile 2021-01-15T17:25:59.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:25:59.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj240 td/osd-backfill-space/datafile 2021-01-15T17:25:59.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:25:59.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj241 td/osd-backfill-space/datafile 2021-01-15T17:25:59.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:25:59.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj242 td/osd-backfill-space/datafile 2021-01-15T17:25:59.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:25:59.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj243 td/osd-backfill-space/datafile 2021-01-15T17:25:59.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:25:59.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:25:59.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:25:59.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:25:59.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj244 td/osd-backfill-space/datafile 2021-01-15T17:26:00.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:26:00.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj245 td/osd-backfill-space/datafile 2021-01-15T17:26:00.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:26:00.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj246 td/osd-backfill-space/datafile 2021-01-15T17:26:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:26:00.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj247 td/osd-backfill-space/datafile 2021-01-15T17:26:00.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:26:00.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj248 td/osd-backfill-space/datafile 2021-01-15T17:26:00.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:26:00.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj249 td/osd-backfill-space/datafile 2021-01-15T17:26:00.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:00.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:00.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:26:00.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:00.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj250 td/osd-backfill-space/datafile 2021-01-15T17:26:01.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:26:01.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj251 td/osd-backfill-space/datafile 2021-01-15T17:26:01.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:26:01.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj252 td/osd-backfill-space/datafile 2021-01-15T17:26:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:26:01.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj253 td/osd-backfill-space/datafile 2021-01-15T17:26:01.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:26:01.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj254 td/osd-backfill-space/datafile 2021-01-15T17:26:01.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:26:01.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj255 td/osd-backfill-space/datafile 2021-01-15T17:26:01.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:26:01.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj256 td/osd-backfill-space/datafile 2021-01-15T17:26:01.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:01.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:01.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:01.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:26:02.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj257 td/osd-backfill-space/datafile 2021-01-15T17:26:02.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:26:02.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj258 td/osd-backfill-space/datafile 2021-01-15T17:26:02.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:26:02.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj259 td/osd-backfill-space/datafile 2021-01-15T17:26:02.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:26:02.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj260 td/osd-backfill-space/datafile 2021-01-15T17:26:02.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:26:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj261 td/osd-backfill-space/datafile 2021-01-15T17:26:02.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:26:02.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj262 td/osd-backfill-space/datafile 2021-01-15T17:26:02.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:02.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:02.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:02.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:26:03.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj263 td/osd-backfill-space/datafile 2021-01-15T17:26:03.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:26:03.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj264 td/osd-backfill-space/datafile 2021-01-15T17:26:03.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:26:03.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj265 td/osd-backfill-space/datafile 2021-01-15T17:26:03.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:26:03.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj266 td/osd-backfill-space/datafile 2021-01-15T17:26:03.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:26:03.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj267 td/osd-backfill-space/datafile 2021-01-15T17:26:03.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:26:03.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj268 td/osd-backfill-space/datafile 2021-01-15T17:26:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:03.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:26:03.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:03.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj269 td/osd-backfill-space/datafile 2021-01-15T17:26:04.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:26:04.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj270 td/osd-backfill-space/datafile 2021-01-15T17:26:04.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:26:04.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj271 td/osd-backfill-space/datafile 2021-01-15T17:26:04.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:26:04.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj272 td/osd-backfill-space/datafile 2021-01-15T17:26:04.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:26:04.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj273 td/osd-backfill-space/datafile 2021-01-15T17:26:04.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:26:04.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj274 td/osd-backfill-space/datafile 2021-01-15T17:26:04.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:04.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:04.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:26:04.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:04.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj275 td/osd-backfill-space/datafile 2021-01-15T17:26:05.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:26:05.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj276 td/osd-backfill-space/datafile 2021-01-15T17:26:05.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:26:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj277 td/osd-backfill-space/datafile 2021-01-15T17:26:05.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:26:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj278 td/osd-backfill-space/datafile 2021-01-15T17:26:05.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:26:05.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj279 td/osd-backfill-space/datafile 2021-01-15T17:26:05.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:26:05.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj280 td/osd-backfill-space/datafile 2021-01-15T17:26:05.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:26:05.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj281 td/osd-backfill-space/datafile 2021-01-15T17:26:05.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:05.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:05.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:05.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:26:06.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj282 td/osd-backfill-space/datafile 2021-01-15T17:26:06.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:26:06.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj283 td/osd-backfill-space/datafile 2021-01-15T17:26:06.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:26:06.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj284 td/osd-backfill-space/datafile 2021-01-15T17:26:06.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:26:06.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj285 td/osd-backfill-space/datafile 2021-01-15T17:26:06.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:26:06.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj286 td/osd-backfill-space/datafile 2021-01-15T17:26:06.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:26:06.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj287 td/osd-backfill-space/datafile 2021-01-15T17:26:06.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:06.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:06.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:06.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:26:07.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj288 td/osd-backfill-space/datafile 2021-01-15T17:26:07.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:26:07.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj289 td/osd-backfill-space/datafile 2021-01-15T17:26:07.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:26:07.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj290 td/osd-backfill-space/datafile 2021-01-15T17:26:07.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:26:07.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj291 td/osd-backfill-space/datafile 2021-01-15T17:26:07.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:26:07.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj292 td/osd-backfill-space/datafile 2021-01-15T17:26:07.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:26:07.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj293 td/osd-backfill-space/datafile 2021-01-15T17:26:07.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:07.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:07.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:07.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:26:08.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj294 td/osd-backfill-space/datafile 2021-01-15T17:26:08.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:26:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj295 td/osd-backfill-space/datafile 2021-01-15T17:26:08.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:26:08.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj296 td/osd-backfill-space/datafile 2021-01-15T17:26:08.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:26:08.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj297 td/osd-backfill-space/datafile 2021-01-15T17:26:08.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:26:08.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj298 td/osd-backfill-space/datafile 2021-01-15T17:26:08.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:26:08.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj299 td/osd-backfill-space/datafile 2021-01-15T17:26:08.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:08.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:08.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:26:08.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:08.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj300 td/osd-backfill-space/datafile 2021-01-15T17:26:09.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:26:09.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj301 td/osd-backfill-space/datafile 2021-01-15T17:26:09.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:26:09.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj302 td/osd-backfill-space/datafile 2021-01-15T17:26:09.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:26:09.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj303 td/osd-backfill-space/datafile 2021-01-15T17:26:09.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:26:09.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj304 td/osd-backfill-space/datafile 2021-01-15T17:26:09.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:26:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj305 td/osd-backfill-space/datafile 2021-01-15T17:26:09.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:26:09.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj306 td/osd-backfill-space/datafile 2021-01-15T17:26:09.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:09.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:09.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:09.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:26:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj307 td/osd-backfill-space/datafile 2021-01-15T17:26:10.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:26:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj308 td/osd-backfill-space/datafile 2021-01-15T17:26:10.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:26:10.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj309 td/osd-backfill-space/datafile 2021-01-15T17:26:10.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:26:10.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj310 td/osd-backfill-space/datafile 2021-01-15T17:26:10.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:26:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj311 td/osd-backfill-space/datafile 2021-01-15T17:26:10.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:26:10.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj312 td/osd-backfill-space/datafile 2021-01-15T17:26:10.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:10.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:10.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:26:11.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj313 td/osd-backfill-space/datafile 2021-01-15T17:26:11.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:26:11.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj314 td/osd-backfill-space/datafile 2021-01-15T17:26:11.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:26:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj315 td/osd-backfill-space/datafile 2021-01-15T17:26:11.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:26:11.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj316 td/osd-backfill-space/datafile 2021-01-15T17:26:11.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:26:11.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj317 td/osd-backfill-space/datafile 2021-01-15T17:26:11.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:26:11.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj318 td/osd-backfill-space/datafile 2021-01-15T17:26:11.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:11.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:11.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:11.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:26:11.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj319 td/osd-backfill-space/datafile 2021-01-15T17:26:12.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:26:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj320 td/osd-backfill-space/datafile 2021-01-15T17:26:12.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:26:12.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj321 td/osd-backfill-space/datafile 2021-01-15T17:26:12.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:26:12.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj322 td/osd-backfill-space/datafile 2021-01-15T17:26:12.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:26:12.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj323 td/osd-backfill-space/datafile 2021-01-15T17:26:12.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:26:12.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj324 td/osd-backfill-space/datafile 2021-01-15T17:26:12.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:12.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:12.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:26:12.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:12.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj325 td/osd-backfill-space/datafile 2021-01-15T17:26:13.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:26:13.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj326 td/osd-backfill-space/datafile 2021-01-15T17:26:13.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:26:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj327 td/osd-backfill-space/datafile 2021-01-15T17:26:13.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:26:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj328 td/osd-backfill-space/datafile 2021-01-15T17:26:13.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:26:13.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj329 td/osd-backfill-space/datafile 2021-01-15T17:26:13.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:26:13.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj330 td/osd-backfill-space/datafile 2021-01-15T17:26:13.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:26:13.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj331 td/osd-backfill-space/datafile 2021-01-15T17:26:13.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:13.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:13.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:13.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:26:14.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj332 td/osd-backfill-space/datafile 2021-01-15T17:26:14.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:26:14.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj333 td/osd-backfill-space/datafile 2021-01-15T17:26:14.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:26:14.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj334 td/osd-backfill-space/datafile 2021-01-15T17:26:14.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:26:14.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj335 td/osd-backfill-space/datafile 2021-01-15T17:26:14.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:26:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj336 td/osd-backfill-space/datafile 2021-01-15T17:26:14.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:26:14.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj337 td/osd-backfill-space/datafile 2021-01-15T17:26:14.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:14.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:14.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:14.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:26:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj338 td/osd-backfill-space/datafile 2021-01-15T17:26:15.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:26:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj339 td/osd-backfill-space/datafile 2021-01-15T17:26:15.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:26:15.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj340 td/osd-backfill-space/datafile 2021-01-15T17:26:15.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:26:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj341 td/osd-backfill-space/datafile 2021-01-15T17:26:15.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:26:15.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj342 td/osd-backfill-space/datafile 2021-01-15T17:26:15.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:26:15.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj343 td/osd-backfill-space/datafile 2021-01-15T17:26:15.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:15.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:15.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:26:15.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:15.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj344 td/osd-backfill-space/datafile 2021-01-15T17:26:16.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:26:16.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj345 td/osd-backfill-space/datafile 2021-01-15T17:26:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:26:16.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj346 td/osd-backfill-space/datafile 2021-01-15T17:26:16.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:26:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj347 td/osd-backfill-space/datafile 2021-01-15T17:26:16.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:26:16.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj348 td/osd-backfill-space/datafile 2021-01-15T17:26:16.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:26:16.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj349 td/osd-backfill-space/datafile 2021-01-15T17:26:16.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:16.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:16.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:26:16.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:16.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj350 td/osd-backfill-space/datafile 2021-01-15T17:26:17.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:26:17.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj351 td/osd-backfill-space/datafile 2021-01-15T17:26:17.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:26:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj352 td/osd-backfill-space/datafile 2021-01-15T17:26:17.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:26:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj353 td/osd-backfill-space/datafile 2021-01-15T17:26:17.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:26:17.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj354 td/osd-backfill-space/datafile 2021-01-15T17:26:17.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:26:17.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj355 td/osd-backfill-space/datafile 2021-01-15T17:26:17.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:26:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj356 td/osd-backfill-space/datafile 2021-01-15T17:26:17.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:17.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:17.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:17.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:26:18.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj357 td/osd-backfill-space/datafile 2021-01-15T17:26:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:26:18.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj358 td/osd-backfill-space/datafile 2021-01-15T17:26:18.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:26:18.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj359 td/osd-backfill-space/datafile 2021-01-15T17:26:18.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:26:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj360 td/osd-backfill-space/datafile 2021-01-15T17:26:18.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:26:18.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj361 td/osd-backfill-space/datafile 2021-01-15T17:26:18.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:26:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj362 td/osd-backfill-space/datafile 2021-01-15T17:26:18.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:18.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:26:18.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:18.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj363 td/osd-backfill-space/datafile 2021-01-15T17:26:19.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:26:19.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj364 td/osd-backfill-space/datafile 2021-01-15T17:26:19.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:26:19.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj365 td/osd-backfill-space/datafile 2021-01-15T17:26:19.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:26:19.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj366 td/osd-backfill-space/datafile 2021-01-15T17:26:19.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:26:19.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj367 td/osd-backfill-space/datafile 2021-01-15T17:26:19.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:26:19.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj368 td/osd-backfill-space/datafile 2021-01-15T17:26:19.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:26:19.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj369 td/osd-backfill-space/datafile 2021-01-15T17:26:19.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:19.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:19.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:19.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:26:20.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj370 td/osd-backfill-space/datafile 2021-01-15T17:26:20.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:26:20.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj371 td/osd-backfill-space/datafile 2021-01-15T17:26:20.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:26:20.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj372 td/osd-backfill-space/datafile 2021-01-15T17:26:20.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:26:20.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj373 td/osd-backfill-space/datafile 2021-01-15T17:26:20.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:26:20.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj374 td/osd-backfill-space/datafile 2021-01-15T17:26:20.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:26:20.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj375 td/osd-backfill-space/datafile 2021-01-15T17:26:20.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:20.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:20.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:20.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:26:21.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj376 td/osd-backfill-space/datafile 2021-01-15T17:26:21.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:26:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj377 td/osd-backfill-space/datafile 2021-01-15T17:26:21.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:26:21.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj378 td/osd-backfill-space/datafile 2021-01-15T17:26:21.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:26:21.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj379 td/osd-backfill-space/datafile 2021-01-15T17:26:21.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:26:21.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj380 td/osd-backfill-space/datafile 2021-01-15T17:26:21.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:26:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj381 td/osd-backfill-space/datafile 2021-01-15T17:26:21.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:21.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:21.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:26:21.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:21.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj382 td/osd-backfill-space/datafile 2021-01-15T17:26:22.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:22.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:22.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:26:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj383 td/osd-backfill-space/datafile 2021-01-15T17:26:22.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:22.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:22.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:26:22.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj384 td/osd-backfill-space/datafile 2021-01-15T17:26:22.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:22.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:22.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:26:22.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj385 td/osd-backfill-space/datafile 2021-01-15T17:26:22.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:22.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:22.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:26:22.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj386 td/osd-backfill-space/datafile 2021-01-15T17:26:22.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:22.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:22.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:22.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:26:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj387 td/osd-backfill-space/datafile 2021-01-15T17:26:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:23.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:23.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:26:23.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj388 td/osd-backfill-space/datafile 2021-01-15T17:26:23.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:23.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:23.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:26:23.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj389 td/osd-backfill-space/datafile 2021-01-15T17:26:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:23.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:23.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:26:23.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj390 td/osd-backfill-space/datafile 2021-01-15T17:26:23.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:23.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:23.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:26:23.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj391 td/osd-backfill-space/datafile 2021-01-15T17:26:23.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:23.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:23.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:26:23.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:23.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj392 td/osd-backfill-space/datafile 2021-01-15T17:26:24.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:26:24.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj393 td/osd-backfill-space/datafile 2021-01-15T17:26:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:26:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj394 td/osd-backfill-space/datafile 2021-01-15T17:26:24.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:26:24.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj395 td/osd-backfill-space/datafile 2021-01-15T17:26:24.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:26:24.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj396 td/osd-backfill-space/datafile 2021-01-15T17:26:24.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:26:24.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj397 td/osd-backfill-space/datafile 2021-01-15T17:26:24.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:24.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:24.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:26:24.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:24.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj398 td/osd-backfill-space/datafile 2021-01-15T17:26:25.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:26:25.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj399 td/osd-backfill-space/datafile 2021-01-15T17:26:25.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:26:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj400 td/osd-backfill-space/datafile 2021-01-15T17:26:25.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:26:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj401 td/osd-backfill-space/datafile 2021-01-15T17:26:25.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:26:25.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj402 td/osd-backfill-space/datafile 2021-01-15T17:26:25.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:26:25.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj403 td/osd-backfill-space/datafile 2021-01-15T17:26:25.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:26:25.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj404 td/osd-backfill-space/datafile 2021-01-15T17:26:25.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:25.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:25.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:25.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:26:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj405 td/osd-backfill-space/datafile 2021-01-15T17:26:26.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:26:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj406 td/osd-backfill-space/datafile 2021-01-15T17:26:26.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:26:26.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj407 td/osd-backfill-space/datafile 2021-01-15T17:26:26.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:26:26.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj408 td/osd-backfill-space/datafile 2021-01-15T17:26:26.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:26:26.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj409 td/osd-backfill-space/datafile 2021-01-15T17:26:26.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:26:26.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj410 td/osd-backfill-space/datafile 2021-01-15T17:26:26.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:26.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:26.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:26.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:26:27.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj411 td/osd-backfill-space/datafile 2021-01-15T17:26:27.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:26:27.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj412 td/osd-backfill-space/datafile 2021-01-15T17:26:27.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:26:27.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj413 td/osd-backfill-space/datafile 2021-01-15T17:26:27.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:26:27.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj414 td/osd-backfill-space/datafile 2021-01-15T17:26:27.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:26:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj415 td/osd-backfill-space/datafile 2021-01-15T17:26:27.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:26:27.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj416 td/osd-backfill-space/datafile 2021-01-15T17:26:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:27.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:26:27.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:27.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj417 td/osd-backfill-space/datafile 2021-01-15T17:26:28.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:26:28.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj418 td/osd-backfill-space/datafile 2021-01-15T17:26:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:26:28.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj419 td/osd-backfill-space/datafile 2021-01-15T17:26:28.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:26:28.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj420 td/osd-backfill-space/datafile 2021-01-15T17:26:28.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:26:28.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj421 td/osd-backfill-space/datafile 2021-01-15T17:26:28.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:26:28.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj422 td/osd-backfill-space/datafile 2021-01-15T17:26:28.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:28.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:28.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:26:28.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:28.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj423 td/osd-backfill-space/datafile 2021-01-15T17:26:29.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:26:29.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj424 td/osd-backfill-space/datafile 2021-01-15T17:26:29.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:26:29.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj425 td/osd-backfill-space/datafile 2021-01-15T17:26:29.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:26:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj426 td/osd-backfill-space/datafile 2021-01-15T17:26:29.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:26:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj427 td/osd-backfill-space/datafile 2021-01-15T17:26:29.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:26:29.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj428 td/osd-backfill-space/datafile 2021-01-15T17:26:29.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:26:29.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj429 td/osd-backfill-space/datafile 2021-01-15T17:26:29.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:29.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:29.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:29.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:26:30.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj430 td/osd-backfill-space/datafile 2021-01-15T17:26:30.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:26:30.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj431 td/osd-backfill-space/datafile 2021-01-15T17:26:30.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:26:30.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj432 td/osd-backfill-space/datafile 2021-01-15T17:26:30.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:26:30.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj433 td/osd-backfill-space/datafile 2021-01-15T17:26:30.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:26:30.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj434 td/osd-backfill-space/datafile 2021-01-15T17:26:30.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:26:30.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj435 td/osd-backfill-space/datafile 2021-01-15T17:26:30.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:30.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:30.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:30.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:26:31.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj436 td/osd-backfill-space/datafile 2021-01-15T17:26:31.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:26:31.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj437 td/osd-backfill-space/datafile 2021-01-15T17:26:31.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:26:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj438 td/osd-backfill-space/datafile 2021-01-15T17:26:31.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:26:31.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj439 td/osd-backfill-space/datafile 2021-01-15T17:26:31.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:26:31.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj440 td/osd-backfill-space/datafile 2021-01-15T17:26:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:26:31.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj441 td/osd-backfill-space/datafile 2021-01-15T17:26:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:31.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:31.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:26:31.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:31.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj442 td/osd-backfill-space/datafile 2021-01-15T17:26:32.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:26:32.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj443 td/osd-backfill-space/datafile 2021-01-15T17:26:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:26:32.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj444 td/osd-backfill-space/datafile 2021-01-15T17:26:32.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:26:32.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj445 td/osd-backfill-space/datafile 2021-01-15T17:26:32.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:26:32.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj446 td/osd-backfill-space/datafile 2021-01-15T17:26:32.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:26:32.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj447 td/osd-backfill-space/datafile 2021-01-15T17:26:32.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:26:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj448 td/osd-backfill-space/datafile 2021-01-15T17:26:32.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:32.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:32.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:32.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:26:33.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj449 td/osd-backfill-space/datafile 2021-01-15T17:26:33.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:26:33.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj450 td/osd-backfill-space/datafile 2021-01-15T17:26:33.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:26:33.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj451 td/osd-backfill-space/datafile 2021-01-15T17:26:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:26:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj452 td/osd-backfill-space/datafile 2021-01-15T17:26:33.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:26:33.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj453 td/osd-backfill-space/datafile 2021-01-15T17:26:33.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:26:33.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj454 td/osd-backfill-space/datafile 2021-01-15T17:26:33.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:33.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:33.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:26:33.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:33.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj455 td/osd-backfill-space/datafile 2021-01-15T17:26:34.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:26:34.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj456 td/osd-backfill-space/datafile 2021-01-15T17:26:34.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:26:34.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj457 td/osd-backfill-space/datafile 2021-01-15T17:26:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:26:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj458 td/osd-backfill-space/datafile 2021-01-15T17:26:34.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:26:34.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj459 td/osd-backfill-space/datafile 2021-01-15T17:26:34.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:26:34.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj460 td/osd-backfill-space/datafile 2021-01-15T17:26:34.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:34.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:34.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:26:34.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:34.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj461 td/osd-backfill-space/datafile 2021-01-15T17:26:35.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:26:35.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj462 td/osd-backfill-space/datafile 2021-01-15T17:26:35.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:26:35.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj463 td/osd-backfill-space/datafile 2021-01-15T17:26:35.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:26:35.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj464 td/osd-backfill-space/datafile 2021-01-15T17:26:35.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:26:35.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj465 td/osd-backfill-space/datafile 2021-01-15T17:26:35.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:26:35.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj466 td/osd-backfill-space/datafile 2021-01-15T17:26:35.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:26:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj467 td/osd-backfill-space/datafile 2021-01-15T17:26:35.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:35.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:35.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:35.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:26:36.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj468 td/osd-backfill-space/datafile 2021-01-15T17:26:36.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:26:36.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj469 td/osd-backfill-space/datafile 2021-01-15T17:26:36.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:26:36.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj470 td/osd-backfill-space/datafile 2021-01-15T17:26:36.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:26:36.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj471 td/osd-backfill-space/datafile 2021-01-15T17:26:36.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:26:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj472 td/osd-backfill-space/datafile 2021-01-15T17:26:36.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:26:36.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj473 td/osd-backfill-space/datafile 2021-01-15T17:26:36.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:36.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:36.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:26:36.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:36.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj474 td/osd-backfill-space/datafile 2021-01-15T17:26:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:26:37.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj475 td/osd-backfill-space/datafile 2021-01-15T17:26:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:26:37.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj476 td/osd-backfill-space/datafile 2021-01-15T17:26:37.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:26:37.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj477 td/osd-backfill-space/datafile 2021-01-15T17:26:37.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:26:37.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj478 td/osd-backfill-space/datafile 2021-01-15T17:26:37.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:26:37.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj479 td/osd-backfill-space/datafile 2021-01-15T17:26:37.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:26:37.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj480 td/osd-backfill-space/datafile 2021-01-15T17:26:37.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:37.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:37.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:37.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:26:38.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj481 td/osd-backfill-space/datafile 2021-01-15T17:26:38.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:26:38.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj482 td/osd-backfill-space/datafile 2021-01-15T17:26:38.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:26:38.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj483 td/osd-backfill-space/datafile 2021-01-15T17:26:38.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:26:38.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj484 td/osd-backfill-space/datafile 2021-01-15T17:26:38.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:26:38.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj485 td/osd-backfill-space/datafile 2021-01-15T17:26:38.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:26:38.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj486 td/osd-backfill-space/datafile 2021-01-15T17:26:38.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:38.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:38.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:38.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:26:39.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj487 td/osd-backfill-space/datafile 2021-01-15T17:26:39.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:26:39.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj488 td/osd-backfill-space/datafile 2021-01-15T17:26:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:26:39.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj489 td/osd-backfill-space/datafile 2021-01-15T17:26:39.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:26:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj490 td/osd-backfill-space/datafile 2021-01-15T17:26:39.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:26:39.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj491 td/osd-backfill-space/datafile 2021-01-15T17:26:39.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:26:39.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj492 td/osd-backfill-space/datafile 2021-01-15T17:26:39.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:39.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:39.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:26:39.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:39.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj493 td/osd-backfill-space/datafile 2021-01-15T17:26:40.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:26:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj494 td/osd-backfill-space/datafile 2021-01-15T17:26:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:26:40.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj495 td/osd-backfill-space/datafile 2021-01-15T17:26:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:26:40.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj496 td/osd-backfill-space/datafile 2021-01-15T17:26:40.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:26:40.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj497 td/osd-backfill-space/datafile 2021-01-15T17:26:40.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:26:40.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj498 td/osd-backfill-space/datafile 2021-01-15T17:26:40.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:40.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:40.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:26:40.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:40.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj499 td/osd-backfill-space/datafile 2021-01-15T17:26:41.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:26:41.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj500 td/osd-backfill-space/datafile 2021-01-15T17:26:41.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:26:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj501 td/osd-backfill-space/datafile 2021-01-15T17:26:41.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:26:41.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj502 td/osd-backfill-space/datafile 2021-01-15T17:26:41.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:26:41.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj503 td/osd-backfill-space/datafile 2021-01-15T17:26:41.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:26:41.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj504 td/osd-backfill-space/datafile 2021-01-15T17:26:41.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:26:41.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj505 td/osd-backfill-space/datafile 2021-01-15T17:26:41.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:41.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:41.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:41.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:26:42.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj506 td/osd-backfill-space/datafile 2021-01-15T17:26:42.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:26:42.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj507 td/osd-backfill-space/datafile 2021-01-15T17:26:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:26:42.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj508 td/osd-backfill-space/datafile 2021-01-15T17:26:42.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:26:42.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj509 td/osd-backfill-space/datafile 2021-01-15T17:26:42.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:26:42.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj510 td/osd-backfill-space/datafile 2021-01-15T17:26:42.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:26:42.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj511 td/osd-backfill-space/datafile 2021-01-15T17:26:42.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:42.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:26:43.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj512 td/osd-backfill-space/datafile 2021-01-15T17:26:43.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:26:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj513 td/osd-backfill-space/datafile 2021-01-15T17:26:43.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:26:43.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj514 td/osd-backfill-space/datafile 2021-01-15T17:26:43.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:26:43.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj515 td/osd-backfill-space/datafile 2021-01-15T17:26:43.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:26:43.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj516 td/osd-backfill-space/datafile 2021-01-15T17:26:43.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:26:43.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj517 td/osd-backfill-space/datafile 2021-01-15T17:26:43.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:43.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:43.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:26:43.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:43.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj518 td/osd-backfill-space/datafile 2021-01-15T17:26:44.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:26:44.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj519 td/osd-backfill-space/datafile 2021-01-15T17:26:44.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:26:44.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj520 td/osd-backfill-space/datafile 2021-01-15T17:26:44.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:26:44.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj521 td/osd-backfill-space/datafile 2021-01-15T17:26:44.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:26:44.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj522 td/osd-backfill-space/datafile 2021-01-15T17:26:44.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:26:44.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj523 td/osd-backfill-space/datafile 2021-01-15T17:26:44.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:44.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:26:44.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:44.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj524 td/osd-backfill-space/datafile 2021-01-15T17:26:45.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:26:45.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj525 td/osd-backfill-space/datafile 2021-01-15T17:26:45.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:26:45.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj526 td/osd-backfill-space/datafile 2021-01-15T17:26:45.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:26:45.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj527 td/osd-backfill-space/datafile 2021-01-15T17:26:45.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:26:45.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj528 td/osd-backfill-space/datafile 2021-01-15T17:26:45.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:26:45.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj529 td/osd-backfill-space/datafile 2021-01-15T17:26:45.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:26:45.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj530 td/osd-backfill-space/datafile 2021-01-15T17:26:45.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:45.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:45.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:45.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:26:46.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj531 td/osd-backfill-space/datafile 2021-01-15T17:26:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:26:46.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj532 td/osd-backfill-space/datafile 2021-01-15T17:26:46.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:26:46.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj533 td/osd-backfill-space/datafile 2021-01-15T17:26:46.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:26:46.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj534 td/osd-backfill-space/datafile 2021-01-15T17:26:46.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:26:46.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj535 td/osd-backfill-space/datafile 2021-01-15T17:26:46.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:26:46.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj536 td/osd-backfill-space/datafile 2021-01-15T17:26:46.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:46.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:46.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:46.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:26:47.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj537 td/osd-backfill-space/datafile 2021-01-15T17:26:47.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:26:47.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj538 td/osd-backfill-space/datafile 2021-01-15T17:26:47.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:26:47.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj539 td/osd-backfill-space/datafile 2021-01-15T17:26:47.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:26:47.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj540 td/osd-backfill-space/datafile 2021-01-15T17:26:47.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:26:47.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj541 td/osd-backfill-space/datafile 2021-01-15T17:26:47.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:26:47.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj542 td/osd-backfill-space/datafile 2021-01-15T17:26:47.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:47.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:47.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:26:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj543 td/osd-backfill-space/datafile 2021-01-15T17:26:48.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:26:48.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj544 td/osd-backfill-space/datafile 2021-01-15T17:26:48.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:26:48.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj545 td/osd-backfill-space/datafile 2021-01-15T17:26:48.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:26:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj546 td/osd-backfill-space/datafile 2021-01-15T17:26:48.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:26:48.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj547 td/osd-backfill-space/datafile 2021-01-15T17:26:48.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:26:48.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj548 td/osd-backfill-space/datafile 2021-01-15T17:26:48.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:26:48.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj549 td/osd-backfill-space/datafile 2021-01-15T17:26:48.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:48.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:48.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:26:49.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj550 td/osd-backfill-space/datafile 2021-01-15T17:26:49.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:26:49.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj551 td/osd-backfill-space/datafile 2021-01-15T17:26:49.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:26:49.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj552 td/osd-backfill-space/datafile 2021-01-15T17:26:49.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:26:49.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj553 td/osd-backfill-space/datafile 2021-01-15T17:26:49.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:26:49.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj554 td/osd-backfill-space/datafile 2021-01-15T17:26:49.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:26:49.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj555 td/osd-backfill-space/datafile 2021-01-15T17:26:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:49.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:49.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:26:49.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:49.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj556 td/osd-backfill-space/datafile 2021-01-15T17:26:50.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:26:50.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj557 td/osd-backfill-space/datafile 2021-01-15T17:26:50.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:26:50.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj558 td/osd-backfill-space/datafile 2021-01-15T17:26:50.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:26:50.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj559 td/osd-backfill-space/datafile 2021-01-15T17:26:50.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:26:50.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj560 td/osd-backfill-space/datafile 2021-01-15T17:26:50.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:26:50.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj561 td/osd-backfill-space/datafile 2021-01-15T17:26:50.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:50.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:50.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:26:50.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:50.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj562 td/osd-backfill-space/datafile 2021-01-15T17:26:51.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:26:51.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj563 td/osd-backfill-space/datafile 2021-01-15T17:26:51.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:26:51.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj564 td/osd-backfill-space/datafile 2021-01-15T17:26:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:26:51.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj565 td/osd-backfill-space/datafile 2021-01-15T17:26:51.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:26:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj566 td/osd-backfill-space/datafile 2021-01-15T17:26:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:26:51.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj567 td/osd-backfill-space/datafile 2021-01-15T17:26:51.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:51.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:51.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:26:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj568 td/osd-backfill-space/datafile 2021-01-15T17:26:52.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:26:52.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj569 td/osd-backfill-space/datafile 2021-01-15T17:26:52.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:26:52.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj570 td/osd-backfill-space/datafile 2021-01-15T17:26:52.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:26:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj571 td/osd-backfill-space/datafile 2021-01-15T17:26:52.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:26:52.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj572 td/osd-backfill-space/datafile 2021-01-15T17:26:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:26:52.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj573 td/osd-backfill-space/datafile 2021-01-15T17:26:52.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:26:52.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj574 td/osd-backfill-space/datafile 2021-01-15T17:26:52.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:52.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:52.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:52.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:26:53.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj575 td/osd-backfill-space/datafile 2021-01-15T17:26:53.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:26:53.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj576 td/osd-backfill-space/datafile 2021-01-15T17:26:53.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:26:53.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj577 td/osd-backfill-space/datafile 2021-01-15T17:26:53.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:26:53.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj578 td/osd-backfill-space/datafile 2021-01-15T17:26:53.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:26:53.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj579 td/osd-backfill-space/datafile 2021-01-15T17:26:53.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:26:53.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj580 td/osd-backfill-space/datafile 2021-01-15T17:26:53.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:53.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:53.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:53.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:26:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj581 td/osd-backfill-space/datafile 2021-01-15T17:26:54.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:54.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:54.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:26:54.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj582 td/osd-backfill-space/datafile 2021-01-15T17:26:54.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:54.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:54.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:26:54.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj583 td/osd-backfill-space/datafile 2021-01-15T17:26:54.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:54.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:54.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:26:54.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:54.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj584 td/osd-backfill-space/datafile 2021-01-15T17:26:55.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:26:55.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj585 td/osd-backfill-space/datafile 2021-01-15T17:26:55.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:26:55.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj586 td/osd-backfill-space/datafile 2021-01-15T17:26:55.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:26:55.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj587 td/osd-backfill-space/datafile 2021-01-15T17:26:55.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:26:55.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj588 td/osd-backfill-space/datafile 2021-01-15T17:26:55.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:26:55.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj589 td/osd-backfill-space/datafile 2021-01-15T17:26:55.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:26:55.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj590 td/osd-backfill-space/datafile 2021-01-15T17:26:55.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:55.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:55.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:55.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:26:56.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj591 td/osd-backfill-space/datafile 2021-01-15T17:26:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:56.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:56.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:26:56.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj592 td/osd-backfill-space/datafile 2021-01-15T17:26:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:56.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:26:56.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj593 td/osd-backfill-space/datafile 2021-01-15T17:26:56.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:56.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:56.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:26:56.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj594 td/osd-backfill-space/datafile 2021-01-15T17:26:56.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:56.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:26:56.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj595 td/osd-backfill-space/datafile 2021-01-15T17:26:56.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:56.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:56.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:26:56.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:56.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj596 td/osd-backfill-space/datafile 2021-01-15T17:26:57.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:57.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:57.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:26:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj597 td/osd-backfill-space/datafile 2021-01-15T17:26:57.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:57.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:57.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:26:57.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj598 td/osd-backfill-space/datafile 2021-01-15T17:26:57.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:57.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:57.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:26:57.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj599 td/osd-backfill-space/datafile 2021-01-15T17:26:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:136: TEST_backfill_test_simple: for o in $(seq 1 $objects) 2021-01-15T17:26:57.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test1 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:26:57.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:138: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:140: TEST_backfill_test_simple: rados -p test2 put obj600 td/osd-backfill-space/datafile 2021-01-15T17:26:57.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:144: TEST_backfill_test_simple: ceph pg dump pgs 2021-01-15T17:26:57.980 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:26:57.981 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 589 0 0 0 0 2412544 0 0 89 89 active+clean 2021-01-15T17:25:16.174163+0000 27'589 27:606 [2] 2 [2] 2 0'0 2021-01-15T17:25:13.869474+0000 0'0 2021-01-15T17:25:13.869474+0000 0 2021-01-15T17:26:57.981 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 573 0 0 0 0 2347008 0 0 73 73 active+clean 2021-01-15T17:25:13.619441+0000 27'573 27:595 [1] 1 [1] 1 0'0 2021-01-15T17:25:11.501858+0000 0'0 2021-01-15T17:25:11.501858+0000 0 2021-01-15T17:26:57.982 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:26:57.982 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:26:57.983 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:26:57.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:26:57.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:57.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:148: TEST_backfill_test_simple: ceph osd pool set test1 size 2 2021-01-15T17:26:58.673 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:26:58.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:146: TEST_backfill_test_simple: for p in $(seq 1 $pools) 2021-01-15T17:26:58.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:148: TEST_backfill_test_simple: ceph osd pool set test2 size 2 2021-01-15T17:26:59.761 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T17:26:59.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:150: TEST_backfill_test_simple: sleep 30 2021-01-15T17:27:29.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:152: TEST_backfill_test_simple: wait_for_not_backfilling 240 2021-01-15T17:27:29.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:27:29.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:27:29.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:27:29.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:27:29.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:27:29.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:27:29.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:27:29.776 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:27:29.776 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:27:29.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:27:29.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:27:29.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:27:29.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:27:29.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:27:29.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:27:29.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:27:29.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:27:30.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:27:30.310 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:27:30.310 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:27:30.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:27:30.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:27:30.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:27:30.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836516 2021-01-15T17:27:30.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836516 2021-01-15T17:27:30.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516' 2021-01-15T17:27:30.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:27:30.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:27:30.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672994 2021-01-15T17:27:30.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672994 2021-01-15T17:27:30.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-42949672994' 2021-01-15T17:27:30.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:27:30.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:27:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509472 2021-01-15T17:27:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509472 2021-01-15T17:27:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836516 1-42949672994 2-64424509472' 2021-01-15T17:27:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:27:30.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836516 2021-01-15T17:27:30.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:27:30.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:27:30.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836516 2021-01-15T17:27:30.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:27:30.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836516 2021-01-15T17:27:30.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2021-01-15T17:27:30.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:27:30.735 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836516 2021-01-15T17:27:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836516 -lt 21474836516 2021-01-15T17:27:31.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:27:31.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672994 2021-01-15T17:27:31.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:27:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:27:31.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672994 2021-01-15T17:27:31.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:27:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672994 2021-01-15T17:27:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2021-01-15T17:27:31.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:27:31.135 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672994 2021-01-15T17:27:31.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672992 -lt 42949672994 2021-01-15T17:27:31.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:27:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:27:32.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:27:32.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672994 -lt 42949672994 2021-01-15T17:27:32.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:27:32.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509472 2021-01-15T17:27:32.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:27:32.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:27:32.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509472 2021-01-15T17:27:32.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:27:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509472 2021-01-15T17:27:32.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2021-01-15T17:27:32.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:27:32.938 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509472 2021-01-15T17:27:33.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509473 -lt 64424509472 2021-01-15T17:27:33.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:27:33.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:27:33.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:27:33.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:27:33.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:27:33.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:27:33.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:27:33.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:27:33.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:27:33.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:27:33.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:27:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:27:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:27:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2021-01-15T17:27:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:27:34.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2021-01-15T17:27:34.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:27:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:27:39.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:27:39.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:27:39.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:27:39.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:27:39.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:27:39.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:27:39.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:27:39.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:27:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:27:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:27:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2021-01-15T17:27:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2021-01-15T17:27:49.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:27:49.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:27:49.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:27:49.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:27:49.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:27:49.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:27:49.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:27:49.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:27:50.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:27:50.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:27:50.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:27:50.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 2 >= 17 )) 2021-01-15T17:27:50.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:28:05.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:28:05.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:28:05.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:28:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:28:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:28:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:28:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:28:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:28:05.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:28:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:28:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:28:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 3 >= 17 )) 2021-01-15T17:28:05.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:28:20.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:28:20.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:28:20.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:28:20.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:28:20.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:28:20.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:28:20.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:28:20.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:28:20.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:28:20.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:28:20.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:28:20.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 4 >= 17 )) 2021-01-15T17:28:20.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:28:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:28:35.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:28:35.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:28:35.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:28:35.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:28:36.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:28:36.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:28:36.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:28:36.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 5 >= 17 )) 2021-01-15T17:28:36.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 15 2021-01-15T17:28:51.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:28:51.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:28:51.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:28:51.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:28:51.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:28:51.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:28:51.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:28:51.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:28:51.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:28:51.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:28:51.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:28:51.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:28:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:153: TEST_backfill_test_simple: wait_for_not_activating 60 2021-01-15T17:28:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:28:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:28:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:28:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:28:51.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:28:51.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:28:51.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:28:51.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:28:51.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:28:51.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:28:51.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:28:51.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:28:51.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:28:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:28:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:28:51.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:28:51.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:28:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:28:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:28:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:28:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:28:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:28:52.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:28:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836534 2021-01-15T17:28:52.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836534 2021-01-15T17:28:52.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534' 2021-01-15T17:28:52.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:28:52.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:28:52.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673012 2021-01-15T17:28:52.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673012 2021-01-15T17:28:52.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2021-01-15T17:28:52.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:28:52.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:28:52.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509491 2021-01-15T17:28:52.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509491 2021-01-15T17:28:52.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509491' 2021-01-15T17:28:52.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:28:52.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836534 2021-01-15T17:28:52.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:28:52.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:28:52.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836534 2021-01-15T17:28:52.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:28:52.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836534 2021-01-15T17:28:52.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2021-01-15T17:28:52.528 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836534 2021-01-15T17:28:52.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:28:52.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836534 -lt 21474836534 2021-01-15T17:28:52.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:28:52.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673012 2021-01-15T17:28:52.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:28:52.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:28:52.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673012 2021-01-15T17:28:52.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:28:52.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673012 2021-01-15T17:28:52.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2021-01-15T17:28:52.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:28:52.943 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673012 2021-01-15T17:28:53.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673012 -lt 42949673012 2021-01-15T17:28:53.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:28:53.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509491 2021-01-15T17:28:53.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:28:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:28:53.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509491 2021-01-15T17:28:53.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:28:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509491 2021-01-15T17:28:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2021-01-15T17:28:53.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:28:53.339 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509491 2021-01-15T17:28:53.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509490 -lt 64424509491 2021-01-15T17:28:53.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:28:54.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:28:54.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:28:55.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509491 -lt 64424509491 2021-01-15T17:28:55.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:28:55.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:28:55.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:28:55.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:28:55.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:28:55.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:28:55.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:28:55.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:28:55.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:28:55.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:28:55.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:28:56.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:28:56.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:28:56.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:28:56.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:28:56.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:155: TEST_backfill_test_simple: ERRORS=0 2021-01-15T17:28:56.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: ceph pg dump pgs 2021-01-15T17:28:56.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: grep +backfill_toofull 2021-01-15T17:28:56.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: wc -l 2021-01-15T17:28:56.433 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:28:56.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:156: TEST_backfill_test_simple: '[' 1 '!=' 1 ']' 2021-01-15T17:28:56.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:162: TEST_backfill_test_simple: expr 2 - 1 2021-01-15T17:28:56.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:162: TEST_backfill_test_simple: expected=1 2021-01-15T17:28:56.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: ceph pg dump pgs 2021-01-15T17:28:56.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: grep active+clean 2021-01-15T17:28:56.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: wc -l 2021-01-15T17:28:56.822 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:28:56.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:163: TEST_backfill_test_simple: '[' 1 '!=' 1 ']' 2021-01-15T17:28:56.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:169: TEST_backfill_test_simple: ceph pg dump pgs 2021-01-15T17:28:57.200 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:28:57.200 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 600 0 600 0 0 2457600 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:28:30.441219+0000 27'600 34:646 [2,1] 2 [2] 2 0'0 2021-01-15T17:25:13.869474+0000 0'0 2021-01-15T17:25:13.869474+0000 0 2021-01-15T17:28:57.200 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:28:37.879196+0000 27'600 35:1849 [1,0] 1 [1,0] 1 0'0 2021-01-15T17:25:11.501858+0000 0'0 2021-01-15T17:25:11.501858+0000 0 2021-01-15T17:28:57.200 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:28:57.201 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:28:57.201 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:28:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:171: TEST_backfill_test_simple: '[' 0 '!=' 0 ']' 2021-01-15T17:28:57.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: seq 1 2 2021-01-15T17:28:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: for i in $(seq 1 $pools) 2021-01-15T17:28:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:178: TEST_backfill_test_simple: delete_pool test1 2021-01-15T17:28:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:28:57.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:28:57.770 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:28:57.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:176: TEST_backfill_test_simple: for i in $(seq 1 $pools) 2021-01-15T17:28:57.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:178: TEST_backfill_test_simple: delete_pool test2 2021-01-15T17:28:57.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T17:28:57.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T17:28:58.299 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T17:28:58.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:180: TEST_backfill_test_simple: kill_daemons td/osd-backfill-space 2021-01-15T17:28:58.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:28:58.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:28:58.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:28:58.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:28:58.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:28:58.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:28:58.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:181: TEST_backfill_test_simple: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2021-01-15T17:28:58.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:28:58.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:28:58.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:28:58.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:28:58.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:28:58.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:28:58.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:28:58.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:28:58.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:28:58.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:28:58.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:28:58.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:28:58.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:28:58.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:28:58.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:28:58.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:28:58.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:28:58.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:28:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:28:58.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:28:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:28:58.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:28:58.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:28:58.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:28:58.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:58.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:58.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:28:58.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:28:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:28:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:28:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:28:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:28:58.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:28:58.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:28:58.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:28:58.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:28:58.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:28:58.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:28:58.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:28:58.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:28:58.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:28:58.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:28:58.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:28:58.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:28:58.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:28:58.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:28:58.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:28:58.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:28:58.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:28:58.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:28:58.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:28:58.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:28:58.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:28:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:28:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:28:58.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:28:58.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:58.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:58.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:28:58.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:28:58.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:28:58.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:28:58.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:28:58.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:58.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:58.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:28:58.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:28:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:28:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:28:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:28:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_grow td/osd-backfill-space 2021-01-15T17:28:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1110: TEST_ec_backfill_grow: local dir=td/osd-backfill-space 2021-01-15T17:28:58.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1111: TEST_ec_backfill_grow: local poolname=test 2021-01-15T17:28:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1112: TEST_ec_backfill_grow: local OSDS=6 2021-01-15T17:28:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1113: TEST_ec_backfill_grow: local k=3 2021-01-15T17:28:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1114: TEST_ec_backfill_grow: local m=2 2021-01-15T17:28:58.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1115: TEST_ec_backfill_grow: expr 600 / 3 2021-01-15T17:28:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1115: TEST_ec_backfill_grow: local ecobjects=200 2021-01-15T17:28:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1117: TEST_ec_backfill_grow: run_mon td/osd-backfill-space a 2021-01-15T17:28:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:28:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:28:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:28:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:28:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:28:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:28:58.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:28:58.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:28:58.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:28:58.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:28:58.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:58.802 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:58.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:28:58.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:28:58.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:28:58.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:28:58.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:28:58.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:28:58.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:28:58.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:28:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:28:58.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:28:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:28:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:28:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:58.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:28:58.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:28:58.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:28:59.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:28:59.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:28:59.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:28:59.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:28:59.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:28:59.088 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:28:59.088 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:28:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:28:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:28:59.090 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:28:59.090 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:28:59.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:28:59.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:28:59.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:28:59.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1118: TEST_ec_backfill_grow: run_mgr td/osd-backfill-space x 2021-01-15T17:28:59.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:28:59.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:28:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:28:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:28:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:28:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:29:00.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:29:00.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:00.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:00.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:00.097 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:00.097 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:00.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:00.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:29:00.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:29:00.122 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: expr 6 - 1 2021-01-15T17:29:00.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: seq 0 5 2021-01-15T17:29:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 0 2021-01-15T17:29:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:29:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:29:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:29:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:29:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:00.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:00.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:00.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:00.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:00.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:29:00.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:00.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c68b2430-6075-4746-94b9-eb9fdac58a38 2021-01-15T17:29:00.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c68b2430-6075-4746-94b9-eb9fdac58a38' 2021-01-15T17:29:00.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:00.135 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 c68b2430-6075-4746-94b9-eb9fdac58a38 2021-01-15T17:29:00.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDc0AFg3nCUCBAAXrUz4Q2qbhZnDlZ1Xzbg3w== 2021-01-15T17:29:00.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDc0AFg3nCUCBAAXrUz4Q2qbhZnDlZ1Xzbg3w=="}' 2021-01-15T17:29:00.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c68b2430-6075-4746-94b9-eb9fdac58a38 -i td/osd-backfill-space/0/new.json 2021-01-15T17:29:00.503 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:29:00.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDc0AFg3nCUCBAAXrUz4Q2qbhZnDlZ1Xzbg3w== --osd-uuid c68b2430-6075-4746-94b9-eb9fdac58a38 2021-01-15T17:29:00.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.541+0000 7fcbfc25d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:00.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.543+0000 7fcbfc25d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:00.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.544+0000 7fcbfc25d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.545+0000 7fcbfc25d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:29:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.545+0000 7fcbfc25d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:29:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.545+0000 7fcbfc25d080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:29:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:00.608+0000 7fcbfc25d080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:29:05.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:29:05.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:05.011 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:29:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:29:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:05.535 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:29:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:29:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:05.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:05.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:05.560 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:05.559+0000 7f6094f54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:05.562 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:05.561+0000 7f6094f54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:05.564 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:05.563+0000 7f6094f54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:05.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:29:05.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:05.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:29:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:05.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:29:05.938 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:06.091 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:06.091+0000 7f6094f54080 -1 Falling back to public interface 2021-01-15T17:29:06.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:06.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:06.875+0000 7f6094f54080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:29:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:07.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:07.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:07.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:29:07.328 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:07.735 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/457983,v1:127.0.0.1:6803/457983] [v2:127.0.0.1:6804/457983,v1:127.0.0.1:6805/457983] exists,up c68b2430-6075-4746-94b9-eb9fdac58a38 2021-01-15T17:29:07.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 1 2021-01-15T17:29:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:29:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:29:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:07.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:29:07.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:29:07.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:07.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:07.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:07.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:07.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:07.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:07.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:07.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:07.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:07.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:07.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:07.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:07.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:07.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:07.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:07.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:07.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:07.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:07.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:29:07.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:07.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=359183b8-9636-4390-92da-9fcd6ef01937 2021-01-15T17:29:07.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 359183b8-9636-4390-92da-9fcd6ef01937' 2021-01-15T17:29:07.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:07.746 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 359183b8-9636-4390-92da-9fcd6ef01937 2021-01-15T17:29:07.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDj0AFgkc3ZLBAAMUq95XwZYYs3s6SwAD3tng== 2021-01-15T17:29:07.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDj0AFgkc3ZLBAAMUq95XwZYYs3s6SwAD3tng=="}' 2021-01-15T17:29:07.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 359183b8-9636-4390-92da-9fcd6ef01937 -i td/osd-backfill-space/1/new.json 2021-01-15T17:29:08.267 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:08.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:29:08.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDj0AFgkc3ZLBAAMUq95XwZYYs3s6SwAD3tng== --osd-uuid 359183b8-9636-4390-92da-9fcd6ef01937 2021-01-15T17:29:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.305+0000 7f3504fe0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.307+0000 7f3504fe0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.309+0000 7f3504fe0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.309+0000 7f3504fe0080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:29:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.309+0000 7f3504fe0080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:29:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.309+0000 7f3504fe0080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:29:08.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:08.358+0000 7f3504fe0080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:29:12.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:29:12.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:12.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:29:12.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:12.890 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:29:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:29:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:13.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:13.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:13.417 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:29:13.442 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:13.440+0000 7f829347d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:13.442 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:13.442+0000 7f829347d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:13.444 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:13.444+0000 7f829347d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:13.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:29:13.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:13.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:29:13.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:13.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:13.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:13.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:13.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:13.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:29:13.811 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:13.963 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:13.963+0000 7f829347d080 -1 Falling back to public interface 2021-01-15T17:29:14.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:14.755 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:14.754+0000 7f829347d080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:29:15.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:15.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:29:15.222 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:15.639 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/459247,v1:127.0.0.1:6811/459247] [v2:127.0.0.1:6812/459247,v1:127.0.0.1:6813/459247] exists,up 359183b8-9636-4390-92da-9fcd6ef01937 2021-01-15T17:29:15.640 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 2cb0455f-a817-420f-9876-f69d7c21fae5 2021-01-15T17:29:15.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:15.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:15.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:15.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:15.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 2 2021-01-15T17:29:15.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:15.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:15.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:29:15.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:15.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:29:15.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:29:15.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:29:15.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:15.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:15.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:15.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:15.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:15.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:15.645 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:15.645 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:15.645 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:15.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:15.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:15.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:15.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:15.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:15.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:15.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:15.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:15.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:15.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:15.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:15.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2cb0455f-a817-420f-9876-f69d7c21fae5 2021-01-15T17:29:15.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 2cb0455f-a817-420f-9876-f69d7c21fae5' 2021-01-15T17:29:15.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:15.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDr0AFglscgJxAAhG7aysT3Bt/re/aPT/Am9A== 2021-01-15T17:29:15.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDr0AFglscgJxAAhG7aysT3Bt/re/aPT/Am9A=="}' 2021-01-15T17:29:15.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2cb0455f-a817-420f-9876-f69d7c21fae5 -i td/osd-backfill-space/2/new.json 2021-01-15T17:29:16.190 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:29:16.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:29:16.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDr0AFglscgJxAAhG7aysT3Bt/re/aPT/Am9A== --osd-uuid 2cb0455f-a817-420f-9876-f69d7c21fae5 2021-01-15T17:29:16.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.229+0000 7fcfce803080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:16.231 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.231+0000 7fcfce803080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.232+0000 7fcfce803080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.233+0000 7fcfce803080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:29:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.233+0000 7fcfce803080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:29:16.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.233+0000 7fcfce803080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:29:16.277 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:16.277+0000 7fcfce803080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:29:20.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:29:20.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:20.789 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:29:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:29:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:21.318 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:29:21.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:29:21.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:21.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:21.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:21.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:21.343 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:21.343+0000 7f9e201b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:21.345 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:21.344+0000 7f9e201b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:21.346 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:21.346+0000 7f9e201b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:21.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:29:21.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:21.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:29:21.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:21.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:21.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:21.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:21.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:21.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:29:21.714 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:21.869 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:21.869+0000 7f9e201b5080 -1 Falling back to public interface 2021-01-15T17:29:22.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:22.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:22.668+0000 7f9e201b5080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:29:23.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:23.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:23.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:23.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:23.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:29:23.146 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:23.555 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/460511,v1:127.0.0.1:6819/460511] [v2:127.0.0.1:6820/460511,v1:127.0.0.1:6821/460511] exists,up 2cb0455f-a817-420f-9876-f69d7c21fae5 2021-01-15T17:29:23.555 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 c815fa48-cec1-49f5-b00e-adba7f41d473 2021-01-15T17:29:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 3 2021-01-15T17:29:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:29:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2021-01-15T17:29:23.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:23.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:23.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2021-01-15T17:29:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2021-01-15T17:29:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:23.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:23.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:23.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:23.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:23.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:23.561 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:23.561 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:23.562 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:23.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:23.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:23.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:23.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:23.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:23.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:23.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2021-01-15T17:29:23.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c815fa48-cec1-49f5-b00e-adba7f41d473 2021-01-15T17:29:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 c815fa48-cec1-49f5-b00e-adba7f41d473' 2021-01-15T17:29:23.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:23.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDz0AFgUpkTIhAABo5rGDChQKVJY8SoPbkB0w== 2021-01-15T17:29:23.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDz0AFgUpkTIhAABo5rGDChQKVJY8SoPbkB0w=="}' 2021-01-15T17:29:23.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c815fa48-cec1-49f5-b00e-adba7f41d473 -i td/osd-backfill-space/3/new.json 2021-01-15T17:29:24.150 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:29:24.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2021-01-15T17:29:24.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDz0AFgUpkTIhAABo5rGDChQKVJY8SoPbkB0w== --osd-uuid c815fa48-cec1-49f5-b00e-adba7f41d473 2021-01-15T17:29:24.192 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.190+0000 7f012afcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:24.215 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.192+0000 7f012afcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:24.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.193+0000 7f012afcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:24.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.194+0000 7f012afcf080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:29:24.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.194+0000 7f012afcf080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:29:24.217 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.194+0000 7f012afcf080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:29:24.266 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:24.266+0000 7f012afcf080 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2021-01-15T17:29:28.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2021-01-15T17:29:28.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:28.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:29:28.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:28.802 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:29:29.330 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:29:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:29:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:29.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:29.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:29.356 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:29.355+0000 7f0ea7d9a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:29.357 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:29.357+0000 7f0ea7d9a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:29.359 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:29.358+0000 7f0ea7d9a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:29.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:29:29.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:29.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:29:29.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:29.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:29.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:29.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:29.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:29.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:29:29.735 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:29.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:29.878+0000 7f0ea7d9a080 -1 Falling back to public interface 2021-01-15T17:29:30.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:30.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:30.692+0000 7f0ea7d9a080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:29:31.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:29:31.145 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:31.580 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/461775,v1:127.0.0.1:6827/461775] [v2:127.0.0.1:6828/461775,v1:127.0.0.1:6829/461775] exists,up c815fa48-cec1-49f5-b00e-adba7f41d473 2021-01-15T17:29:31.580 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 c7fb047d-6bb9-4e2c-b170-c7d361bef95c 2021-01-15T17:29:31.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:31.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:31.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:31.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:31.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 4 2021-01-15T17:29:31.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:31.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:31.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:29:31.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:31.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2021-01-15T17:29:31.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:31.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:31.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:31.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:31.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2021-01-15T17:29:31.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2021-01-15T17:29:31.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:31.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:31.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:31.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:31.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:31.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:31.590 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:31.590 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:31.590 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:31.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:31.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:31.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:31.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:31.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:31.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:31.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2021-01-15T17:29:31.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:31.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c7fb047d-6bb9-4e2c-b170-c7d361bef95c 2021-01-15T17:29:31.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 c7fb047d-6bb9-4e2c-b170-c7d361bef95c' 2021-01-15T17:29:31.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:31.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD70AFgDj9VIxAAKlvvPGQ1PCOsuGiYzfmMnA== 2021-01-15T17:29:31.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD70AFgDj9VIxAAKlvvPGQ1PCOsuGiYzfmMnA=="}' 2021-01-15T17:29:31.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c7fb047d-6bb9-4e2c-b170-c7d361bef95c -i td/osd-backfill-space/4/new.json 2021-01-15T17:29:32.105 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:29:32.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2021-01-15T17:29:32.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD70AFgDj9VIxAAKlvvPGQ1PCOsuGiYzfmMnA== --osd-uuid c7fb047d-6bb9-4e2c-b170-c7d361bef95c 2021-01-15T17:29:32.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.145+0000 7fcfe0a33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:32.147 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.147+0000 7fcfe0a33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:32.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.149+0000 7fcfe0a33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:32.150 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.149+0000 7fcfe0a33080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:29:32.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.149+0000 7fcfe0a33080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:29:32.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.149+0000 7fcfe0a33080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:29:32.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:32.192+0000 7fcfe0a33080 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2021-01-15T17:29:37.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2021-01-15T17:29:37.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:37.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:29:37.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:37.004 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:29:37.535 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:29:37.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:29:37.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:37.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:37.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:37.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:37.561 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:37.559+0000 7f2f73b93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:37.600 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:37.598+0000 7f2f73b93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:37.600 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:37.599+0000 7f2f73b93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:37.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:29:37.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:29:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:37.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:37.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:37.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:29:37.956 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:38.115+0000 7f2f73b93080 -1 Falling back to public interface 2021-01-15T17:29:38.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:38.917 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:38.917+0000 7f2f73b93080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:29:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:39.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:39.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:39.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:29:39.357 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:39.769 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/463047,v1:127.0.0.1:6835/463047] [v2:127.0.0.1:6836/463047,v1:127.0.0.1:6837/463047] exists,up c7fb047d-6bb9-4e2c-b170-c7d361bef95c 2021-01-15T17:29:39.769 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 9d1f2223-7a3c-4cf4-9432-1669331288d5 2021-01-15T17:29:39.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:39.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:39.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:39.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1120: TEST_ec_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:29:39.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1122: TEST_ec_backfill_grow: run_osd td/osd-backfill-space 5 2021-01-15T17:29:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:29:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:29:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:29:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:29:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2021-01-15T17:29:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:29:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:29:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:29:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:29:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2021-01-15T17:29:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2021-01-15T17:29:39.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:29:39.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:29:39.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:29:39.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:29:39.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:29:39.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:29:39.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:29:39.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:29:39.779 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:29:39.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:39.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:29:39.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:29:39.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:29:39.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:29:39.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:29:39.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:29:39.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:29:39.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:29:39.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:39.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:29:39.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:29:39.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2021-01-15T17:29:39.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:29:39.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9d1f2223-7a3c-4cf4-9432-1669331288d5 2021-01-15T17:29:39.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 9d1f2223-7a3c-4cf4-9432-1669331288d5' 2021-01-15T17:29:39.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:29:39.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAD0QFg0TawLhAAv6nOWgkPBHKEM+8K7aLDDQ== 2021-01-15T17:29:39.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAD0QFg0TawLhAAv6nOWgkPBHKEM+8K7aLDDQ=="}' 2021-01-15T17:29:39.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9d1f2223-7a3c-4cf4-9432-1669331288d5 -i td/osd-backfill-space/5/new.json 2021-01-15T17:29:40.307 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:29:40.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2021-01-15T17:29:40.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAD0QFg0TawLhAAv6nOWgkPBHKEM+8K7aLDDQ== --osd-uuid 9d1f2223-7a3c-4cf4-9432-1669331288d5 2021-01-15T17:29:40.347 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.347+0000 7f2d60a9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:40.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.357+0000 7f2d60a9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:40.359 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.359+0000 7f2d60a9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.360+0000 7f2d60a9e080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:29:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.360+0000 7f2d60a9e080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:29:40.361 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.360+0000 7f2d60a9e080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:29:40.424 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:40.424+0000 7f2d60a9e080 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2021-01-15T17:29:45.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2021-01-15T17:29:45.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:29:45.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:29:45.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:29:45.055 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:29:45.583 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:29:45.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:29:45.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:29:45.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:29:45.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:29:45.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:29:45.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:45.608+0000 7fa167d4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:45.633+0000 7fa167d4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:45.634 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:45.634+0000 7fa167d4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:29:45.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:29:45.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:29:45.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:29:45.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:29:45.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:29:45.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:46.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:29:46.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:46.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:29:46.001 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:29:46.150 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:46.149+0000 7fa167d4b080 -1 Falling back to public interface 2021-01-15T17:29:46.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:29:46.951 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:29:46.950+0000 7fa167d4b080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:29:47.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:29:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:29:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:29:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:29:47.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:29:47.419 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:29:47.819 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/464312,v1:127.0.0.1:6843/464312] [v2:127.0.0.1:6844/464312,v1:127.0.0.1:6845/464312] exists,up 9d1f2223-7a3c-4cf4-9432-1669331288d5 2021-01-15T17:29:47.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:29:47.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:29:47.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:29:47.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1125: TEST_ec_backfill_grow: ceph osd set-backfillfull-ratio .85 2021-01-15T17:29:48.603 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:29:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1127: TEST_ec_backfill_grow: ceph osd set-require-min-compat-client luminous 2021-01-15T17:29:49.524 INFO:tasks.workunit.client.0.smithi145.stderr:set require_min_compat_client to luminous 2021-01-15T17:29:49.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1128: TEST_ec_backfill_grow: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2021-01-15T17:29:50.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1129: TEST_ec_backfill_grow: ceph osd pool create test 1 1 erasure ec-profile 2021-01-15T17:29:50.942 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:29:50.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1131: TEST_ec_backfill_grow: wait_for_clean 2021-01-15T17:29:50.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:29:50.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:29:50.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:29:50.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:29:50.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:29:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:29:50.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:29:50.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:29:50.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:29:50.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:29:51.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:29:51.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:29:51.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:29:51.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:29:51.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:29:51.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:29:51.442 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:29:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:29:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:29:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:29:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:29:51.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:29:51.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:51.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:29:51.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T17:29:51.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T17:29:51.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T17:29:51.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:51.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:29:51.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T17:29:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T17:29:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-01-15T17:29:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:51.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:29:51.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T17:29:51.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T17:29:51.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448' 2021-01-15T17:29:51.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:51.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:29:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:29:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:29:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927' 2021-01-15T17:29:52.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:52.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:29:52.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T17:29:52.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T17:29:52.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405' 2021-01-15T17:29:52.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:29:52.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:29:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018883 2021-01-15T17:29:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018883 2021-01-15T17:29:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405 5-128849018883' 2021-01-15T17:29:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:52.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T17:29:52.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:52.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:29:52.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T17:29:52.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:52.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T17:29:52.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T17:29:52.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:29:52.346 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T17:29:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2021-01-15T17:29:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:52.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T17:29:52.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:29:52.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T17:29:52.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T17:29:52.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T17:29:52.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:29:52.756 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T17:29:53.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T17:29:53.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:53.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T17:29:53.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:53.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:29:53.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T17:29:53.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:53.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T17:29:53.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T17:29:53.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:29:53.157 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T17:29:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509448 2021-01-15T17:29:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:29:54.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:29:54.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:29:54.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T17:29:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:54.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:29:54.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:54.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:29:54.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:29:54.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:54.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:29:54.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:29:54.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:29:54.962 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:29:55.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345927 2021-01-15T17:29:55.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:55.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T17:29:55.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:55.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:29:55.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T17:29:55.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:55.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T17:29:55.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T17:29:55.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:29:55.360 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T17:29:55.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T17:29:55.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:29:55.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018883 2021-01-15T17:29:55.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:29:55.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:29:55.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018883 2021-01-15T17:29:55.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:29:55.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018883 2021-01-15T17:29:55.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-01-15T17:29:55.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:29:55.780 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018883 2021-01-15T17:29:56.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018884 -lt 128849018883 2021-01-15T17:29:56.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:29:56.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:29:56.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:29:56.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:29:56.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:29:56.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:29:56.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:29:56.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:29:56.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:29:56.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:29:56.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:29:57.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:29:57.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:29:57.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:29:57.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:29:57.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:29:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:29:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:29:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:29:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:29:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:29:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:29:57.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:29:57.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:29:57.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:29:57.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:29:57.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:29:57.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:29:57.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:29:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:29:58.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:29:58.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:29:58.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:29:58.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:29:58.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:29:58.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:29:58.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1133: TEST_ec_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/12kdata bs=1k count=12 2021-01-15T17:29:58.636 INFO:tasks.workunit.client.0.smithi145.stderr:12+0 records in 2021-01-15T17:29:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:12+0 records out 2021-01-15T17:29:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:12288 bytes (12 kB, 12 KiB) copied, 0.00011087 s, 111 MB/s 2021-01-15T17:29:58.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: seq 1 200 2021-01-15T17:29:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:58.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj1 td/osd-backfill-space/12kdata 2021-01-15T17:29:58.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:58.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj2 td/osd-backfill-space/12kdata 2021-01-15T17:29:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:59.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj3 td/osd-backfill-space/12kdata 2021-01-15T17:29:59.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:59.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj4 td/osd-backfill-space/12kdata 2021-01-15T17:29:59.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:59.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj5 td/osd-backfill-space/12kdata 2021-01-15T17:29:59.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:29:59.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj6 td/osd-backfill-space/12kdata 2021-01-15T17:30:00.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:00.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj7 td/osd-backfill-space/12kdata 2021-01-15T17:30:00.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:00.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj8 td/osd-backfill-space/12kdata 2021-01-15T17:30:00.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj9 td/osd-backfill-space/12kdata 2021-01-15T17:30:00.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:00.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj10 td/osd-backfill-space/12kdata 2021-01-15T17:30:01.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:01.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj11 td/osd-backfill-space/12kdata 2021-01-15T17:30:01.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:01.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj12 td/osd-backfill-space/12kdata 2021-01-15T17:30:01.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:01.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj13 td/osd-backfill-space/12kdata 2021-01-15T17:30:01.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:01.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj14 td/osd-backfill-space/12kdata 2021-01-15T17:30:02.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:02.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj15 td/osd-backfill-space/12kdata 2021-01-15T17:30:02.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:02.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj16 td/osd-backfill-space/12kdata 2021-01-15T17:30:02.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:02.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj17 td/osd-backfill-space/12kdata 2021-01-15T17:30:02.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:02.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj18 td/osd-backfill-space/12kdata 2021-01-15T17:30:03.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:03.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj19 td/osd-backfill-space/12kdata 2021-01-15T17:30:03.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:03.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj20 td/osd-backfill-space/12kdata 2021-01-15T17:30:03.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:03.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj21 td/osd-backfill-space/12kdata 2021-01-15T17:30:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj22 td/osd-backfill-space/12kdata 2021-01-15T17:30:04.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:04.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj23 td/osd-backfill-space/12kdata 2021-01-15T17:30:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj24 td/osd-backfill-space/12kdata 2021-01-15T17:30:04.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:04.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj25 td/osd-backfill-space/12kdata 2021-01-15T17:30:05.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj26 td/osd-backfill-space/12kdata 2021-01-15T17:30:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj27 td/osd-backfill-space/12kdata 2021-01-15T17:30:05.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:05.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj28 td/osd-backfill-space/12kdata 2021-01-15T17:30:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj29 td/osd-backfill-space/12kdata 2021-01-15T17:30:06.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:06.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj30 td/osd-backfill-space/12kdata 2021-01-15T17:30:06.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:06.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj31 td/osd-backfill-space/12kdata 2021-01-15T17:30:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:06.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj32 td/osd-backfill-space/12kdata 2021-01-15T17:30:06.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:06.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj33 td/osd-backfill-space/12kdata 2021-01-15T17:30:07.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:07.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj34 td/osd-backfill-space/12kdata 2021-01-15T17:30:07.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:07.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj35 td/osd-backfill-space/12kdata 2021-01-15T17:30:07.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:07.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj36 td/osd-backfill-space/12kdata 2021-01-15T17:30:07.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:07.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj37 td/osd-backfill-space/12kdata 2021-01-15T17:30:08.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:08.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj38 td/osd-backfill-space/12kdata 2021-01-15T17:30:08.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:08.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj39 td/osd-backfill-space/12kdata 2021-01-15T17:30:08.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:08.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj40 td/osd-backfill-space/12kdata 2021-01-15T17:30:08.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:08.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj41 td/osd-backfill-space/12kdata 2021-01-15T17:30:09.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:09.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj42 td/osd-backfill-space/12kdata 2021-01-15T17:30:09.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:09.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj43 td/osd-backfill-space/12kdata 2021-01-15T17:30:09.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:09.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj44 td/osd-backfill-space/12kdata 2021-01-15T17:30:09.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:09.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj45 td/osd-backfill-space/12kdata 2021-01-15T17:30:10.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:10.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj46 td/osd-backfill-space/12kdata 2021-01-15T17:30:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj47 td/osd-backfill-space/12kdata 2021-01-15T17:30:10.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:10.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj48 td/osd-backfill-space/12kdata 2021-01-15T17:30:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj49 td/osd-backfill-space/12kdata 2021-01-15T17:30:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj50 td/osd-backfill-space/12kdata 2021-01-15T17:30:11.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:11.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj51 td/osd-backfill-space/12kdata 2021-01-15T17:30:11.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:11.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj52 td/osd-backfill-space/12kdata 2021-01-15T17:30:12.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:12.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj53 td/osd-backfill-space/12kdata 2021-01-15T17:30:12.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:12.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj54 td/osd-backfill-space/12kdata 2021-01-15T17:30:12.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:12.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj55 td/osd-backfill-space/12kdata 2021-01-15T17:30:12.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:12.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj56 td/osd-backfill-space/12kdata 2021-01-15T17:30:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj57 td/osd-backfill-space/12kdata 2021-01-15T17:30:13.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:13.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj58 td/osd-backfill-space/12kdata 2021-01-15T17:30:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj59 td/osd-backfill-space/12kdata 2021-01-15T17:30:13.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:13.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj60 td/osd-backfill-space/12kdata 2021-01-15T17:30:14.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:14.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj61 td/osd-backfill-space/12kdata 2021-01-15T17:30:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj62 td/osd-backfill-space/12kdata 2021-01-15T17:30:14.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:14.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj63 td/osd-backfill-space/12kdata 2021-01-15T17:30:15.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:15.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj64 td/osd-backfill-space/12kdata 2021-01-15T17:30:15.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:15.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj65 td/osd-backfill-space/12kdata 2021-01-15T17:30:15.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:15.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj66 td/osd-backfill-space/12kdata 2021-01-15T17:30:15.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:15.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj67 td/osd-backfill-space/12kdata 2021-01-15T17:30:15.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:15.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj68 td/osd-backfill-space/12kdata 2021-01-15T17:30:16.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:16.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj69 td/osd-backfill-space/12kdata 2021-01-15T17:30:16.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:16.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj70 td/osd-backfill-space/12kdata 2021-01-15T17:30:16.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:16.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj71 td/osd-backfill-space/12kdata 2021-01-15T17:30:16.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:16.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj72 td/osd-backfill-space/12kdata 2021-01-15T17:30:17.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:17.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj73 td/osd-backfill-space/12kdata 2021-01-15T17:30:17.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:17.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj74 td/osd-backfill-space/12kdata 2021-01-15T17:30:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj75 td/osd-backfill-space/12kdata 2021-01-15T17:30:18.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:18.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj76 td/osd-backfill-space/12kdata 2021-01-15T17:30:18.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:18.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj77 td/osd-backfill-space/12kdata 2021-01-15T17:30:18.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:18.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj78 td/osd-backfill-space/12kdata 2021-01-15T17:30:18.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:18.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj79 td/osd-backfill-space/12kdata 2021-01-15T17:30:19.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:19.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj80 td/osd-backfill-space/12kdata 2021-01-15T17:30:19.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:19.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj81 td/osd-backfill-space/12kdata 2021-01-15T17:30:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj82 td/osd-backfill-space/12kdata 2021-01-15T17:30:19.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:19.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj83 td/osd-backfill-space/12kdata 2021-01-15T17:30:20.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:20.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj84 td/osd-backfill-space/12kdata 2021-01-15T17:30:20.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:20.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj85 td/osd-backfill-space/12kdata 2021-01-15T17:30:20.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:20.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj86 td/osd-backfill-space/12kdata 2021-01-15T17:30:20.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:20.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj87 td/osd-backfill-space/12kdata 2021-01-15T17:30:21.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:21.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj88 td/osd-backfill-space/12kdata 2021-01-15T17:30:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:21.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj89 td/osd-backfill-space/12kdata 2021-01-15T17:30:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj90 td/osd-backfill-space/12kdata 2021-01-15T17:30:21.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:21.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj91 td/osd-backfill-space/12kdata 2021-01-15T17:30:22.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:22.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj92 td/osd-backfill-space/12kdata 2021-01-15T17:30:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj93 td/osd-backfill-space/12kdata 2021-01-15T17:30:22.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:22.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj94 td/osd-backfill-space/12kdata 2021-01-15T17:30:22.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:22.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj95 td/osd-backfill-space/12kdata 2021-01-15T17:30:23.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:23.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj96 td/osd-backfill-space/12kdata 2021-01-15T17:30:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj97 td/osd-backfill-space/12kdata 2021-01-15T17:30:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:23.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj98 td/osd-backfill-space/12kdata 2021-01-15T17:30:23.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:23.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj99 td/osd-backfill-space/12kdata 2021-01-15T17:30:24.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:24.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj100 td/osd-backfill-space/12kdata 2021-01-15T17:30:24.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj101 td/osd-backfill-space/12kdata 2021-01-15T17:30:24.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:24.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj102 td/osd-backfill-space/12kdata 2021-01-15T17:30:24.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:24.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj103 td/osd-backfill-space/12kdata 2021-01-15T17:30:25.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:25.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj104 td/osd-backfill-space/12kdata 2021-01-15T17:30:25.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:25.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj105 td/osd-backfill-space/12kdata 2021-01-15T17:30:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj106 td/osd-backfill-space/12kdata 2021-01-15T17:30:25.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:25.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj107 td/osd-backfill-space/12kdata 2021-01-15T17:30:26.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:26.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj108 td/osd-backfill-space/12kdata 2021-01-15T17:30:26.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:26.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj109 td/osd-backfill-space/12kdata 2021-01-15T17:30:26.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:26.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj110 td/osd-backfill-space/12kdata 2021-01-15T17:30:26.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:26.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj111 td/osd-backfill-space/12kdata 2021-01-15T17:30:27.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:27.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj112 td/osd-backfill-space/12kdata 2021-01-15T17:30:27.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:27.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj113 td/osd-backfill-space/12kdata 2021-01-15T17:30:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj114 td/osd-backfill-space/12kdata 2021-01-15T17:30:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:27.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj115 td/osd-backfill-space/12kdata 2021-01-15T17:30:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:28.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj116 td/osd-backfill-space/12kdata 2021-01-15T17:30:28.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:28.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj117 td/osd-backfill-space/12kdata 2021-01-15T17:30:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:28.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj118 td/osd-backfill-space/12kdata 2021-01-15T17:30:28.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:28.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj119 td/osd-backfill-space/12kdata 2021-01-15T17:30:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj120 td/osd-backfill-space/12kdata 2021-01-15T17:30:29.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:29.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj121 td/osd-backfill-space/12kdata 2021-01-15T17:30:29.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:29.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj122 td/osd-backfill-space/12kdata 2021-01-15T17:30:29.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj123 td/osd-backfill-space/12kdata 2021-01-15T17:30:30.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:30.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj124 td/osd-backfill-space/12kdata 2021-01-15T17:30:30.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:30.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj125 td/osd-backfill-space/12kdata 2021-01-15T17:30:30.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:30.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj126 td/osd-backfill-space/12kdata 2021-01-15T17:30:31.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:31.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj127 td/osd-backfill-space/12kdata 2021-01-15T17:30:31.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:31.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj128 td/osd-backfill-space/12kdata 2021-01-15T17:30:31.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:31.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj129 td/osd-backfill-space/12kdata 2021-01-15T17:30:31.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:31.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj130 td/osd-backfill-space/12kdata 2021-01-15T17:30:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj131 td/osd-backfill-space/12kdata 2021-01-15T17:30:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj132 td/osd-backfill-space/12kdata 2021-01-15T17:30:32.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj133 td/osd-backfill-space/12kdata 2021-01-15T17:30:32.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:32.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj134 td/osd-backfill-space/12kdata 2021-01-15T17:30:32.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:32.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj135 td/osd-backfill-space/12kdata 2021-01-15T17:30:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj136 td/osd-backfill-space/12kdata 2021-01-15T17:30:33.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj137 td/osd-backfill-space/12kdata 2021-01-15T17:30:33.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj138 td/osd-backfill-space/12kdata 2021-01-15T17:30:33.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:33.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj139 td/osd-backfill-space/12kdata 2021-01-15T17:30:34.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:34.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj140 td/osd-backfill-space/12kdata 2021-01-15T17:30:34.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:34.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj141 td/osd-backfill-space/12kdata 2021-01-15T17:30:34.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:34.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj142 td/osd-backfill-space/12kdata 2021-01-15T17:30:35.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:35.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj143 td/osd-backfill-space/12kdata 2021-01-15T17:30:35.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:35.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj144 td/osd-backfill-space/12kdata 2021-01-15T17:30:35.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:35.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj145 td/osd-backfill-space/12kdata 2021-01-15T17:30:35.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:35.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj146 td/osd-backfill-space/12kdata 2021-01-15T17:30:36.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:36.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj147 td/osd-backfill-space/12kdata 2021-01-15T17:30:36.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:36.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj148 td/osd-backfill-space/12kdata 2021-01-15T17:30:36.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:36.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj149 td/osd-backfill-space/12kdata 2021-01-15T17:30:36.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:36.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj150 td/osd-backfill-space/12kdata 2021-01-15T17:30:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:37.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj151 td/osd-backfill-space/12kdata 2021-01-15T17:30:37.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:37.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj152 td/osd-backfill-space/12kdata 2021-01-15T17:30:37.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:37.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj153 td/osd-backfill-space/12kdata 2021-01-15T17:30:37.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:37.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj154 td/osd-backfill-space/12kdata 2021-01-15T17:30:38.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:38.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj155 td/osd-backfill-space/12kdata 2021-01-15T17:30:38.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:38.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj156 td/osd-backfill-space/12kdata 2021-01-15T17:30:38.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:38.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj157 td/osd-backfill-space/12kdata 2021-01-15T17:30:38.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj158 td/osd-backfill-space/12kdata 2021-01-15T17:30:39.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:39.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj159 td/osd-backfill-space/12kdata 2021-01-15T17:30:39.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:39.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj160 td/osd-backfill-space/12kdata 2021-01-15T17:30:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:39.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj161 td/osd-backfill-space/12kdata 2021-01-15T17:30:39.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:39.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj162 td/osd-backfill-space/12kdata 2021-01-15T17:30:40.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:40.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj163 td/osd-backfill-space/12kdata 2021-01-15T17:30:40.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:40.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj164 td/osd-backfill-space/12kdata 2021-01-15T17:30:40.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:40.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj165 td/osd-backfill-space/12kdata 2021-01-15T17:30:40.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:40.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj166 td/osd-backfill-space/12kdata 2021-01-15T17:30:41.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:41.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj167 td/osd-backfill-space/12kdata 2021-01-15T17:30:41.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:41.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj168 td/osd-backfill-space/12kdata 2021-01-15T17:30:41.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:41.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj169 td/osd-backfill-space/12kdata 2021-01-15T17:30:41.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:41.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj170 td/osd-backfill-space/12kdata 2021-01-15T17:30:42.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:42.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj171 td/osd-backfill-space/12kdata 2021-01-15T17:30:42.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:42.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj172 td/osd-backfill-space/12kdata 2021-01-15T17:30:42.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:42.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj173 td/osd-backfill-space/12kdata 2021-01-15T17:30:42.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:42.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj174 td/osd-backfill-space/12kdata 2021-01-15T17:30:43.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:43.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj175 td/osd-backfill-space/12kdata 2021-01-15T17:30:43.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:43.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj176 td/osd-backfill-space/12kdata 2021-01-15T17:30:43.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:43.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj177 td/osd-backfill-space/12kdata 2021-01-15T17:30:43.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:43.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj178 td/osd-backfill-space/12kdata 2021-01-15T17:30:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:44.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj179 td/osd-backfill-space/12kdata 2021-01-15T17:30:44.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:44.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj180 td/osd-backfill-space/12kdata 2021-01-15T17:30:44.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:44.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj181 td/osd-backfill-space/12kdata 2021-01-15T17:30:44.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:44.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj182 td/osd-backfill-space/12kdata 2021-01-15T17:30:45.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:45.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj183 td/osd-backfill-space/12kdata 2021-01-15T17:30:45.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:45.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj184 td/osd-backfill-space/12kdata 2021-01-15T17:30:45.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:45.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj185 td/osd-backfill-space/12kdata 2021-01-15T17:30:46.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:46.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj186 td/osd-backfill-space/12kdata 2021-01-15T17:30:46.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:46.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj187 td/osd-backfill-space/12kdata 2021-01-15T17:30:46.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:46.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj188 td/osd-backfill-space/12kdata 2021-01-15T17:30:46.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:46.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj189 td/osd-backfill-space/12kdata 2021-01-15T17:30:47.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:47.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj190 td/osd-backfill-space/12kdata 2021-01-15T17:30:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj191 td/osd-backfill-space/12kdata 2021-01-15T17:30:47.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:47.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj192 td/osd-backfill-space/12kdata 2021-01-15T17:30:47.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:47.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj193 td/osd-backfill-space/12kdata 2021-01-15T17:30:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:48.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj194 td/osd-backfill-space/12kdata 2021-01-15T17:30:48.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:48.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj195 td/osd-backfill-space/12kdata 2021-01-15T17:30:48.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:48.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj196 td/osd-backfill-space/12kdata 2021-01-15T17:30:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj197 td/osd-backfill-space/12kdata 2021-01-15T17:30:49.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:49.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj198 td/osd-backfill-space/12kdata 2021-01-15T17:30:49.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:49.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj199 td/osd-backfill-space/12kdata 2021-01-15T17:30:49.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1134: TEST_ec_backfill_grow: for i in $(seq 1 $ecobjects) 2021-01-15T17:30:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1136: TEST_ec_backfill_grow: rados -p test put obj200 td/osd-backfill-space/12kdata 2021-01-15T17:30:49.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1139: TEST_ec_backfill_grow: get_pg test obj1 2021-01-15T17:30:49.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:30:49.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:30:49.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:30:49.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:30:50.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1139: TEST_ec_backfill_grow: local PG=1.0 2021-01-15T17:30:50.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1141: TEST_ec_backfill_grow: get_primary test obj1 2021-01-15T17:30:50.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:30:50.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:30:50.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:30:50.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:30:50.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1141: TEST_ec_backfill_grow: local primary=1 2021-01-15T17:30:50.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1142: TEST_ec_backfill_grow: get_not_primary test obj1 2021-01-15T17:30:50.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:30:50.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:30:50.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:30:50.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:30:50.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:30:50.693 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:30:50.693 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:30:51.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:30:51.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:30:51.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:30:51.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1142: TEST_ec_backfill_grow: local otherosd=0 2021-01-15T17:30:51.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1144: TEST_ec_backfill_grow: ceph osd set noout 2021-01-15T17:30:52.286 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T17:30:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1145: TEST_ec_backfill_grow: kill_daemons td/osd-backfill-space TERM 0 2021-01-15T17:30:52.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:30:52.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:30:52.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:30:52.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:30:52.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:30:52.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:30:52.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1147: TEST_ec_backfill_grow: expr 200 / 4 2021-01-15T17:30:52.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1147: TEST_ec_backfill_grow: rmobjects=50 2021-01-15T17:30:52.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: seq 1 50 2021-01-15T17:30:52.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:52.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj1 2021-01-15T17:30:53.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:53.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj2 2021-01-15T17:30:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj3 2021-01-15T17:30:53.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:53.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj4 2021-01-15T17:30:53.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:53.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj5 2021-01-15T17:30:54.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:54.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj6 2021-01-15T17:30:54.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj7 2021-01-15T17:30:54.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:54.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj8 2021-01-15T17:30:54.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:54.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj9 2021-01-15T17:30:55.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:55.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj10 2021-01-15T17:30:55.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:55.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj11 2021-01-15T17:30:55.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:55.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj12 2021-01-15T17:30:55.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:55.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj13 2021-01-15T17:30:56.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:56.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj14 2021-01-15T17:30:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj15 2021-01-15T17:30:56.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:56.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj16 2021-01-15T17:30:56.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:56.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj17 2021-01-15T17:30:57.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:57.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj18 2021-01-15T17:30:57.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:57.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj19 2021-01-15T17:30:57.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:57.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj20 2021-01-15T17:30:57.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:57.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj21 2021-01-15T17:30:57.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:57.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj22 2021-01-15T17:30:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:58.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj23 2021-01-15T17:30:58.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:58.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj24 2021-01-15T17:30:58.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:58.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj25 2021-01-15T17:30:58.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:58.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj26 2021-01-15T17:30:58.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:58.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj27 2021-01-15T17:30:59.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:59.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj28 2021-01-15T17:30:59.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:59.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj29 2021-01-15T17:30:59.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:59.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj30 2021-01-15T17:30:59.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:30:59.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj31 2021-01-15T17:31:00.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj32 2021-01-15T17:31:00.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:00.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj33 2021-01-15T17:31:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:00.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj34 2021-01-15T17:31:00.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj35 2021-01-15T17:31:00.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:00.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj36 2021-01-15T17:31:01.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:01.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj37 2021-01-15T17:31:01.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:01.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj38 2021-01-15T17:31:01.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj39 2021-01-15T17:31:01.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:01.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj40 2021-01-15T17:31:01.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:01.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj41 2021-01-15T17:31:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj42 2021-01-15T17:31:02.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:02.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj43 2021-01-15T17:31:02.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:02.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj44 2021-01-15T17:31:02.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:02.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj45 2021-01-15T17:31:03.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:03.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj46 2021-01-15T17:31:03.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:03.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj47 2021-01-15T17:31:03.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:03.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj48 2021-01-15T17:31:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:03.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj49 2021-01-15T17:31:03.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1148: TEST_ec_backfill_grow: for i in $(seq 1 $rmobjects) 2021-01-15T17:31:03.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1150: TEST_ec_backfill_grow: rados -p test rm obj50 2021-01-15T17:31:04.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1153: TEST_ec_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/36kdata bs=1k count=36 2021-01-15T17:31:04.065 INFO:tasks.workunit.client.0.smithi145.stderr:36+0 records in 2021-01-15T17:31:04.065 INFO:tasks.workunit.client.0.smithi145.stderr:36+0 records out 2021-01-15T17:31:04.066 INFO:tasks.workunit.client.0.smithi145.stderr:36864 bytes (37 kB, 36 KiB) copied, 0.00040874 s, 90.2 MB/s 2021-01-15T17:31:04.066 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: expr 50 + 1 2021-01-15T17:31:04.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: seq 51 200 2021-01-15T17:31:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj51 td/osd-backfill-space/36kdata 2021-01-15T17:31:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:04.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj52 td/osd-backfill-space/36kdata 2021-01-15T17:31:04.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:04.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj53 td/osd-backfill-space/36kdata 2021-01-15T17:31:04.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:04.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj54 td/osd-backfill-space/36kdata 2021-01-15T17:31:04.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:04.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj55 td/osd-backfill-space/36kdata 2021-01-15T17:31:05.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:05.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj56 td/osd-backfill-space/36kdata 2021-01-15T17:31:05.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:05.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj57 td/osd-backfill-space/36kdata 2021-01-15T17:31:05.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:05.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj58 td/osd-backfill-space/36kdata 2021-01-15T17:31:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj59 td/osd-backfill-space/36kdata 2021-01-15T17:31:06.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:06.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj60 td/osd-backfill-space/36kdata 2021-01-15T17:31:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj61 td/osd-backfill-space/36kdata 2021-01-15T17:31:06.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:06.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj62 td/osd-backfill-space/36kdata 2021-01-15T17:31:07.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:07.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj63 td/osd-backfill-space/36kdata 2021-01-15T17:31:07.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:07.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj64 td/osd-backfill-space/36kdata 2021-01-15T17:31:07.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:07.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj65 td/osd-backfill-space/36kdata 2021-01-15T17:31:07.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:07.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj66 td/osd-backfill-space/36kdata 2021-01-15T17:31:08.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:08.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj67 td/osd-backfill-space/36kdata 2021-01-15T17:31:08.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj68 td/osd-backfill-space/36kdata 2021-01-15T17:31:08.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:08.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj69 td/osd-backfill-space/36kdata 2021-01-15T17:31:08.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:08.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj70 td/osd-backfill-space/36kdata 2021-01-15T17:31:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj71 td/osd-backfill-space/36kdata 2021-01-15T17:31:09.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:09.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj72 td/osd-backfill-space/36kdata 2021-01-15T17:31:09.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:09.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj73 td/osd-backfill-space/36kdata 2021-01-15T17:31:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj74 td/osd-backfill-space/36kdata 2021-01-15T17:31:10.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:10.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj75 td/osd-backfill-space/36kdata 2021-01-15T17:31:10.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:10.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj76 td/osd-backfill-space/36kdata 2021-01-15T17:31:10.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:10.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj77 td/osd-backfill-space/36kdata 2021-01-15T17:31:10.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:10.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj78 td/osd-backfill-space/36kdata 2021-01-15T17:31:10.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:10.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj79 td/osd-backfill-space/36kdata 2021-01-15T17:31:11.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:11.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj80 td/osd-backfill-space/36kdata 2021-01-15T17:31:11.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:11.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj81 td/osd-backfill-space/36kdata 2021-01-15T17:31:11.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:11.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj82 td/osd-backfill-space/36kdata 2021-01-15T17:31:11.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:11.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj83 td/osd-backfill-space/36kdata 2021-01-15T17:31:12.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:12.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj84 td/osd-backfill-space/36kdata 2021-01-15T17:31:12.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:12.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj85 td/osd-backfill-space/36kdata 2021-01-15T17:31:12.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:12.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj86 td/osd-backfill-space/36kdata 2021-01-15T17:31:12.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:12.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj87 td/osd-backfill-space/36kdata 2021-01-15T17:31:13.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:13.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj88 td/osd-backfill-space/36kdata 2021-01-15T17:31:13.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:13.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj89 td/osd-backfill-space/36kdata 2021-01-15T17:31:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:13.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj90 td/osd-backfill-space/36kdata 2021-01-15T17:31:13.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:13.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj91 td/osd-backfill-space/36kdata 2021-01-15T17:31:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj92 td/osd-backfill-space/36kdata 2021-01-15T17:31:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj93 td/osd-backfill-space/36kdata 2021-01-15T17:31:14.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:14.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj94 td/osd-backfill-space/36kdata 2021-01-15T17:31:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj95 td/osd-backfill-space/36kdata 2021-01-15T17:31:15.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:15.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj96 td/osd-backfill-space/36kdata 2021-01-15T17:31:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:15.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj97 td/osd-backfill-space/36kdata 2021-01-15T17:31:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:15.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj98 td/osd-backfill-space/36kdata 2021-01-15T17:31:15.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:15.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj99 td/osd-backfill-space/36kdata 2021-01-15T17:31:16.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:16.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj100 td/osd-backfill-space/36kdata 2021-01-15T17:31:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj101 td/osd-backfill-space/36kdata 2021-01-15T17:31:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:16.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj102 td/osd-backfill-space/36kdata 2021-01-15T17:31:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj103 td/osd-backfill-space/36kdata 2021-01-15T17:31:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj104 td/osd-backfill-space/36kdata 2021-01-15T17:31:17.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:17.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj105 td/osd-backfill-space/36kdata 2021-01-15T17:31:17.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:17.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj106 td/osd-backfill-space/36kdata 2021-01-15T17:31:17.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:17.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj107 td/osd-backfill-space/36kdata 2021-01-15T17:31:18.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:18.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj108 td/osd-backfill-space/36kdata 2021-01-15T17:31:18.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:18.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj109 td/osd-backfill-space/36kdata 2021-01-15T17:31:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:18.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj110 td/osd-backfill-space/36kdata 2021-01-15T17:31:18.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:18.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj111 td/osd-backfill-space/36kdata 2021-01-15T17:31:18.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:18.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj112 td/osd-backfill-space/36kdata 2021-01-15T17:31:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:19.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj113 td/osd-backfill-space/36kdata 2021-01-15T17:31:19.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:19.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj114 td/osd-backfill-space/36kdata 2021-01-15T17:31:19.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:19.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj115 td/osd-backfill-space/36kdata 2021-01-15T17:31:19.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:19.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj116 td/osd-backfill-space/36kdata 2021-01-15T17:31:20.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj117 td/osd-backfill-space/36kdata 2021-01-15T17:31:20.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:20.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj118 td/osd-backfill-space/36kdata 2021-01-15T17:31:20.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj119 td/osd-backfill-space/36kdata 2021-01-15T17:31:20.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:20.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj120 td/osd-backfill-space/36kdata 2021-01-15T17:31:21.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:21.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj121 td/osd-backfill-space/36kdata 2021-01-15T17:31:21.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:21.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj122 td/osd-backfill-space/36kdata 2021-01-15T17:31:21.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:21.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj123 td/osd-backfill-space/36kdata 2021-01-15T17:31:21.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:21.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj124 td/osd-backfill-space/36kdata 2021-01-15T17:31:22.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:22.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj125 td/osd-backfill-space/36kdata 2021-01-15T17:31:22.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:22.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj126 td/osd-backfill-space/36kdata 2021-01-15T17:31:22.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:22.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj127 td/osd-backfill-space/36kdata 2021-01-15T17:31:22.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:22.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj128 td/osd-backfill-space/36kdata 2021-01-15T17:31:23.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:23.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj129 td/osd-backfill-space/36kdata 2021-01-15T17:31:23.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:23.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj130 td/osd-backfill-space/36kdata 2021-01-15T17:31:23.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:23.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj131 td/osd-backfill-space/36kdata 2021-01-15T17:31:23.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:23.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj132 td/osd-backfill-space/36kdata 2021-01-15T17:31:23.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:23.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj133 td/osd-backfill-space/36kdata 2021-01-15T17:31:24.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:24.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj134 td/osd-backfill-space/36kdata 2021-01-15T17:31:24.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:24.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj135 td/osd-backfill-space/36kdata 2021-01-15T17:31:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:24.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj136 td/osd-backfill-space/36kdata 2021-01-15T17:31:24.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:24.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj137 td/osd-backfill-space/36kdata 2021-01-15T17:31:25.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:25.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj138 td/osd-backfill-space/36kdata 2021-01-15T17:31:25.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:25.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj139 td/osd-backfill-space/36kdata 2021-01-15T17:31:25.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:25.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj140 td/osd-backfill-space/36kdata 2021-01-15T17:31:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj141 td/osd-backfill-space/36kdata 2021-01-15T17:31:26.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:26.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj142 td/osd-backfill-space/36kdata 2021-01-15T17:31:26.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:26.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj143 td/osd-backfill-space/36kdata 2021-01-15T17:31:26.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:26.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj144 td/osd-backfill-space/36kdata 2021-01-15T17:31:26.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:26.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj145 td/osd-backfill-space/36kdata 2021-01-15T17:31:26.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:26.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj146 td/osd-backfill-space/36kdata 2021-01-15T17:31:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:27.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj147 td/osd-backfill-space/36kdata 2021-01-15T17:31:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj148 td/osd-backfill-space/36kdata 2021-01-15T17:31:27.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:27.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj149 td/osd-backfill-space/36kdata 2021-01-15T17:31:27.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:27.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj150 td/osd-backfill-space/36kdata 2021-01-15T17:31:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj151 td/osd-backfill-space/36kdata 2021-01-15T17:31:28.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:28.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj152 td/osd-backfill-space/36kdata 2021-01-15T17:31:28.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:28.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj153 td/osd-backfill-space/36kdata 2021-01-15T17:31:28.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:28.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj154 td/osd-backfill-space/36kdata 2021-01-15T17:31:29.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:29.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj155 td/osd-backfill-space/36kdata 2021-01-15T17:31:29.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:29.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj156 td/osd-backfill-space/36kdata 2021-01-15T17:31:29.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj157 td/osd-backfill-space/36kdata 2021-01-15T17:31:29.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:29.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj158 td/osd-backfill-space/36kdata 2021-01-15T17:31:30.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:30.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj159 td/osd-backfill-space/36kdata 2021-01-15T17:31:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj160 td/osd-backfill-space/36kdata 2021-01-15T17:31:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:30.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj161 td/osd-backfill-space/36kdata 2021-01-15T17:31:30.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:30.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj162 td/osd-backfill-space/36kdata 2021-01-15T17:31:30.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:30.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj163 td/osd-backfill-space/36kdata 2021-01-15T17:31:31.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:31.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj164 td/osd-backfill-space/36kdata 2021-01-15T17:31:31.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:31.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj165 td/osd-backfill-space/36kdata 2021-01-15T17:31:31.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:31.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj166 td/osd-backfill-space/36kdata 2021-01-15T17:31:31.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:31.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj167 td/osd-backfill-space/36kdata 2021-01-15T17:31:32.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:32.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj168 td/osd-backfill-space/36kdata 2021-01-15T17:31:32.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:32.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj169 td/osd-backfill-space/36kdata 2021-01-15T17:31:32.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj170 td/osd-backfill-space/36kdata 2021-01-15T17:31:32.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:32.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj171 td/osd-backfill-space/36kdata 2021-01-15T17:31:32.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:32.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj172 td/osd-backfill-space/36kdata 2021-01-15T17:31:33.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:33.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj173 td/osd-backfill-space/36kdata 2021-01-15T17:31:33.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:33.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj174 td/osd-backfill-space/36kdata 2021-01-15T17:31:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj175 td/osd-backfill-space/36kdata 2021-01-15T17:31:33.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:33.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj176 td/osd-backfill-space/36kdata 2021-01-15T17:31:34.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:34.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj177 td/osd-backfill-space/36kdata 2021-01-15T17:31:34.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:34.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj178 td/osd-backfill-space/36kdata 2021-01-15T17:31:34.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:34.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj179 td/osd-backfill-space/36kdata 2021-01-15T17:31:34.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:34.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj180 td/osd-backfill-space/36kdata 2021-01-15T17:31:34.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:34.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj181 td/osd-backfill-space/36kdata 2021-01-15T17:31:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:35.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj182 td/osd-backfill-space/36kdata 2021-01-15T17:31:35.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:35.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj183 td/osd-backfill-space/36kdata 2021-01-15T17:31:35.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:35.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj184 td/osd-backfill-space/36kdata 2021-01-15T17:31:35.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:35.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj185 td/osd-backfill-space/36kdata 2021-01-15T17:31:36.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:36.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj186 td/osd-backfill-space/36kdata 2021-01-15T17:31:36.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:36.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj187 td/osd-backfill-space/36kdata 2021-01-15T17:31:36.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:36.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj188 td/osd-backfill-space/36kdata 2021-01-15T17:31:37.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:37.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj189 td/osd-backfill-space/36kdata 2021-01-15T17:31:37.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:37.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj190 td/osd-backfill-space/36kdata 2021-01-15T17:31:37.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:37.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj191 td/osd-backfill-space/36kdata 2021-01-15T17:31:37.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:37.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj192 td/osd-backfill-space/36kdata 2021-01-15T17:31:37.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:37.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj193 td/osd-backfill-space/36kdata 2021-01-15T17:31:38.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:38.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj194 td/osd-backfill-space/36kdata 2021-01-15T17:31:38.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:38.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj195 td/osd-backfill-space/36kdata 2021-01-15T17:31:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj196 td/osd-backfill-space/36kdata 2021-01-15T17:31:38.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:38.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj197 td/osd-backfill-space/36kdata 2021-01-15T17:31:39.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj198 td/osd-backfill-space/36kdata 2021-01-15T17:31:39.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:39.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj199 td/osd-backfill-space/36kdata 2021-01-15T17:31:39.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1154: TEST_ec_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $ecobjects) 2021-01-15T17:31:39.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1156: TEST_ec_backfill_grow: rados -p test put obj200 td/osd-backfill-space/36kdata 2021-01-15T17:31:39.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1159: TEST_ec_backfill_grow: activate_osd td/osd-backfill-space 0 2021-01-15T17:31:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T17:31:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T17:31:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T17:31:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T17:31:39.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:31:39.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:31:39.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:31:39.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:31:39.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:31:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:31:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:31:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T17:31:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T17:31:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:31:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T17:31:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:31:39.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:31:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:31:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:31:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:31:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:31:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:31:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:31:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T17:31:39.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T17:31:39.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:31:39.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T17:31:39.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:31:39.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/0/whoami 2021-01-15T17:31:39.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T17:31:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T17:31:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T17:31:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T17:31:39.703 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:31:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:31:39.722+0000 7feac0380080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:31:39.725 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:31:39.725+0000 7feac0380080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:31:39.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:31:39.727+0000 7feac0380080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:31:40.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T17:31:40.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:31:40.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:31:40.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:31:40.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:31:40.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:31:40.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:31:40.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:31:40.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:31:40.108 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:31:40.242 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:31:40.241+0000 7feac0380080 -1 Falling back to public interface 2021-01-15T17:31:40.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:31:41.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:31:41.458+0000 7feac0380080 -1 osd.0 41 log_to_monitors {default=true} 2021-01-15T17:31:41.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:31:41.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:31:41.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:31:41.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:31:41.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:31:41.525 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:31:41.968 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 45 up_thru 0 down_at 43 last_clean_interval [5,41) [v2:127.0.0.1:6802/473673,v1:127.0.0.1:6803/473673] [v2:127.0.0.1:6804/473673,v1:127.0.0.1:6805/473673] exists,up c68b2430-6075-4746-94b9-eb9fdac58a38 2021-01-15T17:31:41.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:31:41.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:31:41.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:31:41.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1161: TEST_ec_backfill_grow: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:31:42.105 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:31:42.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1163: TEST_ec_backfill_grow: sleep 2 2021-01-15T17:31:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1165: TEST_ec_backfill_grow: wait_for_clean 2021-01-15T17:31:44.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:31:44.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:31:44.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:31:44.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:31:44.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:31:44.117 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:31:44.118 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:31:44.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:31:44.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:31:44.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:31:44.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:31:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:31:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:31:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:31:44.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:31:44.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:31:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:31:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:31:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:31:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:31:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:31:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:31:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:44.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:31:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528323 2021-01-15T17:31:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528323 2021-01-15T17:31:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323' 2021-01-15T17:31:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:44.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:31:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672994 2021-01-15T17:31:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672994 2021-01-15T17:31:44.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323 1-42949672994' 2021-01-15T17:31:44.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:44.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:31:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509473 2021-01-15T17:31:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509473 2021-01-15T17:31:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323 1-42949672994 2-64424509473' 2021-01-15T17:31:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:45.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:31:45.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345951 2021-01-15T17:31:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345951 2021-01-15T17:31:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323 1-42949672994 2-64424509473 3-85899345951' 2021-01-15T17:31:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:45.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:31:45.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182430 2021-01-15T17:31:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182430 2021-01-15T17:31:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323 1-42949672994 2-64424509473 3-85899345951 4-107374182430' 2021-01-15T17:31:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:31:45.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:31:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018908 2021-01-15T17:31:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018908 2021-01-15T17:31:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-193273528323 1-42949672994 2-64424509473 3-85899345951 4-107374182430 5-128849018908' 2021-01-15T17:31:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:45.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-193273528323 2021-01-15T17:31:45.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:45.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:31:45.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-193273528323 2021-01-15T17:31:45.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:45.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528323 2021-01-15T17:31:45.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 193273528323' 2021-01-15T17:31:45.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:31:45.466 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 193273528323 2021-01-15T17:31:45.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 1 -lt 193273528323 2021-01-15T17:31:45.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:31:46.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:31:46.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:31:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528323 -lt 193273528323 2021-01-15T17:31:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:47.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672994 2021-01-15T17:31:47.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:47.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:31:47.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672994 2021-01-15T17:31:47.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:47.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672994 2021-01-15T17:31:47.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2021-01-15T17:31:47.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:31:47.265 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672994 2021-01-15T17:31:47.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672994 -lt 42949672994 2021-01-15T17:31:47.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:47.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509473 2021-01-15T17:31:47.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:47.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:31:47.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509473 2021-01-15T17:31:47.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:47.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509473 2021-01-15T17:31:47.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2021-01-15T17:31:47.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:31:47.815 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509473 2021-01-15T17:31:48.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509473 -lt 64424509473 2021-01-15T17:31:48.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:48.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345951 2021-01-15T17:31:48.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:48.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:31:48.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345951 2021-01-15T17:31:48.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:48.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345951 2021-01-15T17:31:48.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2021-01-15T17:31:48.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:31:48.218 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345951 2021-01-15T17:31:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345952 -lt 85899345951 2021-01-15T17:31:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:48.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182430 2021-01-15T17:31:48.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:48.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:31:48.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182430 2021-01-15T17:31:48.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:48.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182430 2021-01-15T17:31:48.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2021-01-15T17:31:48.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:31:48.618 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182430 2021-01-15T17:31:49.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182430 -lt 107374182430 2021-01-15T17:31:49.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:31:49.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018908 2021-01-15T17:31:49.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:31:49.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:31:49.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018908 2021-01-15T17:31:49.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:31:49.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018908 2021-01-15T17:31:49.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018908' 2021-01-15T17:31:49.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:31:49.042 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018908 2021-01-15T17:31:49.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018908 -lt 128849018908 2021-01-15T17:31:49.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:31:49.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:49.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:49.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:31:49.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:49.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:49.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:49.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:49.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:49.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:49.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:50.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:50.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:50.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:50.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:50.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:31:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:31:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:31:50.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:50.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:50.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:50.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:50.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:50.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:50.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:51.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:51.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:51.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:51.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:51.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:51.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:51.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:51.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:51.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:51.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:51.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:31:52.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=127209 2021-01-15T17:31:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 127209 '!=' null 2021-01-15T17:31:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:31:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:52.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:52.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:52.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:52.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:52.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:52.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:52.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:52.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:52.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:52.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:52.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:52.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:53.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:53.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:53.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:53.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:53.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:53.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:53.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:53.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:53.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:31:54.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=127209 2021-01-15T17:31:54.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 127209 '!=' null 2021-01-15T17:31:54.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:31:54.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:54.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:54.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:54.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:54.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:54.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:54.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:54.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:54.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:54.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:54.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:54.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:54.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:54.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:55.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:55.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:55.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:31:55.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=108642 2021-01-15T17:31:55.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 108642 '!=' null 2021-01-15T17:31:55.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:31:55.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:55.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:55.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:55.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:55.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:55.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:55.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:55.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:55.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:55.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:56.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:56.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:56.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:56.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:56.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:56.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:56.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:56.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:56.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:31:57.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=101360 2021-01-15T17:31:57.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 101360 '!=' null 2021-01-15T17:31:57.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:31:57.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:57.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:57.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:57.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:57.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:57.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:57.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:57.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:57.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:57.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:57.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:57.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:57.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:58.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:31:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=156649 2021-01-15T17:31:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 156649 '!=' null 2021-01-15T17:31:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:31:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:31:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:31:58.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:31:58.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:31:58.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:31:58.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:31:58.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:31:58.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:31:58.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:31:58.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:31:59.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:31:59.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:31:59.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:31:59.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:31:59.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:31:59.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:31:59.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:31:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:31:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:31:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:31:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:31:59.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:31:59.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:00.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=141291 2021-01-15T17:32:00.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 141291 '!=' null 2021-01-15T17:32:00.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:00.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:00.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:00.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:00.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:00.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:00.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:00.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:00.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:00.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:00.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:00.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:01.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:01.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:01.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:01.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:01.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:01.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:01.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:01.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=141291 2021-01-15T17:32:01.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 141291 '!=' null 2021-01-15T17:32:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:01.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:01.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:01.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:01.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:01.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:01.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:01.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:01.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:02.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:02.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:02.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:02.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:02.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:02.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:02.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:02.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:02.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:02.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:03.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=208865 2021-01-15T17:32:03.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 208865 '!=' null 2021-01-15T17:32:03.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:03.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:03.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:03.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:03.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:03.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:03.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:03.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:03.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:03.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:03.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:03.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:03.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:03.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:03.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:04.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:04.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:04.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:04.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:04.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:04.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:04.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:04.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:04.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:04.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138219 2021-01-15T17:32:04.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138219 '!=' null 2021-01-15T17:32:04.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:04.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:04.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:04.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:04.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:04.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:04.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:04.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:04.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:05.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:05.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:05.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:05.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:05.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:05.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:05.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:05.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:05.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:06.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138218 2021-01-15T17:32:06.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138218 '!=' null 2021-01-15T17:32:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:06.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:06.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:06.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:06.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:06.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:06.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:06.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:06.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:06.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:06.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:06.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:06.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:06.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:07.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:07.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:07.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:07.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:07.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:07.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:07.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:07.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:07.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:07.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138218 2021-01-15T17:32:07.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138218 '!=' null 2021-01-15T17:32:07.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:07.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:07.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:07.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:07.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:07.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:07.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:07.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:07.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:07.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:07.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:08.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:08.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:08.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:08.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:08.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:08.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:08.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:08.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:08.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:08.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:08.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=208863 2021-01-15T17:32:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 208863 '!=' null 2021-01-15T17:32:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:09.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:09.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:09.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:09.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:09.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:09.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:09.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:09.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:09.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:09.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:09.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:10.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:10.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:10.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:10.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138216 2021-01-15T17:32:10.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138216 '!=' null 2021-01-15T17:32:10.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:11.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:11.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:11.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:11.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:11.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:11.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:11.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:11.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:11.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:11.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:11.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:11.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:11.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:11.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:11.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:11.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:11.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:11.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:11.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:11.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:11.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:12.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=205790 2021-01-15T17:32:12.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 205790 '!=' null 2021-01-15T17:32:12.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:12.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:12.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:12.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:12.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:12.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:12.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:12.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:12.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:12.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:12.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:12.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:12.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:12.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:12.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:13.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:13.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:13.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:13.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:13.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:14.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=205790 2021-01-15T17:32:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 205790 '!=' null 2021-01-15T17:32:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:14.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:14.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:14.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:14.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:14.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:14.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:14.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:14.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:15.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:15.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:15.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138216 2021-01-15T17:32:15.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138216 '!=' null 2021-01-15T17:32:15.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:15.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:15.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:15.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:15.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:15.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:15.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:15.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:15.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:15.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:15.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:16.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:16.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:16.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:16.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:16.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:16.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:16.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:16.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=138216 2021-01-15T17:32:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 138216 '!=' null 2021-01-15T17:32:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:17.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:17.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:17.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:17.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:17.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:17.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:17.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:17.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:17.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:17.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:17.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:17.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:17.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:18.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:18.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:18.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:18.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=202724 2021-01-15T17:32:18.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 202724 '!=' null 2021-01-15T17:32:18.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:18.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:18.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:18.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:18.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:18.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:18.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:18.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:18.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:18.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:18.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:19.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:19.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:19.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:19.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:19.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:19.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:19.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:19.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:19.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:19.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:19.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:19.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:19.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=101361 2021-01-15T17:32:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 101361 '!=' null 2021-01-15T17:32:20.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:20.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:20.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:20.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:20.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:20.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:20.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:20.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:20.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:20.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:20.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:20.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:32:20.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:20.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:20.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:32:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:32:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:32:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:32:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:32:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:32:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:32:21.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:32:21.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:32:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=101361 2021-01-15T17:32:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 101361 '!=' null 2021-01-15T17:32:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:32:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:32:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:32:21.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:32:21.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:32:21.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:32:21.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:32:21.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:32:21.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:32:21.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:32:21.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:32:22.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:32:22.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:32:22.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:32:22.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:32:22.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:32:22.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:32:22.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:32:22.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1167: TEST_ec_backfill_grow: delete_pool test 2021-01-15T17:32:22.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:32:22.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:32:23.278 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:32:23.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:1168: TEST_ec_backfill_grow: kill_daemons td/osd-backfill-space 2021-01-15T17:32:23.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:32:23.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:32:23.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:32:23.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:32:23.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:32:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:32:23.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:32:23.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:32:23.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:32:23.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:32:23.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:32:23.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:32:23.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:32:23.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:32:23.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:32:23.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:32:23.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:32:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:32:23.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:32:23.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:32:23.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:32:23.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:32:23.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:32:23.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:32:23.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:32:23.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:32:23.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:32:23.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:32:23.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:32:23.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:32:23.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:23.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:32:23.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:32:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:32:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:32:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:32:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:32:23.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:32:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:32:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:32:23.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:32:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:32:23.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:32:23.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:32:23.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:32:23.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:32:23.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:32:23.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:32:23.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:32:23.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:32:23.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:32:23.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:32:23.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:32:23.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:32:23.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:32:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:32:23.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:32:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:32:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:32:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:32:23.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:32:23.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:23.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:23.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:32:23.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:32:23.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:32:23.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:32:23.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:32:23.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:23.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:23.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:32:23.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:32:23.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:32:23.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:32:23.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:32:23.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_multi td/osd-backfill-space 2021-01-15T17:32:23.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:748: TEST_ec_backfill_multi: local dir=td/osd-backfill-space 2021-01-15T17:32:23.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:749: TEST_ec_backfill_multi: local EC= 2021-01-15T17:32:23.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:750: TEST_ec_backfill_multi: local pools=2 2021-01-15T17:32:23.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:751: TEST_ec_backfill_multi: local OSDS=6 2021-01-15T17:32:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:752: TEST_ec_backfill_multi: local k=3 2021-01-15T17:32:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:753: TEST_ec_backfill_multi: local m=2 2021-01-15T17:32:23.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:754: TEST_ec_backfill_multi: expr 600 / 3 2021-01-15T17:32:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:754: TEST_ec_backfill_multi: local ecobjects=200 2021-01-15T17:32:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:756: TEST_ec_backfill_multi: run_mon td/osd-backfill-space a 2021-01-15T17:32:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:32:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:32:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:32:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:32:23.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:32:23.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:32:23.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:32:23.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:23.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:23.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:23.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:23.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:23.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:23.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:32:23.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:32:23.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:32:23.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:32:23.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:32:23.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:32:23.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:32:23.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:32:23.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:32:23.940 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:32:23.940 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:32:23.941 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:23.941 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:23.941 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:32:23.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:32:23.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:32:24.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:32:24.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:32:24.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:32:24.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:32:24.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:32:24.040 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:32:24.041 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:32:24.041 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:32:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:32:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:24.042 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:32:24.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:32:24.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:32:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:757: TEST_ec_backfill_multi: run_mgr td/osd-backfill-space x 2021-01-15T17:32:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:32:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:32:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:32:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:32:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:32:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:32:24.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:32:24.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:24.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:24.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:24.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:24.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:24.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:24.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:32:24.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:32:24.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:758: TEST_ec_backfill_multi: export CEPH_ARGS 2021-01-15T17:32:24.665 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: expr 6 - 1 2021-01-15T17:32:24.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: seq 0 5 2021-01-15T17:32:24.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:32:24.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 0 2021-01-15T17:32:24.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:32:24.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:32:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:32:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:32:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:32:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:32:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:32:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:32:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:32:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:32:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:32:24.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:32:24.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:32:24.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:32:24.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:32:24.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:24.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:24.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:24.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:24.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:24.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:32:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:32:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:32:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:32:24.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:32:24.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:32:24.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:32:24.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9860a1fe-1945-49e3-b4ee-2b78703ff430 2021-01-15T17:32:24.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 9860a1fe-1945-49e3-b4ee-2b78703ff430' 2021-01-15T17:32:24.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:32:24.678 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 9860a1fe-1945-49e3-b4ee-2b78703ff430 2021-01-15T17:32:24.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCo0QFgQhz+KBAA7fKYCT+3fwAne/0hwTh/6A== 2021-01-15T17:32:24.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCo0QFgQhz+KBAA7fKYCT+3fwAne/0hwTh/6A=="}' 2021-01-15T17:32:24.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9860a1fe-1945-49e3-b4ee-2b78703ff430 -i td/osd-backfill-space/0/new.json 2021-01-15T17:32:25.047 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:32:25.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:32:25.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCo0QFgQhz+KBAA7fKYCT+3fwAne/0hwTh/6A== --osd-uuid 9860a1fe-1945-49e3-b4ee-2b78703ff430 2021-01-15T17:32:25.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.085+0000 7f725a394080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:25.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.087+0000 7f725a394080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:25.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.088+0000 7f725a394080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:25.089 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.089+0000 7f725a394080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:32:25.089 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.089+0000 7f725a394080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:32:25.090 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.089+0000 7f725a394080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:32:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:25.128+0000 7f725a394080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:32:29.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:32:29.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:32:29.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:32:29.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:32:29.679 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:32:30.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:32:30.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:30.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:32:30.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:32:30.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:32:30.213 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:32:30.238 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:30.237+0000 7f44c8bfc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:30.269 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:30.268+0000 7f44c8bfc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:30.272 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:30.271+0000 7f44c8bfc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:32:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:32:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:32:30.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:32:30.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:32:30.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:30.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:32:30.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:30.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:32:30.614 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:32:30.790 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:30.789+0000 7f44c8bfc080 -1 Falling back to public interface 2021-01-15T17:32:31.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:32:31.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:31.570+0000 7f44c8bfc080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:32:32.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:32:32.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:32.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:32:32.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:32.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:32:32.020 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:32:32.446 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/477500,v1:127.0.0.1:6803/477500] [v2:127.0.0.1:6804/477500,v1:127.0.0.1:6805/477500] exists,up 9860a1fe-1945-49e3-b4ee-2b78703ff430 2021-01-15T17:32:32.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:32:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:32:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:32:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:32:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 1 2021-01-15T17:32:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:32:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:32:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:32:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:32:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:32:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:32:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:32:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:32:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:32:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:32:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:32:32.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:32:32.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:32:32.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:32:32.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:32:32.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:32.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:32.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:32.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:32.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:32:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:32:32.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:32:32.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:32:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:32:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:32:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:32:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:32:32.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:32:32.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=43061b82-5b32-4bea-a751-42f2d74eb4d1 2021-01-15T17:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 43061b82-5b32-4bea-a751-42f2d74eb4d1' 2021-01-15T17:32:32.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:32:32.456 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 43061b82-5b32-4bea-a751-42f2d74eb4d1 2021-01-15T17:32:32.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCw0QFgm8yjGxAAOcdwKDtTiV1BywZWR5/s2g== 2021-01-15T17:32:32.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCw0QFgm8yjGxAAOcdwKDtTiV1BywZWR5/s2g=="}' 2021-01-15T17:32:32.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 43061b82-5b32-4bea-a751-42f2d74eb4d1 -i td/osd-backfill-space/1/new.json 2021-01-15T17:32:32.970 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:32:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:32:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCw0QFgm8yjGxAAOcdwKDtTiV1BywZWR5/s2g== --osd-uuid 43061b82-5b32-4bea-a751-42f2d74eb4d1 2021-01-15T17:32:33.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.008+0000 7f4cac255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:33.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.010+0000 7f4cac255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:33.011 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.011+0000 7f4cac255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:33.012 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.011+0000 7f4cac255080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:32:33.012 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.011+0000 7f4cac255080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:32:33.012 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.011+0000 7f4cac255080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:32:33.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:33.059+0000 7f4cac255080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:32:37.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:32:37.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:32:37.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:32:37.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:32:37.550 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:32:38.084 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:32:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:32:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:32:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:32:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:32:38.109 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:38.109+0000 7f7329bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:38.140 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:38.140+0000 7f7329bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:38.142 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:38.141+0000 7f7329bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:38.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:32:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:32:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:32:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:32:38.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:32:38.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:38.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:32:38.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:38.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:32:38.483 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:32:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:38.656+0000 7f7329bc7080 -1 Falling back to public interface 2021-01-15T17:32:38.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:32:39.459 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:39.458+0000 7f7329bc7080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:32:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:32:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:39.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:32:39.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:39.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:32:39.905 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:32:40.335 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/478772,v1:127.0.0.1:6811/478772] [v2:127.0.0.1:6812/478772,v1:127.0.0.1:6813/478772] exists,up 43061b82-5b32-4bea-a751-42f2d74eb4d1 2021-01-15T17:32:40.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:32:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:32:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:32:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:32:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 2 2021-01-15T17:32:40.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:32:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:32:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:32:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:32:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:32:40.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:32:40.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:32:40.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:32:40.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:32:40.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:32:40.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:32:40.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:32:40.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:32:40.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:32:40.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:32:40.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:40.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:40.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:40.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:40.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:40.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:32:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:32:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:32:40.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:32:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:32:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:32:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:32:40.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:32:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:32:40.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:32:40.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:32:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=96c842f4-e7ed-4a52-a10b-13b29ee8ff2f 2021-01-15T17:32:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 96c842f4-e7ed-4a52-a10b-13b29ee8ff2f' 2021-01-15T17:32:40.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:32:40.345 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 96c842f4-e7ed-4a52-a10b-13b29ee8ff2f 2021-01-15T17:32:40.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC40QFguyURFRAA4qx4BSTy3QDp+YxsUzKJhQ== 2021-01-15T17:32:40.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC40QFguyURFRAA4qx4BSTy3QDp+YxsUzKJhQ=="}' 2021-01-15T17:32:40.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 96c842f4-e7ed-4a52-a10b-13b29ee8ff2f -i td/osd-backfill-space/2/new.json 2021-01-15T17:32:40.903 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:32:40.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:32:40.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC40QFguyURFRAA4qx4BSTy3QDp+YxsUzKJhQ== --osd-uuid 96c842f4-e7ed-4a52-a10b-13b29ee8ff2f 2021-01-15T17:32:40.942 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.942+0000 7fe54e037080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:40.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.944+0000 7fe54e037080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.945+0000 7fe54e037080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:40.946 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.946+0000 7fe54e037080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:32:40.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.946+0000 7fe54e037080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:32:40.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.946+0000 7fe54e037080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:32:40.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:40.990+0000 7fe54e037080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:32:45.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:32:45.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:32:45.600 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:32:45.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:32:45.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:32:46.125 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:32:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:32:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:32:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:32:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:32:46.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:46.150+0000 7f4e84255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:46.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:46.152+0000 7f4e84255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:46.153 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:46.153+0000 7f4e84255080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:46.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:32:46.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:32:46.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:32:46.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:32:46.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:32:46.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:46.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:32:46.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:46.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:32:46.522 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:32:46.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:46.669+0000 7f4e84255080 -1 Falling back to public interface 2021-01-15T17:32:46.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:32:47.459 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:47.457+0000 7f4e84255080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:32:47.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:32:47.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:47.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:32:47.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:47.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:32:47.972 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:32:48.380 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/480036,v1:127.0.0.1:6819/480036] [v2:127.0.0.1:6820/480036,v1:127.0.0.1:6821/480036] exists,up 96c842f4-e7ed-4a52-a10b-13b29ee8ff2f 2021-01-15T17:32:48.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:32:48.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:32:48.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:32:48.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:32:48.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 3 2021-01-15T17:32:48.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:32:48.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:32:48.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:32:48.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:32:48.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2021-01-15T17:32:48.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:32:48.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:32:48.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:32:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:32:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2021-01-15T17:32:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2021-01-15T17:32:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:32:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:32:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:32:48.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:32:48.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:48.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:48.386 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:48.386 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:48.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:48.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:32:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:32:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:32:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:32:48.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:32:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:32:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:32:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:32:48.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:32:48.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2021-01-15T17:32:48.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:32:48.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0746d45f-51a3-4a5b-8057-5936ccad7d73 2021-01-15T17:32:48.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 0746d45f-51a3-4a5b-8057-5936ccad7d73' 2021-01-15T17:32:48.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:32:48.390 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 0746d45f-51a3-4a5b-8057-5936ccad7d73 2021-01-15T17:32:48.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDA0QFgoku5FxAACgQJYyXeeeBOID1AvS1qWA== 2021-01-15T17:32:48.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDA0QFgoku5FxAACgQJYyXeeeBOID1AvS1qWA=="}' 2021-01-15T17:32:48.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0746d45f-51a3-4a5b-8057-5936ccad7d73 -i td/osd-backfill-space/3/new.json 2021-01-15T17:32:48.901 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:32:48.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2021-01-15T17:32:48.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDA0QFgoku5FxAACgQJYyXeeeBOID1AvS1qWA== --osd-uuid 0746d45f-51a3-4a5b-8057-5936ccad7d73 2021-01-15T17:32:48.941 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.941+0000 7fb605c72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.942+0000 7fb605c72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.943+0000 7fb605c72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.944+0000 7fb605c72080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:32:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.944+0000 7fb605c72080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:32:48.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.944+0000 7fb605c72080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:32:48.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:48.988+0000 7fb605c72080 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2021-01-15T17:32:53.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2021-01-15T17:32:53.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:32:53.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:32:53.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:32:53.759 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:32:54.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:32:54.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:54.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:32:54.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:32:54.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:32:54.317 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:32:54.341 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:54.340+0000 7fdbc7536080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:54.365 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:54.364+0000 7fdbc7536080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:54.366 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:54.366+0000 7fdbc7536080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:54.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:32:54.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:32:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:32:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:32:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:32:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:32:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:32:54.719 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:32:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:54.891+0000 7fdbc7536080 -1 Falling back to public interface 2021-01-15T17:32:55.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:32:55.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:55.699+0000 7fdbc7536080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:32:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:32:56.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:32:56.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:32:56.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:32:56.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:32:56.131 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:32:56.538 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/481300,v1:127.0.0.1:6827/481300] [v2:127.0.0.1:6828/481300,v1:127.0.0.1:6829/481300] exists,up 0746d45f-51a3-4a5b-8057-5936ccad7d73 2021-01-15T17:32:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:32:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:32:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:32:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:32:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 4 2021-01-15T17:32:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:32:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:32:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:32:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:32:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2021-01-15T17:32:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:32:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:32:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:32:56.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:32:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2021-01-15T17:32:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2021-01-15T17:32:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:32:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:32:56.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:32:56.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:32:56.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:32:56.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:32:56.543 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:32:56.543 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:32:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:32:56.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:32:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:32:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:32:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:32:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:32:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:32:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:32:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:32:56.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:32:56.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:32:56.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:32:56.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2021-01-15T17:32:56.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:32:56.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=10b02837-6e62-483a-be2b-f57b20aac6fd 2021-01-15T17:32:56.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 10b02837-6e62-483a-be2b-f57b20aac6fd' 2021-01-15T17:32:56.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:32:56.548 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 10b02837-6e62-483a-be2b-f57b20aac6fd 2021-01-15T17:32:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDI0QFgEUcsIRAAsB9lrx9+HzQ6YtNLIOoTng== 2021-01-15T17:32:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDI0QFgEUcsIRAAsB9lrx9+HzQ6YtNLIOoTng=="}' 2021-01-15T17:32:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 10b02837-6e62-483a-be2b-f57b20aac6fd -i td/osd-backfill-space/4/new.json 2021-01-15T17:32:57.094 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:32:57.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2021-01-15T17:32:57.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDI0QFgEUcsIRAAsB9lrx9+HzQ6YtNLIOoTng== --osd-uuid 10b02837-6e62-483a-be2b-f57b20aac6fd 2021-01-15T17:32:57.134 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.133+0000 7f935fa1a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:57.135 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.135+0000 7f935fa1a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:57.136 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.135+0000 7f935fa1a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:32:57.137 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.136+0000 7f935fa1a080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:32:57.137 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.136+0000 7f935fa1a080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:32:57.137 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.136+0000 7f935fa1a080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:32:57.215 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:32:57.215+0000 7f935fa1a080 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2021-01-15T17:33:01.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2021-01-15T17:33:01.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:33:01.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:33:01.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:33:01.967 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:33:02.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:33:02.494 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:33:02.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:33:02.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:33:02.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:33:02.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:33:02.520 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:02.520+0000 7f8b5db1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:02.522 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:02.522+0000 7f8b5db1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:02.523 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:02.523+0000 7f8b5db1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:02.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:33:02.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:33:02.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:33:02.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:33:02.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:33:02.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:33:02.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:33:02.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:33:02.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:33:02.908 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:33:03.043 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:03.043+0000 7f8b5db1b080 -1 Falling back to public interface 2021-01-15T17:33:03.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:33:03.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:03.842+0000 7f8b5db1b080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:33:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:33:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:33:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:33:04.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:33:04.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:33:04.304 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:33:04.707 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/482565,v1:127.0.0.1:6835/482565] [v2:127.0.0.1:6836/482565,v1:127.0.0.1:6837/482565] exists,up 10b02837-6e62-483a-be2b-f57b20aac6fd 2021-01-15T17:33:04.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:33:04.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:33:04.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:33:04.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:760: TEST_ec_backfill_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:04.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:762: TEST_ec_backfill_multi: run_osd td/osd-backfill-space 5 2021-01-15T17:33:04.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:33:04.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:33:04.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:33:04.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:33:04.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2021-01-15T17:33:04.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:33:04.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:33:04.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:33:04.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:33:04.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2021-01-15T17:33:04.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2021-01-15T17:33:04.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:33:04.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:33:04.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:33:04.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:33:04.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:33:04.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:33:04.712 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:33:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:33:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:33:04.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:33:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:33:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:33:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:33:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:33:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:33:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:33:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:33:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:33:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:33:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:33:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:33:04.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2021-01-15T17:33:04.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:33:04.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=cfa1043b-51c4-45dd-b64e-eef3d6984651 2021-01-15T17:33:04.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 cfa1043b-51c4-45dd-b64e-eef3d6984651' 2021-01-15T17:33:04.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:33:04.717 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 cfa1043b-51c4-45dd-b64e-eef3d6984651 2021-01-15T17:33:04.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDQ0QFgqAY2KxAAhzX4cnLd8XeRsg3rTr02QA== 2021-01-15T17:33:04.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDQ0QFgqAY2KxAAhzX4cnLd8XeRsg3rTr02QA=="}' 2021-01-15T17:33:04.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new cfa1043b-51c4-45dd-b64e-eef3d6984651 -i td/osd-backfill-space/5/new.json 2021-01-15T17:33:05.252 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:33:05.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2021-01-15T17:33:05.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQ0QFgqAY2KxAAhzX4cnLd8XeRsg3rTr02QA== --osd-uuid cfa1043b-51c4-45dd-b64e-eef3d6984651 2021-01-15T17:33:05.291 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.291+0000 7fc7f4afd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.307+0000 7fc7f4afd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:05.309 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.308+0000 7fc7f4afd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:05.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.309+0000 7fc7f4afd080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:33:05.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.309+0000 7fc7f4afd080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:33:05.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.309+0000 7fc7f4afd080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:33:05.374 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:05.374+0000 7fc7f4afd080 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2021-01-15T17:33:10.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2021-01-15T17:33:10.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:33:10.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:33:10.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:33:10.215 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:33:10.747 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:33:10.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:33:10.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:33:10.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:33:10.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:33:10.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:33:10.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:10.772+0000 7f5632ee0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:10.800 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:10.799+0000 7f5632ee0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:10.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:10.801+0000 7f5632ee0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:33:11.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:33:11.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:33:11.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:33:11.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:33:11.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:33:11.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:33:11.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:33:11.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:33:11.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:33:11.144 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:33:11.318 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:11.318+0000 7f5632ee0080 -1 Falling back to public interface 2021-01-15T17:33:11.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:33:12.143 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:33:12.143+0000 7f5632ee0080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:33:12.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:33:12.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:33:12.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:33:12.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:33:12.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:33:12.553 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:33:12.955 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/483829,v1:127.0.0.1:6843/483829] [v2:127.0.0.1:6844/483829,v1:127.0.0.1:6845/483829] exists,up cfa1043b-51c4-45dd-b64e-eef3d6984651 2021-01-15T17:33:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:33:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:33:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:33:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:769: TEST_ec_backfill_multi: ceph osd set-backfillfull-ratio .85 2021-01-15T17:33:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:33:13.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:771: TEST_ec_backfill_multi: ceph osd set-require-min-compat-client luminous 2021-01-15T17:33:14.539 INFO:tasks.workunit.client.0.smithi145.stderr:set require_min_compat_client to luminous 2021-01-15T17:33:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:772: TEST_ec_backfill_multi: create_pool fillpool 1 1 2021-01-15T17:33:14.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2021-01-15T17:33:15.127 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' already exists 2021-01-15T17:33:15.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:33:16.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:773: TEST_ec_backfill_multi: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2021-01-15T17:33:16.873 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:33:16.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:781: TEST_ec_backfill_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=9 2021-01-15T17:33:16.884 INFO:tasks.workunit.client.0.smithi145.stderr:9+0 records in 2021-01-15T17:33:16.884 INFO:tasks.workunit.client.0.smithi145.stderr:9+0 records out 2021-01-15T17:33:16.884 INFO:tasks.workunit.client.0.smithi145.stderr:9216 bytes (9.2 kB, 9.0 KiB) copied, 9.4893e-05 s, 97.1 MB/s 2021-01-15T17:33:16.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: seq 1 200 2021-01-15T17:33:16.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:16.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj1 td/osd-backfill-space/datafile 2021-01-15T17:33:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj2 td/osd-backfill-space/datafile 2021-01-15T17:33:17.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:17.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj3 td/osd-backfill-space/datafile 2021-01-15T17:33:17.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:17.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj4 td/osd-backfill-space/datafile 2021-01-15T17:33:17.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:17.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj5 td/osd-backfill-space/datafile 2021-01-15T17:33:17.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:17.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj6 td/osd-backfill-space/datafile 2021-01-15T17:33:18.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj7 td/osd-backfill-space/datafile 2021-01-15T17:33:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj8 td/osd-backfill-space/datafile 2021-01-15T17:33:18.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj9 td/osd-backfill-space/datafile 2021-01-15T17:33:18.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj10 td/osd-backfill-space/datafile 2021-01-15T17:33:18.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj11 td/osd-backfill-space/datafile 2021-01-15T17:33:18.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj12 td/osd-backfill-space/datafile 2021-01-15T17:33:18.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj13 td/osd-backfill-space/datafile 2021-01-15T17:33:18.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj14 td/osd-backfill-space/datafile 2021-01-15T17:33:18.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj15 td/osd-backfill-space/datafile 2021-01-15T17:33:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj16 td/osd-backfill-space/datafile 2021-01-15T17:33:18.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj17 td/osd-backfill-space/datafile 2021-01-15T17:33:18.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:18.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj18 td/osd-backfill-space/datafile 2021-01-15T17:33:19.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj19 td/osd-backfill-space/datafile 2021-01-15T17:33:19.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj20 td/osd-backfill-space/datafile 2021-01-15T17:33:19.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj21 td/osd-backfill-space/datafile 2021-01-15T17:33:19.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj22 td/osd-backfill-space/datafile 2021-01-15T17:33:19.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj23 td/osd-backfill-space/datafile 2021-01-15T17:33:19.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj24 td/osd-backfill-space/datafile 2021-01-15T17:33:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj25 td/osd-backfill-space/datafile 2021-01-15T17:33:19.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj26 td/osd-backfill-space/datafile 2021-01-15T17:33:19.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj27 td/osd-backfill-space/datafile 2021-01-15T17:33:19.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj28 td/osd-backfill-space/datafile 2021-01-15T17:33:19.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj29 td/osd-backfill-space/datafile 2021-01-15T17:33:19.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj30 td/osd-backfill-space/datafile 2021-01-15T17:33:19.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:19.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj31 td/osd-backfill-space/datafile 2021-01-15T17:33:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj32 td/osd-backfill-space/datafile 2021-01-15T17:33:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj33 td/osd-backfill-space/datafile 2021-01-15T17:33:20.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj34 td/osd-backfill-space/datafile 2021-01-15T17:33:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj35 td/osd-backfill-space/datafile 2021-01-15T17:33:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj36 td/osd-backfill-space/datafile 2021-01-15T17:33:20.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj37 td/osd-backfill-space/datafile 2021-01-15T17:33:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj38 td/osd-backfill-space/datafile 2021-01-15T17:33:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj39 td/osd-backfill-space/datafile 2021-01-15T17:33:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj40 td/osd-backfill-space/datafile 2021-01-15T17:33:20.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj41 td/osd-backfill-space/datafile 2021-01-15T17:33:20.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:20.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj42 td/osd-backfill-space/datafile 2021-01-15T17:33:21.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj43 td/osd-backfill-space/datafile 2021-01-15T17:33:21.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj44 td/osd-backfill-space/datafile 2021-01-15T17:33:21.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj45 td/osd-backfill-space/datafile 2021-01-15T17:33:21.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj46 td/osd-backfill-space/datafile 2021-01-15T17:33:21.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj47 td/osd-backfill-space/datafile 2021-01-15T17:33:21.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj48 td/osd-backfill-space/datafile 2021-01-15T17:33:21.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj49 td/osd-backfill-space/datafile 2021-01-15T17:33:21.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj50 td/osd-backfill-space/datafile 2021-01-15T17:33:21.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj51 td/osd-backfill-space/datafile 2021-01-15T17:33:21.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj52 td/osd-backfill-space/datafile 2021-01-15T17:33:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj53 td/osd-backfill-space/datafile 2021-01-15T17:33:21.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj54 td/osd-backfill-space/datafile 2021-01-15T17:33:21.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:21.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj55 td/osd-backfill-space/datafile 2021-01-15T17:33:22.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj56 td/osd-backfill-space/datafile 2021-01-15T17:33:22.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj57 td/osd-backfill-space/datafile 2021-01-15T17:33:22.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj58 td/osd-backfill-space/datafile 2021-01-15T17:33:22.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj59 td/osd-backfill-space/datafile 2021-01-15T17:33:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj60 td/osd-backfill-space/datafile 2021-01-15T17:33:22.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj61 td/osd-backfill-space/datafile 2021-01-15T17:33:22.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj62 td/osd-backfill-space/datafile 2021-01-15T17:33:22.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj63 td/osd-backfill-space/datafile 2021-01-15T17:33:22.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj64 td/osd-backfill-space/datafile 2021-01-15T17:33:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj65 td/osd-backfill-space/datafile 2021-01-15T17:33:22.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj66 td/osd-backfill-space/datafile 2021-01-15T17:33:22.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:22.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj67 td/osd-backfill-space/datafile 2021-01-15T17:33:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj68 td/osd-backfill-space/datafile 2021-01-15T17:33:23.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj69 td/osd-backfill-space/datafile 2021-01-15T17:33:23.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj70 td/osd-backfill-space/datafile 2021-01-15T17:33:23.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj71 td/osd-backfill-space/datafile 2021-01-15T17:33:23.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj72 td/osd-backfill-space/datafile 2021-01-15T17:33:23.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj73 td/osd-backfill-space/datafile 2021-01-15T17:33:23.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj74 td/osd-backfill-space/datafile 2021-01-15T17:33:23.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj75 td/osd-backfill-space/datafile 2021-01-15T17:33:23.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj76 td/osd-backfill-space/datafile 2021-01-15T17:33:23.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj77 td/osd-backfill-space/datafile 2021-01-15T17:33:23.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj78 td/osd-backfill-space/datafile 2021-01-15T17:33:23.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj79 td/osd-backfill-space/datafile 2021-01-15T17:33:23.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:23.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj80 td/osd-backfill-space/datafile 2021-01-15T17:33:24.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj81 td/osd-backfill-space/datafile 2021-01-15T17:33:24.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj82 td/osd-backfill-space/datafile 2021-01-15T17:33:24.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj83 td/osd-backfill-space/datafile 2021-01-15T17:33:24.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj84 td/osd-backfill-space/datafile 2021-01-15T17:33:24.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj85 td/osd-backfill-space/datafile 2021-01-15T17:33:24.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj86 td/osd-backfill-space/datafile 2021-01-15T17:33:24.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj87 td/osd-backfill-space/datafile 2021-01-15T17:33:24.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj88 td/osd-backfill-space/datafile 2021-01-15T17:33:24.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj89 td/osd-backfill-space/datafile 2021-01-15T17:33:24.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj90 td/osd-backfill-space/datafile 2021-01-15T17:33:24.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj91 td/osd-backfill-space/datafile 2021-01-15T17:33:24.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:24.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj92 td/osd-backfill-space/datafile 2021-01-15T17:33:25.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj93 td/osd-backfill-space/datafile 2021-01-15T17:33:25.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj94 td/osd-backfill-space/datafile 2021-01-15T17:33:25.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj95 td/osd-backfill-space/datafile 2021-01-15T17:33:25.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj96 td/osd-backfill-space/datafile 2021-01-15T17:33:25.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj97 td/osd-backfill-space/datafile 2021-01-15T17:33:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj98 td/osd-backfill-space/datafile 2021-01-15T17:33:25.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj99 td/osd-backfill-space/datafile 2021-01-15T17:33:25.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj100 td/osd-backfill-space/datafile 2021-01-15T17:33:25.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj101 td/osd-backfill-space/datafile 2021-01-15T17:33:25.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj102 td/osd-backfill-space/datafile 2021-01-15T17:33:25.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj103 td/osd-backfill-space/datafile 2021-01-15T17:33:25.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:25.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj104 td/osd-backfill-space/datafile 2021-01-15T17:33:26.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj105 td/osd-backfill-space/datafile 2021-01-15T17:33:26.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj106 td/osd-backfill-space/datafile 2021-01-15T17:33:26.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj107 td/osd-backfill-space/datafile 2021-01-15T17:33:26.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj108 td/osd-backfill-space/datafile 2021-01-15T17:33:26.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj109 td/osd-backfill-space/datafile 2021-01-15T17:33:26.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj110 td/osd-backfill-space/datafile 2021-01-15T17:33:26.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj111 td/osd-backfill-space/datafile 2021-01-15T17:33:26.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj112 td/osd-backfill-space/datafile 2021-01-15T17:33:26.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj113 td/osd-backfill-space/datafile 2021-01-15T17:33:26.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj114 td/osd-backfill-space/datafile 2021-01-15T17:33:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj115 td/osd-backfill-space/datafile 2021-01-15T17:33:26.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:26.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj116 td/osd-backfill-space/datafile 2021-01-15T17:33:27.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj117 td/osd-backfill-space/datafile 2021-01-15T17:33:27.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj118 td/osd-backfill-space/datafile 2021-01-15T17:33:27.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj119 td/osd-backfill-space/datafile 2021-01-15T17:33:27.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj120 td/osd-backfill-space/datafile 2021-01-15T17:33:27.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj121 td/osd-backfill-space/datafile 2021-01-15T17:33:27.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj122 td/osd-backfill-space/datafile 2021-01-15T17:33:27.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj123 td/osd-backfill-space/datafile 2021-01-15T17:33:27.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj124 td/osd-backfill-space/datafile 2021-01-15T17:33:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj125 td/osd-backfill-space/datafile 2021-01-15T17:33:27.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj126 td/osd-backfill-space/datafile 2021-01-15T17:33:27.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj127 td/osd-backfill-space/datafile 2021-01-15T17:33:27.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj128 td/osd-backfill-space/datafile 2021-01-15T17:33:27.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:27.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj129 td/osd-backfill-space/datafile 2021-01-15T17:33:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj130 td/osd-backfill-space/datafile 2021-01-15T17:33:28.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj131 td/osd-backfill-space/datafile 2021-01-15T17:33:28.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj132 td/osd-backfill-space/datafile 2021-01-15T17:33:28.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj133 td/osd-backfill-space/datafile 2021-01-15T17:33:28.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj134 td/osd-backfill-space/datafile 2021-01-15T17:33:28.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj135 td/osd-backfill-space/datafile 2021-01-15T17:33:28.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj136 td/osd-backfill-space/datafile 2021-01-15T17:33:28.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj137 td/osd-backfill-space/datafile 2021-01-15T17:33:28.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj138 td/osd-backfill-space/datafile 2021-01-15T17:33:28.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj139 td/osd-backfill-space/datafile 2021-01-15T17:33:28.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj140 td/osd-backfill-space/datafile 2021-01-15T17:33:28.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:28.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj141 td/osd-backfill-space/datafile 2021-01-15T17:33:29.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj142 td/osd-backfill-space/datafile 2021-01-15T17:33:29.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj143 td/osd-backfill-space/datafile 2021-01-15T17:33:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj144 td/osd-backfill-space/datafile 2021-01-15T17:33:29.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj145 td/osd-backfill-space/datafile 2021-01-15T17:33:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj146 td/osd-backfill-space/datafile 2021-01-15T17:33:29.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj147 td/osd-backfill-space/datafile 2021-01-15T17:33:29.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj148 td/osd-backfill-space/datafile 2021-01-15T17:33:29.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj149 td/osd-backfill-space/datafile 2021-01-15T17:33:29.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj150 td/osd-backfill-space/datafile 2021-01-15T17:33:29.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj151 td/osd-backfill-space/datafile 2021-01-15T17:33:29.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj152 td/osd-backfill-space/datafile 2021-01-15T17:33:29.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj153 td/osd-backfill-space/datafile 2021-01-15T17:33:29.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:29.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj154 td/osd-backfill-space/datafile 2021-01-15T17:33:30.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj155 td/osd-backfill-space/datafile 2021-01-15T17:33:30.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj156 td/osd-backfill-space/datafile 2021-01-15T17:33:30.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj157 td/osd-backfill-space/datafile 2021-01-15T17:33:30.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj158 td/osd-backfill-space/datafile 2021-01-15T17:33:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj159 td/osd-backfill-space/datafile 2021-01-15T17:33:30.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj160 td/osd-backfill-space/datafile 2021-01-15T17:33:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj161 td/osd-backfill-space/datafile 2021-01-15T17:33:30.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj162 td/osd-backfill-space/datafile 2021-01-15T17:33:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj163 td/osd-backfill-space/datafile 2021-01-15T17:33:30.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj164 td/osd-backfill-space/datafile 2021-01-15T17:33:30.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:30.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj165 td/osd-backfill-space/datafile 2021-01-15T17:33:31.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj166 td/osd-backfill-space/datafile 2021-01-15T17:33:31.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj167 td/osd-backfill-space/datafile 2021-01-15T17:33:31.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj168 td/osd-backfill-space/datafile 2021-01-15T17:33:31.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj169 td/osd-backfill-space/datafile 2021-01-15T17:33:31.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj170 td/osd-backfill-space/datafile 2021-01-15T17:33:31.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj171 td/osd-backfill-space/datafile 2021-01-15T17:33:31.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj172 td/osd-backfill-space/datafile 2021-01-15T17:33:31.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj173 td/osd-backfill-space/datafile 2021-01-15T17:33:31.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj174 td/osd-backfill-space/datafile 2021-01-15T17:33:31.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj175 td/osd-backfill-space/datafile 2021-01-15T17:33:31.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj176 td/osd-backfill-space/datafile 2021-01-15T17:33:31.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj177 td/osd-backfill-space/datafile 2021-01-15T17:33:31.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj178 td/osd-backfill-space/datafile 2021-01-15T17:33:31.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:31.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj179 td/osd-backfill-space/datafile 2021-01-15T17:33:32.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj180 td/osd-backfill-space/datafile 2021-01-15T17:33:32.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj181 td/osd-backfill-space/datafile 2021-01-15T17:33:32.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj182 td/osd-backfill-space/datafile 2021-01-15T17:33:32.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj183 td/osd-backfill-space/datafile 2021-01-15T17:33:32.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj184 td/osd-backfill-space/datafile 2021-01-15T17:33:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj185 td/osd-backfill-space/datafile 2021-01-15T17:33:32.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj186 td/osd-backfill-space/datafile 2021-01-15T17:33:32.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj187 td/osd-backfill-space/datafile 2021-01-15T17:33:32.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj188 td/osd-backfill-space/datafile 2021-01-15T17:33:32.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj189 td/osd-backfill-space/datafile 2021-01-15T17:33:32.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj190 td/osd-backfill-space/datafile 2021-01-15T17:33:32.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:32.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj191 td/osd-backfill-space/datafile 2021-01-15T17:33:33.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj192 td/osd-backfill-space/datafile 2021-01-15T17:33:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj193 td/osd-backfill-space/datafile 2021-01-15T17:33:33.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj194 td/osd-backfill-space/datafile 2021-01-15T17:33:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj195 td/osd-backfill-space/datafile 2021-01-15T17:33:33.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj196 td/osd-backfill-space/datafile 2021-01-15T17:33:33.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj197 td/osd-backfill-space/datafile 2021-01-15T17:33:33.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj198 td/osd-backfill-space/datafile 2021-01-15T17:33:33.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj199 td/osd-backfill-space/datafile 2021-01-15T17:33:33.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:782: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:33.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:784: TEST_ec_backfill_multi: rados -p fillpool put obj200 td/osd-backfill-space/datafile 2021-01-15T17:33:33.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:787: TEST_ec_backfill_multi: get_primary fillpool obj1 2021-01-15T17:33:33.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2021-01-15T17:33:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:33:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj1 2021-01-15T17:33:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:33:34.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:787: TEST_ec_backfill_multi: local fillosd=1 2021-01-15T17:33:34.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:788: TEST_ec_backfill_multi: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2021-01-15T17:33:34.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:790: TEST_ec_backfill_multi: osdlist 6 1 2021-01-15T17:33:34.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:723: osdlist: local OSDS=6 2021-01-15T17:33:34.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:724: osdlist: local excludeosd=1 2021-01-15T17:33:34.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:726: osdlist: osds= 2021-01-15T17:33:34.708 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: expr 6 - 1 2021-01-15T17:33:34.708 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: seq 0 5 2021-01-15T17:33:34.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 0 = 1 ']' 2021-01-15T17:33:34.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '' ']' 2021-01-15T17:33:34.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds=0 2021-01-15T17:33:34.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 1 = 1 ']' 2021-01-15T17:33:34.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:731: osdlist: continue 2021-01-15T17:33:34.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 2 = 1 ']' 2021-01-15T17:33:34.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n 0 ']' 2021-01-15T17:33:34.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 ' 2021-01-15T17:33:34.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2' 2021-01-15T17:33:34.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 3 = 1 ']' 2021-01-15T17:33:34.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2' ']' 2021-01-15T17:33:34.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 ' 2021-01-15T17:33:34.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3' 2021-01-15T17:33:34.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 4 = 1 ']' 2021-01-15T17:33:34.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2 3' ']' 2021-01-15T17:33:34.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 3 ' 2021-01-15T17:33:34.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3 4' 2021-01-15T17:33:34.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:727: osdlist: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:33:34.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:729: osdlist: '[' 5 = 1 ']' 2021-01-15T17:33:34.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:733: osdlist: '[' -n '0 2 3 4' ']' 2021-01-15T17:33:34.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:734: osdlist: osds='0 2 3 4 ' 2021-01-15T17:33:34.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:736: osdlist: osds='0 2 3 4 5' 2021-01-15T17:33:34.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:738: osdlist: echo 0 2 3 4 5 2021-01-15T17:33:34.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:790: TEST_ec_backfill_multi: nonfillosds='0 2 3 4 5' 2021-01-15T17:33:34.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:34.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:34.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:794: TEST_ec_backfill_multi: ceph osd pool create test1 1 1 erasure ec-profile 2021-01-15T17:33:35.433 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T17:33:35.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: expr 1 + 1 2021-01-15T17:33:35.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: ceph osd pg-upmap 2.0 0 2 3 4 5 2021-01-15T17:33:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:set 2.0 pg_upmap mapping to [0,2,3,4,5] 2021-01-15T17:33:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:792: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:794: TEST_ec_backfill_multi: ceph osd pool create test2 1 1 erasure ec-profile 2021-01-15T17:33:37.206 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T17:33:37.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: expr 2 + 1 2021-01-15T17:33:37.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:795: TEST_ec_backfill_multi: ceph osd pg-upmap 3.0 0 2 3 4 5 2021-01-15T17:33:38.049 INFO:tasks.workunit.client.0.smithi145.stderr:set 3.0 pg_upmap mapping to [0,2,3,4,5] 2021-01-15T17:33:38.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:800: TEST_ec_backfill_multi: sleep 15 2021-01-15T17:33:53.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:802: TEST_ec_backfill_multi: ceph pg dump pgs 2021-01-15T17:33:53.439 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:33:53.439 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:33:39.279834+0000 0'0 52:13 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2021-01-15T17:33:36.872099+0000 0'0 2021-01-15T17:33:36.872099+0000 0 2021-01-15T17:33:53.439 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2021-01-15T17:33:17.349256+0000 40'200 52:227 [1] 1 [1] 1 0'0 2021-01-15T17:33:14.912061+0000 0'0 2021-01-15T17:33:14.912061+0000 0 2021-01-15T17:33:53.440 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T17:33:37.334304+0000 0'0 52:17 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2021-01-15T17:33:35.212946+0000 0'0 2021-01-15T17:33:35.212946+0000 0 2021-01-15T17:33:53.440 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:33:53.440 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:33:53.441 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:33:53.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:804: TEST_ec_backfill_multi: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=12 2021-01-15T17:33:53.451 INFO:tasks.workunit.client.0.smithi145.stderr:12+0 records in 2021-01-15T17:33:53.451 INFO:tasks.workunit.client.0.smithi145.stderr:12+0 records out 2021-01-15T17:33:53.451 INFO:tasks.workunit.client.0.smithi145.stderr:12288 bytes (12 kB, 12 KiB) copied, 0.000149349 s, 82.3 MB/s 2021-01-15T17:33:53.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: seq 1 200 2021-01-15T17:33:53.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:53.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:53.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj1-1 td/osd-backfill-space/datafile 2021-01-15T17:33:53.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:53.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj1-2 td/osd-backfill-space/datafile 2021-01-15T17:33:53.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:53.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:53.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:53.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj2-1 td/osd-backfill-space/datafile 2021-01-15T17:33:54.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:54.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj2-2 td/osd-backfill-space/datafile 2021-01-15T17:33:54.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:54.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:54.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj3-1 td/osd-backfill-space/datafile 2021-01-15T17:33:54.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:54.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj3-2 td/osd-backfill-space/datafile 2021-01-15T17:33:54.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:54.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:54.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:54.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj4-1 td/osd-backfill-space/datafile 2021-01-15T17:33:55.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:55.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj4-2 td/osd-backfill-space/datafile 2021-01-15T17:33:55.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:55.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:55.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:55.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj5-1 td/osd-backfill-space/datafile 2021-01-15T17:33:55.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:55.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj5-2 td/osd-backfill-space/datafile 2021-01-15T17:33:56.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:56.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:56.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:56.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj6-1 td/osd-backfill-space/datafile 2021-01-15T17:33:56.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:56.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj6-2 td/osd-backfill-space/datafile 2021-01-15T17:33:56.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:56.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:56.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:56.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj7-1 td/osd-backfill-space/datafile 2021-01-15T17:33:56.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:56.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj7-2 td/osd-backfill-space/datafile 2021-01-15T17:33:56.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:56.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:56.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:56.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj8-1 td/osd-backfill-space/datafile 2021-01-15T17:33:57.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:57.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj8-2 td/osd-backfill-space/datafile 2021-01-15T17:33:57.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:57.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:57.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:57.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj9-1 td/osd-backfill-space/datafile 2021-01-15T17:33:57.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:57.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj9-2 td/osd-backfill-space/datafile 2021-01-15T17:33:57.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:57.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:57.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:57.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj10-1 td/osd-backfill-space/datafile 2021-01-15T17:33:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:58.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj10-2 td/osd-backfill-space/datafile 2021-01-15T17:33:58.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:58.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:58.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:58.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj11-1 td/osd-backfill-space/datafile 2021-01-15T17:33:58.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:58.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj11-2 td/osd-backfill-space/datafile 2021-01-15T17:33:58.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:58.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:58.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:58.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj12-1 td/osd-backfill-space/datafile 2021-01-15T17:33:59.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:59.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj12-2 td/osd-backfill-space/datafile 2021-01-15T17:33:59.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:59.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:59.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:59.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj13-1 td/osd-backfill-space/datafile 2021-01-15T17:33:59.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:59.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj13-2 td/osd-backfill-space/datafile 2021-01-15T17:33:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:33:59.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:33:59.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:33:59.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj14-1 td/osd-backfill-space/datafile 2021-01-15T17:34:00.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:00.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj14-2 td/osd-backfill-space/datafile 2021-01-15T17:34:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:00.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj15-1 td/osd-backfill-space/datafile 2021-01-15T17:34:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj15-2 td/osd-backfill-space/datafile 2021-01-15T17:34:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:00.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:00.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:00.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj16-1 td/osd-backfill-space/datafile 2021-01-15T17:34:01.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:01.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj16-2 td/osd-backfill-space/datafile 2021-01-15T17:34:01.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:01.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:01.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:01.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj17-1 td/osd-backfill-space/datafile 2021-01-15T17:34:01.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:01.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj17-2 td/osd-backfill-space/datafile 2021-01-15T17:34:01.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:01.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:01.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:01.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj18-1 td/osd-backfill-space/datafile 2021-01-15T17:34:02.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:02.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj18-2 td/osd-backfill-space/datafile 2021-01-15T17:34:02.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:02.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:02.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:02.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj19-1 td/osd-backfill-space/datafile 2021-01-15T17:34:02.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:02.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj19-2 td/osd-backfill-space/datafile 2021-01-15T17:34:02.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:02.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:02.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:02.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj20-1 td/osd-backfill-space/datafile 2021-01-15T17:34:03.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:03.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj20-2 td/osd-backfill-space/datafile 2021-01-15T17:34:03.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:03.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:03.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:03.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj21-1 td/osd-backfill-space/datafile 2021-01-15T17:34:03.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:03.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj21-2 td/osd-backfill-space/datafile 2021-01-15T17:34:03.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:03.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:03.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:03.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj22-1 td/osd-backfill-space/datafile 2021-01-15T17:34:04.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:04.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj22-2 td/osd-backfill-space/datafile 2021-01-15T17:34:04.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:04.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:04.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:04.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj23-1 td/osd-backfill-space/datafile 2021-01-15T17:34:04.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:04.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj23-2 td/osd-backfill-space/datafile 2021-01-15T17:34:04.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:04.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:04.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:04.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj24-1 td/osd-backfill-space/datafile 2021-01-15T17:34:05.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:05.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj24-2 td/osd-backfill-space/datafile 2021-01-15T17:34:05.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:05.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:05.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:05.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj25-1 td/osd-backfill-space/datafile 2021-01-15T17:34:05.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:05.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj25-2 td/osd-backfill-space/datafile 2021-01-15T17:34:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:05.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:05.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:05.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj26-1 td/osd-backfill-space/datafile 2021-01-15T17:34:06.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:06.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj26-2 td/osd-backfill-space/datafile 2021-01-15T17:34:06.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:06.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:06.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj27-1 td/osd-backfill-space/datafile 2021-01-15T17:34:06.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:06.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj27-2 td/osd-backfill-space/datafile 2021-01-15T17:34:06.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:06.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:06.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:06.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj28-1 td/osd-backfill-space/datafile 2021-01-15T17:34:07.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:07.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj28-2 td/osd-backfill-space/datafile 2021-01-15T17:34:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:07.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:07.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:07.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj29-1 td/osd-backfill-space/datafile 2021-01-15T17:34:07.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:07.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj29-2 td/osd-backfill-space/datafile 2021-01-15T17:34:07.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:07.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj30-1 td/osd-backfill-space/datafile 2021-01-15T17:34:07.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:07.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj30-2 td/osd-backfill-space/datafile 2021-01-15T17:34:08.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:08.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:08.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:08.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj31-1 td/osd-backfill-space/datafile 2021-01-15T17:34:08.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:08.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj31-2 td/osd-backfill-space/datafile 2021-01-15T17:34:08.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:08.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:08.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:08.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj32-1 td/osd-backfill-space/datafile 2021-01-15T17:34:09.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:09.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj32-2 td/osd-backfill-space/datafile 2021-01-15T17:34:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:09.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:09.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:09.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj33-1 td/osd-backfill-space/datafile 2021-01-15T17:34:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:09.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj33-2 td/osd-backfill-space/datafile 2021-01-15T17:34:09.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:09.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:09.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:09.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj34-1 td/osd-backfill-space/datafile 2021-01-15T17:34:09.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:09.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj34-2 td/osd-backfill-space/datafile 2021-01-15T17:34:10.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:10.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:10.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:10.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj35-1 td/osd-backfill-space/datafile 2021-01-15T17:34:10.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:10.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj35-2 td/osd-backfill-space/datafile 2021-01-15T17:34:10.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:10.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:10.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:10.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj36-1 td/osd-backfill-space/datafile 2021-01-15T17:34:10.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:10.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj36-2 td/osd-backfill-space/datafile 2021-01-15T17:34:11.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:11.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:11.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:11.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj37-1 td/osd-backfill-space/datafile 2021-01-15T17:34:11.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:11.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj37-2 td/osd-backfill-space/datafile 2021-01-15T17:34:11.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:11.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:11.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:11.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj38-1 td/osd-backfill-space/datafile 2021-01-15T17:34:11.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:11.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj38-2 td/osd-backfill-space/datafile 2021-01-15T17:34:12.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj39-1 td/osd-backfill-space/datafile 2021-01-15T17:34:12.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:12.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj39-2 td/osd-backfill-space/datafile 2021-01-15T17:34:12.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:12.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:12.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj40-1 td/osd-backfill-space/datafile 2021-01-15T17:34:12.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:12.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj40-2 td/osd-backfill-space/datafile 2021-01-15T17:34:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:13.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj41-1 td/osd-backfill-space/datafile 2021-01-15T17:34:13.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:13.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj41-2 td/osd-backfill-space/datafile 2021-01-15T17:34:13.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:13.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:13.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:13.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj42-1 td/osd-backfill-space/datafile 2021-01-15T17:34:13.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:13.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj42-2 td/osd-backfill-space/datafile 2021-01-15T17:34:14.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:14.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:14.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:14.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj43-1 td/osd-backfill-space/datafile 2021-01-15T17:34:14.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj43-2 td/osd-backfill-space/datafile 2021-01-15T17:34:14.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:14.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj44-1 td/osd-backfill-space/datafile 2021-01-15T17:34:14.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:14.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj44-2 td/osd-backfill-space/datafile 2021-01-15T17:34:15.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:15.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:15.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:15.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj45-1 td/osd-backfill-space/datafile 2021-01-15T17:34:15.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:15.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj45-2 td/osd-backfill-space/datafile 2021-01-15T17:34:15.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:15.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj46-1 td/osd-backfill-space/datafile 2021-01-15T17:34:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj46-2 td/osd-backfill-space/datafile 2021-01-15T17:34:16.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:16.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj47-1 td/osd-backfill-space/datafile 2021-01-15T17:34:16.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:16.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj47-2 td/osd-backfill-space/datafile 2021-01-15T17:34:16.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:16.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:16.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:16.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj48-1 td/osd-backfill-space/datafile 2021-01-15T17:34:16.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj48-2 td/osd-backfill-space/datafile 2021-01-15T17:34:17.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:17.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj49-1 td/osd-backfill-space/datafile 2021-01-15T17:34:17.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:17.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj49-2 td/osd-backfill-space/datafile 2021-01-15T17:34:17.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:17.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:17.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:17.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj50-1 td/osd-backfill-space/datafile 2021-01-15T17:34:17.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:17.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj50-2 td/osd-backfill-space/datafile 2021-01-15T17:34:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:18.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:18.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:18.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj51-1 td/osd-backfill-space/datafile 2021-01-15T17:34:18.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:18.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj51-2 td/osd-backfill-space/datafile 2021-01-15T17:34:18.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:18.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:18.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:18.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj52-1 td/osd-backfill-space/datafile 2021-01-15T17:34:18.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:18.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj52-2 td/osd-backfill-space/datafile 2021-01-15T17:34:19.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:19.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:19.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:19.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj53-1 td/osd-backfill-space/datafile 2021-01-15T17:34:19.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:19.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj53-2 td/osd-backfill-space/datafile 2021-01-15T17:34:19.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:19.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:19.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:19.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj54-1 td/osd-backfill-space/datafile 2021-01-15T17:34:19.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:19.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj54-2 td/osd-backfill-space/datafile 2021-01-15T17:34:20.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:20.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:20.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:20.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj55-1 td/osd-backfill-space/datafile 2021-01-15T17:34:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj55-2 td/osd-backfill-space/datafile 2021-01-15T17:34:20.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:20.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj56-1 td/osd-backfill-space/datafile 2021-01-15T17:34:20.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:20.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj56-2 td/osd-backfill-space/datafile 2021-01-15T17:34:21.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:21.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:21.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:21.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj57-1 td/osd-backfill-space/datafile 2021-01-15T17:34:21.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:21.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj57-2 td/osd-backfill-space/datafile 2021-01-15T17:34:21.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:21.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:21.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj58-1 td/osd-backfill-space/datafile 2021-01-15T17:34:21.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:21.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj58-2 td/osd-backfill-space/datafile 2021-01-15T17:34:21.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:21.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:21.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:21.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj59-1 td/osd-backfill-space/datafile 2021-01-15T17:34:22.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:22.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj59-2 td/osd-backfill-space/datafile 2021-01-15T17:34:22.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:22.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:22.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:22.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj60-1 td/osd-backfill-space/datafile 2021-01-15T17:34:22.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:22.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj60-2 td/osd-backfill-space/datafile 2021-01-15T17:34:23.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:23.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:23.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:23.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj61-1 td/osd-backfill-space/datafile 2021-01-15T17:34:23.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:23.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj61-2 td/osd-backfill-space/datafile 2021-01-15T17:34:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:23.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:23.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:23.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj62-1 td/osd-backfill-space/datafile 2021-01-15T17:34:23.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:23.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj62-2 td/osd-backfill-space/datafile 2021-01-15T17:34:23.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:23.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:23.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:23.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj63-1 td/osd-backfill-space/datafile 2021-01-15T17:34:24.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:24.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj63-2 td/osd-backfill-space/datafile 2021-01-15T17:34:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:24.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:24.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:24.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj64-1 td/osd-backfill-space/datafile 2021-01-15T17:34:24.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:24.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj64-2 td/osd-backfill-space/datafile 2021-01-15T17:34:25.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:25.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:25.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:25.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj65-1 td/osd-backfill-space/datafile 2021-01-15T17:34:25.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:25.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj65-2 td/osd-backfill-space/datafile 2021-01-15T17:34:25.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:25.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:25.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:25.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj66-1 td/osd-backfill-space/datafile 2021-01-15T17:34:25.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:25.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj66-2 td/osd-backfill-space/datafile 2021-01-15T17:34:25.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:25.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:25.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:25.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj67-1 td/osd-backfill-space/datafile 2021-01-15T17:34:26.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:26.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj67-2 td/osd-backfill-space/datafile 2021-01-15T17:34:26.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:26.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:26.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:26.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj68-1 td/osd-backfill-space/datafile 2021-01-15T17:34:26.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:26.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj68-2 td/osd-backfill-space/datafile 2021-01-15T17:34:27.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:27.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:27.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:27.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj69-1 td/osd-backfill-space/datafile 2021-01-15T17:34:27.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:27.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj69-2 td/osd-backfill-space/datafile 2021-01-15T17:34:27.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:27.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:27.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:27.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj70-1 td/osd-backfill-space/datafile 2021-01-15T17:34:27.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:27.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj70-2 td/osd-backfill-space/datafile 2021-01-15T17:34:27.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:27.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:27.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:27.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj71-1 td/osd-backfill-space/datafile 2021-01-15T17:34:28.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:28.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj71-2 td/osd-backfill-space/datafile 2021-01-15T17:34:28.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:28.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:28.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:28.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj72-1 td/osd-backfill-space/datafile 2021-01-15T17:34:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj72-2 td/osd-backfill-space/datafile 2021-01-15T17:34:29.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:29.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:29.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:29.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj73-1 td/osd-backfill-space/datafile 2021-01-15T17:34:29.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:29.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj73-2 td/osd-backfill-space/datafile 2021-01-15T17:34:29.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:29.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:29.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:29.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj74-1 td/osd-backfill-space/datafile 2021-01-15T17:34:29.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:29.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj74-2 td/osd-backfill-space/datafile 2021-01-15T17:34:30.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:30.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:30.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj75-1 td/osd-backfill-space/datafile 2021-01-15T17:34:30.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj75-2 td/osd-backfill-space/datafile 2021-01-15T17:34:30.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:30.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:30.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:30.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj76-1 td/osd-backfill-space/datafile 2021-01-15T17:34:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj76-2 td/osd-backfill-space/datafile 2021-01-15T17:34:30.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:30.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:30.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:30.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj77-1 td/osd-backfill-space/datafile 2021-01-15T17:34:31.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:31.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj77-2 td/osd-backfill-space/datafile 2021-01-15T17:34:31.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:31.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:31.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:31.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj78-1 td/osd-backfill-space/datafile 2021-01-15T17:34:31.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:31.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj78-2 td/osd-backfill-space/datafile 2021-01-15T17:34:32.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:32.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:32.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:32.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj79-1 td/osd-backfill-space/datafile 2021-01-15T17:34:32.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:32.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj79-2 td/osd-backfill-space/datafile 2021-01-15T17:34:32.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:32.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:32.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:32.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj80-1 td/osd-backfill-space/datafile 2021-01-15T17:34:32.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:32.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj80-2 td/osd-backfill-space/datafile 2021-01-15T17:34:33.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:33.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:33.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:33.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj81-1 td/osd-backfill-space/datafile 2021-01-15T17:34:33.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:33.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj81-2 td/osd-backfill-space/datafile 2021-01-15T17:34:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:33.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:33.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:33.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj82-1 td/osd-backfill-space/datafile 2021-01-15T17:34:33.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj82-2 td/osd-backfill-space/datafile 2021-01-15T17:34:33.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:33.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj83-1 td/osd-backfill-space/datafile 2021-01-15T17:34:34.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:34.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj83-2 td/osd-backfill-space/datafile 2021-01-15T17:34:34.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:34.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:34.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:34.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj84-1 td/osd-backfill-space/datafile 2021-01-15T17:34:34.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:34.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj84-2 td/osd-backfill-space/datafile 2021-01-15T17:34:35.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:35.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:35.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:35.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj85-1 td/osd-backfill-space/datafile 2021-01-15T17:34:35.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:35.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj85-2 td/osd-backfill-space/datafile 2021-01-15T17:34:35.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:35.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:35.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:35.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj86-1 td/osd-backfill-space/datafile 2021-01-15T17:34:35.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:35.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj86-2 td/osd-backfill-space/datafile 2021-01-15T17:34:36.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:36.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:36.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:36.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj87-1 td/osd-backfill-space/datafile 2021-01-15T17:34:36.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:36.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj87-2 td/osd-backfill-space/datafile 2021-01-15T17:34:36.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:36.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj88-1 td/osd-backfill-space/datafile 2021-01-15T17:34:36.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:36.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj88-2 td/osd-backfill-space/datafile 2021-01-15T17:34:37.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:37.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:37.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:37.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj89-1 td/osd-backfill-space/datafile 2021-01-15T17:34:37.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:37.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj89-2 td/osd-backfill-space/datafile 2021-01-15T17:34:37.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:37.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:37.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:37.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj90-1 td/osd-backfill-space/datafile 2021-01-15T17:34:37.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:37.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj90-2 td/osd-backfill-space/datafile 2021-01-15T17:34:38.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:38.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:38.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:38.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj91-1 td/osd-backfill-space/datafile 2021-01-15T17:34:38.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:38.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj91-2 td/osd-backfill-space/datafile 2021-01-15T17:34:38.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:38.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:38.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj92-1 td/osd-backfill-space/datafile 2021-01-15T17:34:38.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:38.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj92-2 td/osd-backfill-space/datafile 2021-01-15T17:34:38.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:38.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:38.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:38.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj93-1 td/osd-backfill-space/datafile 2021-01-15T17:34:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:39.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj93-2 td/osd-backfill-space/datafile 2021-01-15T17:34:39.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:39.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:39.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:39.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj94-1 td/osd-backfill-space/datafile 2021-01-15T17:34:39.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj94-2 td/osd-backfill-space/datafile 2021-01-15T17:34:39.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:39.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:39.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:39.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj95-1 td/osd-backfill-space/datafile 2021-01-15T17:34:40.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:40.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj95-2 td/osd-backfill-space/datafile 2021-01-15T17:34:40.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:40.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:40.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:40.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj96-1 td/osd-backfill-space/datafile 2021-01-15T17:34:40.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:40.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj96-2 td/osd-backfill-space/datafile 2021-01-15T17:34:40.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:40.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:40.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:40.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj97-1 td/osd-backfill-space/datafile 2021-01-15T17:34:41.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:41.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj97-2 td/osd-backfill-space/datafile 2021-01-15T17:34:41.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:41.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:41.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:41.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj98-1 td/osd-backfill-space/datafile 2021-01-15T17:34:41.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:41.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj98-2 td/osd-backfill-space/datafile 2021-01-15T17:34:41.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:41.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:41.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:41.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj99-1 td/osd-backfill-space/datafile 2021-01-15T17:34:42.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:42.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj99-2 td/osd-backfill-space/datafile 2021-01-15T17:34:42.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:42.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:42.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:42.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj100-1 td/osd-backfill-space/datafile 2021-01-15T17:34:42.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:42.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj100-2 td/osd-backfill-space/datafile 2021-01-15T17:34:42.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:42.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:42.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:42.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj101-1 td/osd-backfill-space/datafile 2021-01-15T17:34:43.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:43.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj101-2 td/osd-backfill-space/datafile 2021-01-15T17:34:43.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:43.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:43.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:43.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj102-1 td/osd-backfill-space/datafile 2021-01-15T17:34:43.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:43.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj102-2 td/osd-backfill-space/datafile 2021-01-15T17:34:43.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:43.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:43.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:43.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj103-1 td/osd-backfill-space/datafile 2021-01-15T17:34:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:44.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj103-2 td/osd-backfill-space/datafile 2021-01-15T17:34:44.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:44.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:44.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:44.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj104-1 td/osd-backfill-space/datafile 2021-01-15T17:34:44.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:44.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj104-2 td/osd-backfill-space/datafile 2021-01-15T17:34:44.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:44.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:44.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:44.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj105-1 td/osd-backfill-space/datafile 2021-01-15T17:34:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj105-2 td/osd-backfill-space/datafile 2021-01-15T17:34:45.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:45.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:45.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:45.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj106-1 td/osd-backfill-space/datafile 2021-01-15T17:34:45.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:45.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj106-2 td/osd-backfill-space/datafile 2021-01-15T17:34:45.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:45.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:45.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:45.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj107-1 td/osd-backfill-space/datafile 2021-01-15T17:34:45.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:45.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj107-2 td/osd-backfill-space/datafile 2021-01-15T17:34:46.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:46.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:46.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:46.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj108-1 td/osd-backfill-space/datafile 2021-01-15T17:34:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj108-2 td/osd-backfill-space/datafile 2021-01-15T17:34:46.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:46.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:46.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:46.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj109-1 td/osd-backfill-space/datafile 2021-01-15T17:34:46.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:46.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj109-2 td/osd-backfill-space/datafile 2021-01-15T17:34:47.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:47.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:47.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:47.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj110-1 td/osd-backfill-space/datafile 2021-01-15T17:34:47.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:47.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj110-2 td/osd-backfill-space/datafile 2021-01-15T17:34:47.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:47.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:47.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:47.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj111-1 td/osd-backfill-space/datafile 2021-01-15T17:34:47.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:47.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj111-2 td/osd-backfill-space/datafile 2021-01-15T17:34:48.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:48.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:48.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:48.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj112-1 td/osd-backfill-space/datafile 2021-01-15T17:34:48.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:48.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj112-2 td/osd-backfill-space/datafile 2021-01-15T17:34:48.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:48.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:48.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:48.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj113-1 td/osd-backfill-space/datafile 2021-01-15T17:34:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj113-2 td/osd-backfill-space/datafile 2021-01-15T17:34:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:49.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj114-1 td/osd-backfill-space/datafile 2021-01-15T17:34:49.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:49.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj114-2 td/osd-backfill-space/datafile 2021-01-15T17:34:49.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:49.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:49.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:49.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj115-1 td/osd-backfill-space/datafile 2021-01-15T17:34:49.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:49.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj115-2 td/osd-backfill-space/datafile 2021-01-15T17:34:50.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:50.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj116-1 td/osd-backfill-space/datafile 2021-01-15T17:34:50.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:50.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj116-2 td/osd-backfill-space/datafile 2021-01-15T17:34:50.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:50.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:50.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:50.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj117-1 td/osd-backfill-space/datafile 2021-01-15T17:34:50.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:50.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj117-2 td/osd-backfill-space/datafile 2021-01-15T17:34:51.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:51.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:51.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:51.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj118-1 td/osd-backfill-space/datafile 2021-01-15T17:34:51.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:51.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj118-2 td/osd-backfill-space/datafile 2021-01-15T17:34:51.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:51.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj119-1 td/osd-backfill-space/datafile 2021-01-15T17:34:51.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:51.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj119-2 td/osd-backfill-space/datafile 2021-01-15T17:34:52.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:52.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:52.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj120-1 td/osd-backfill-space/datafile 2021-01-15T17:34:52.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:52.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj120-2 td/osd-backfill-space/datafile 2021-01-15T17:34:52.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:52.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:52.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:52.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj121-1 td/osd-backfill-space/datafile 2021-01-15T17:34:53.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:53.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj121-2 td/osd-backfill-space/datafile 2021-01-15T17:34:53.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:53.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:53.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:53.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj122-1 td/osd-backfill-space/datafile 2021-01-15T17:34:53.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:53.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj122-2 td/osd-backfill-space/datafile 2021-01-15T17:34:53.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:53.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj123-1 td/osd-backfill-space/datafile 2021-01-15T17:34:53.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:53.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj123-2 td/osd-backfill-space/datafile 2021-01-15T17:34:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:54.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:54.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:54.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj124-1 td/osd-backfill-space/datafile 2021-01-15T17:34:54.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:54.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj124-2 td/osd-backfill-space/datafile 2021-01-15T17:34:54.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:54.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:54.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:54.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj125-1 td/osd-backfill-space/datafile 2021-01-15T17:34:54.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:54.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj125-2 td/osd-backfill-space/datafile 2021-01-15T17:34:55.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:55.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:55.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:55.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj126-1 td/osd-backfill-space/datafile 2021-01-15T17:34:55.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:55.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj126-2 td/osd-backfill-space/datafile 2021-01-15T17:34:55.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:55.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:55.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:55.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj127-1 td/osd-backfill-space/datafile 2021-01-15T17:34:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:55.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj127-2 td/osd-backfill-space/datafile 2021-01-15T17:34:56.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:56.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:56.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:56.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj128-1 td/osd-backfill-space/datafile 2021-01-15T17:34:56.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:56.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj128-2 td/osd-backfill-space/datafile 2021-01-15T17:34:56.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:56.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:56.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj129-1 td/osd-backfill-space/datafile 2021-01-15T17:34:56.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:56.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj129-2 td/osd-backfill-space/datafile 2021-01-15T17:34:57.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:57.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:57.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:57.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj130-1 td/osd-backfill-space/datafile 2021-01-15T17:34:57.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:57.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj130-2 td/osd-backfill-space/datafile 2021-01-15T17:34:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:57.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:57.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:57.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj131-1 td/osd-backfill-space/datafile 2021-01-15T17:34:57.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:57.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj131-2 td/osd-backfill-space/datafile 2021-01-15T17:34:58.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:58.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:58.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:58.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj132-1 td/osd-backfill-space/datafile 2021-01-15T17:34:58.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj132-2 td/osd-backfill-space/datafile 2021-01-15T17:34:58.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:58.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj133-1 td/osd-backfill-space/datafile 2021-01-15T17:34:58.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:58.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj133-2 td/osd-backfill-space/datafile 2021-01-15T17:34:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:59.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:59.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:59.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj134-1 td/osd-backfill-space/datafile 2021-01-15T17:34:59.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:59.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj134-2 td/osd-backfill-space/datafile 2021-01-15T17:34:59.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:34:59.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:34:59.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:59.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj135-1 td/osd-backfill-space/datafile 2021-01-15T17:34:59.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:34:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj135-2 td/osd-backfill-space/datafile 2021-01-15T17:35:00.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:00.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj136-1 td/osd-backfill-space/datafile 2021-01-15T17:35:00.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:00.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj136-2 td/osd-backfill-space/datafile 2021-01-15T17:35:00.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:00.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:00.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:00.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj137-1 td/osd-backfill-space/datafile 2021-01-15T17:35:00.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj137-2 td/osd-backfill-space/datafile 2021-01-15T17:35:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:01.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:01.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:01.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj138-1 td/osd-backfill-space/datafile 2021-01-15T17:35:01.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:01.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj138-2 td/osd-backfill-space/datafile 2021-01-15T17:35:01.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:01.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:01.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:01.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj139-1 td/osd-backfill-space/datafile 2021-01-15T17:35:01.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:01.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj139-2 td/osd-backfill-space/datafile 2021-01-15T17:35:02.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:02.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:02.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj140-1 td/osd-backfill-space/datafile 2021-01-15T17:35:02.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:02.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj140-2 td/osd-backfill-space/datafile 2021-01-15T17:35:02.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:02.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:02.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:02.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj141-1 td/osd-backfill-space/datafile 2021-01-15T17:35:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj141-2 td/osd-backfill-space/datafile 2021-01-15T17:35:03.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:03.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:03.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:03.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj142-1 td/osd-backfill-space/datafile 2021-01-15T17:35:03.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:03.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj142-2 td/osd-backfill-space/datafile 2021-01-15T17:35:03.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:03.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:03.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:03.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj143-1 td/osd-backfill-space/datafile 2021-01-15T17:35:03.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:03.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj143-2 td/osd-backfill-space/datafile 2021-01-15T17:35:04.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:04.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:04.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:04.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj144-1 td/osd-backfill-space/datafile 2021-01-15T17:35:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj144-2 td/osd-backfill-space/datafile 2021-01-15T17:35:04.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:04.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:04.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:04.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj145-1 td/osd-backfill-space/datafile 2021-01-15T17:35:04.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:04.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj145-2 td/osd-backfill-space/datafile 2021-01-15T17:35:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:05.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj146-1 td/osd-backfill-space/datafile 2021-01-15T17:35:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj146-2 td/osd-backfill-space/datafile 2021-01-15T17:35:05.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:05.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:05.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:05.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj147-1 td/osd-backfill-space/datafile 2021-01-15T17:35:05.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:05.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj147-2 td/osd-backfill-space/datafile 2021-01-15T17:35:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:06.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj148-1 td/osd-backfill-space/datafile 2021-01-15T17:35:06.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:06.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj148-2 td/osd-backfill-space/datafile 2021-01-15T17:35:06.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:06.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:06.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:06.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj149-1 td/osd-backfill-space/datafile 2021-01-15T17:35:06.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:06.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj149-2 td/osd-backfill-space/datafile 2021-01-15T17:35:07.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:07.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:07.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:07.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj150-1 td/osd-backfill-space/datafile 2021-01-15T17:35:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj150-2 td/osd-backfill-space/datafile 2021-01-15T17:35:07.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:07.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:07.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:07.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj151-1 td/osd-backfill-space/datafile 2021-01-15T17:35:07.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:07.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj151-2 td/osd-backfill-space/datafile 2021-01-15T17:35:08.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:08.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:08.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:08.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj152-1 td/osd-backfill-space/datafile 2021-01-15T17:35:08.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:08.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj152-2 td/osd-backfill-space/datafile 2021-01-15T17:35:08.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:08.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:08.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:08.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj153-1 td/osd-backfill-space/datafile 2021-01-15T17:35:08.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:08.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj153-2 td/osd-backfill-space/datafile 2021-01-15T17:35:09.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:09.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:09.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:09.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj154-1 td/osd-backfill-space/datafile 2021-01-15T17:35:09.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:09.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj154-2 td/osd-backfill-space/datafile 2021-01-15T17:35:09.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:09.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj155-1 td/osd-backfill-space/datafile 2021-01-15T17:35:09.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:09.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj155-2 td/osd-backfill-space/datafile 2021-01-15T17:35:10.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:10.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:10.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:10.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj156-1 td/osd-backfill-space/datafile 2021-01-15T17:35:10.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:10.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj156-2 td/osd-backfill-space/datafile 2021-01-15T17:35:10.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:10.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:10.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:10.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj157-1 td/osd-backfill-space/datafile 2021-01-15T17:35:10.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:10.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj157-2 td/osd-backfill-space/datafile 2021-01-15T17:35:11.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:11.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:11.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:11.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj158-1 td/osd-backfill-space/datafile 2021-01-15T17:35:11.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:11.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj158-2 td/osd-backfill-space/datafile 2021-01-15T17:35:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:11.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:11.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:11.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj159-1 td/osd-backfill-space/datafile 2021-01-15T17:35:11.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:11.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj159-2 td/osd-backfill-space/datafile 2021-01-15T17:35:12.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:12.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:12.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:12.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj160-1 td/osd-backfill-space/datafile 2021-01-15T17:35:12.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:12.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj160-2 td/osd-backfill-space/datafile 2021-01-15T17:35:12.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:12.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:12.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:12.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj161-1 td/osd-backfill-space/datafile 2021-01-15T17:35:12.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:12.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj161-2 td/osd-backfill-space/datafile 2021-01-15T17:35:12.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:12.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj162-1 td/osd-backfill-space/datafile 2021-01-15T17:35:13.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:13.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj162-2 td/osd-backfill-space/datafile 2021-01-15T17:35:13.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:13.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:13.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:13.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj163-1 td/osd-backfill-space/datafile 2021-01-15T17:35:13.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:13.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj163-2 td/osd-backfill-space/datafile 2021-01-15T17:35:14.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:14.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:14.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj164-1 td/osd-backfill-space/datafile 2021-01-15T17:35:14.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:14.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj164-2 td/osd-backfill-space/datafile 2021-01-15T17:35:14.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:14.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:14.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:14.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj165-1 td/osd-backfill-space/datafile 2021-01-15T17:35:14.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:14.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj165-2 td/osd-backfill-space/datafile 2021-01-15T17:35:15.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:15.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:15.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:15.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj166-1 td/osd-backfill-space/datafile 2021-01-15T17:35:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:15.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj166-2 td/osd-backfill-space/datafile 2021-01-15T17:35:15.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:15.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj167-1 td/osd-backfill-space/datafile 2021-01-15T17:35:15.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:15.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj167-2 td/osd-backfill-space/datafile 2021-01-15T17:35:15.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:15.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:15.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj168-1 td/osd-backfill-space/datafile 2021-01-15T17:35:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj168-2 td/osd-backfill-space/datafile 2021-01-15T17:35:16.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:16.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj169-1 td/osd-backfill-space/datafile 2021-01-15T17:35:16.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:16.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj169-2 td/osd-backfill-space/datafile 2021-01-15T17:35:16.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:16.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:16.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:16.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj170-1 td/osd-backfill-space/datafile 2021-01-15T17:35:17.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:17.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj170-2 td/osd-backfill-space/datafile 2021-01-15T17:35:17.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:17.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:17.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:17.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj171-1 td/osd-backfill-space/datafile 2021-01-15T17:35:17.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:17.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj171-2 td/osd-backfill-space/datafile 2021-01-15T17:35:17.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:17.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:17.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:17.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj172-1 td/osd-backfill-space/datafile 2021-01-15T17:35:18.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:18.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj172-2 td/osd-backfill-space/datafile 2021-01-15T17:35:18.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:18.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj173-1 td/osd-backfill-space/datafile 2021-01-15T17:35:18.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:18.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj173-2 td/osd-backfill-space/datafile 2021-01-15T17:35:18.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:18.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:18.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:18.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj174-1 td/osd-backfill-space/datafile 2021-01-15T17:35:19.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:19.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj174-2 td/osd-backfill-space/datafile 2021-01-15T17:35:19.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:19.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:19.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:19.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj175-1 td/osd-backfill-space/datafile 2021-01-15T17:35:19.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:19.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj175-2 td/osd-backfill-space/datafile 2021-01-15T17:35:19.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:19.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:19.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:19.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj176-1 td/osd-backfill-space/datafile 2021-01-15T17:35:20.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:20.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj176-2 td/osd-backfill-space/datafile 2021-01-15T17:35:20.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:20.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:20.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:20.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj177-1 td/osd-backfill-space/datafile 2021-01-15T17:35:20.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:20.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj177-2 td/osd-backfill-space/datafile 2021-01-15T17:35:20.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:20.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:20.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:20.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj178-1 td/osd-backfill-space/datafile 2021-01-15T17:35:21.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:21.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj178-2 td/osd-backfill-space/datafile 2021-01-15T17:35:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:21.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:21.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj179-1 td/osd-backfill-space/datafile 2021-01-15T17:35:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj179-2 td/osd-backfill-space/datafile 2021-01-15T17:35:21.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:21.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:21.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:21.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj180-1 td/osd-backfill-space/datafile 2021-01-15T17:35:22.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:22.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj180-2 td/osd-backfill-space/datafile 2021-01-15T17:35:22.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:22.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:22.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:22.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj181-1 td/osd-backfill-space/datafile 2021-01-15T17:35:22.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:22.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj181-2 td/osd-backfill-space/datafile 2021-01-15T17:35:22.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:22.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:22.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:22.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj182-1 td/osd-backfill-space/datafile 2021-01-15T17:35:23.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:23.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj182-2 td/osd-backfill-space/datafile 2021-01-15T17:35:23.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:23.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:23.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:23.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj183-1 td/osd-backfill-space/datafile 2021-01-15T17:35:23.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:23.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj183-2 td/osd-backfill-space/datafile 2021-01-15T17:35:23.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:23.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:23.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:23.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj184-1 td/osd-backfill-space/datafile 2021-01-15T17:35:24.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:24.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj184-2 td/osd-backfill-space/datafile 2021-01-15T17:35:24.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:24.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:24.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:24.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj185-1 td/osd-backfill-space/datafile 2021-01-15T17:35:24.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:24.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj185-2 td/osd-backfill-space/datafile 2021-01-15T17:35:24.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:24.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:24.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:24.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj186-1 td/osd-backfill-space/datafile 2021-01-15T17:35:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj186-2 td/osd-backfill-space/datafile 2021-01-15T17:35:25.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:25.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:25.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:25.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj187-1 td/osd-backfill-space/datafile 2021-01-15T17:35:25.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:25.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj187-2 td/osd-backfill-space/datafile 2021-01-15T17:35:25.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:25.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:25.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:25.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj188-1 td/osd-backfill-space/datafile 2021-01-15T17:35:26.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:26.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj188-2 td/osd-backfill-space/datafile 2021-01-15T17:35:26.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:26.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj189-1 td/osd-backfill-space/datafile 2021-01-15T17:35:26.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:26.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj189-2 td/osd-backfill-space/datafile 2021-01-15T17:35:26.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:26.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:26.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:26.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj190-1 td/osd-backfill-space/datafile 2021-01-15T17:35:26.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:26.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj190-2 td/osd-backfill-space/datafile 2021-01-15T17:35:27.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:27.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:27.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:27.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj191-1 td/osd-backfill-space/datafile 2021-01-15T17:35:27.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:27.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj191-2 td/osd-backfill-space/datafile 2021-01-15T17:35:27.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:27.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:27.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:27.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj192-1 td/osd-backfill-space/datafile 2021-01-15T17:35:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj192-2 td/osd-backfill-space/datafile 2021-01-15T17:35:28.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:28.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj193-1 td/osd-backfill-space/datafile 2021-01-15T17:35:28.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:28.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj193-2 td/osd-backfill-space/datafile 2021-01-15T17:35:28.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:28.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:28.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:28.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj194-1 td/osd-backfill-space/datafile 2021-01-15T17:35:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj194-2 td/osd-backfill-space/datafile 2021-01-15T17:35:29.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:29.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:29.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:29.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj195-1 td/osd-backfill-space/datafile 2021-01-15T17:35:29.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:29.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj195-2 td/osd-backfill-space/datafile 2021-01-15T17:35:29.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:29.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:29.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:29.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj196-1 td/osd-backfill-space/datafile 2021-01-15T17:35:30.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj196-2 td/osd-backfill-space/datafile 2021-01-15T17:35:30.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:30.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:30.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:30.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj197-1 td/osd-backfill-space/datafile 2021-01-15T17:35:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj197-2 td/osd-backfill-space/datafile 2021-01-15T17:35:31.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:31.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:31.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:31.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj198-1 td/osd-backfill-space/datafile 2021-01-15T17:35:31.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:31.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj198-2 td/osd-backfill-space/datafile 2021-01-15T17:35:32.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:32.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:32.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:32.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj199-1 td/osd-backfill-space/datafile 2021-01-15T17:35:32.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:32.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj199-2 td/osd-backfill-space/datafile 2021-01-15T17:35:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:805: TEST_ec_backfill_multi: for o in $(seq 1 $ecobjects) 2021-01-15T17:35:32.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:32.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:32.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test1 put obj200-1 td/osd-backfill-space/datafile 2021-01-15T17:35:33.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:807: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:33.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:809: TEST_ec_backfill_multi: rados -p test2 put obj200-2 td/osd-backfill-space/datafile 2021-01-15T17:35:33.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:813: TEST_ec_backfill_multi: ceph pg dump pgs 2021-01-15T17:35:33.842 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:35:33.842 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 191 0 0 0 0 2347008 0 0 91 91 active+clean 2021-01-15T17:33:39.279834+0000 53'191 53:204 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2021-01-15T17:33:36.872099+0000 0'0 2021-01-15T17:33:36.872099+0000 0 2021-01-15T17:35:33.843 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2021-01-15T17:33:17.349256+0000 40'200 52:227 [1] 1 [1] 1 0'0 2021-01-15T17:33:14.912061+0000 0'0 2021-01-15T17:33:14.912061+0000 0 2021-01-15T17:35:33.843 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 191 0 0 0 0 2347008 0 0 91 91 active+clean 2021-01-15T17:33:37.334304+0000 53'191 53:208 [0,2,3,4,5] 0 [0,2,3,4,5] 0 0'0 2021-01-15T17:33:35.212946+0000 0'0 2021-01-15T17:33:35.212946+0000 0 2021-01-15T17:35:33.843 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:35:33.844 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:35:33.844 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:35:33.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:35:33.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:33.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: expr 1 + 1 2021-01-15T17:35:33.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: ceph osd pg-upmap 2.0 0 2 3 4 1 2021-01-15T17:35:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:set 2.0 pg_upmap mapping to [0,2,3,4,1] 2021-01-15T17:35:34.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:815: TEST_ec_backfill_multi: for p in $(seq 1 $pools) 2021-01-15T17:35:34.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: expr 2 + 1 2021-01-15T17:35:34.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:817: TEST_ec_backfill_multi: ceph osd pg-upmap 3.0 0 2 3 4 1 2021-01-15T17:35:36.008 INFO:tasks.workunit.client.0.smithi145.stderr:set 3.0 pg_upmap mapping to [0,2,3,4,1] 2021-01-15T17:35:36.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:820: TEST_ec_backfill_multi: sleep 30 2021-01-15T17:36:06.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:822: TEST_ec_backfill_multi: wait_for_not_backfilling 240 2021-01-15T17:36:06.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:36:06.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:36:06.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:36:06.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:36:06.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:36:06.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:36:06.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:36:06.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:36:06.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:36:06.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:36:06.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:36:06.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:36:06.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:36:06.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:36:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:36:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:36:06.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:36:06.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:36:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:36:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:36:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:36:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:36:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:36:06.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:36:06.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:06.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:36:06.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836525 2021-01-15T17:36:06.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836525 2021-01-15T17:36:06.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525' 2021-01-15T17:36:06.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:06.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:36:06.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673004 2021-01-15T17:36:06.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673004 2021-01-15T17:36:06.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525 1-42949673004' 2021-01-15T17:36:06.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:06.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:36:07.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509482 2021-01-15T17:36:07.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509482 2021-01-15T17:36:07.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509482' 2021-01-15T17:36:07.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:07.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:36:07.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345962 2021-01-15T17:36:07.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345962 2021-01-15T17:36:07.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509482 3-85899345962' 2021-01-15T17:36:07.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:07.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:36:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182439 2021-01-15T17:36:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182439 2021-01-15T17:36:07.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509482 3-85899345962 4-107374182439' 2021-01-15T17:36:07.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:07.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:36:07.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018917 2021-01-15T17:36:07.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018917 2021-01-15T17:36:07.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836525 1-42949673004 2-64424509482 3-85899345962 4-107374182439 5-128849018917' 2021-01-15T17:36:07.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:07.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836525 2021-01-15T17:36:07.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:07.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:36:07.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836525 2021-01-15T17:36:07.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:07.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836525 2021-01-15T17:36:07.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2021-01-15T17:36:07.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:36:07.432 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836525 2021-01-15T17:36:07.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836523 -lt 21474836525 2021-01-15T17:36:07.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:36:08.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:36:08.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:36:09.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836526 -lt 21474836525 2021-01-15T17:36:09.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:09.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673004 2021-01-15T17:36:09.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:09.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:36:09.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673004 2021-01-15T17:36:09.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673004 2021-01-15T17:36:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2021-01-15T17:36:09.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:36:09.240 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673004 2021-01-15T17:36:09.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673004 -lt 42949673004 2021-01-15T17:36:09.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:09.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509482 2021-01-15T17:36:09.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:09.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:36:09.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509482 2021-01-15T17:36:09.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:09.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509482 2021-01-15T17:36:09.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2021-01-15T17:36:09.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:36:09.645 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509482 2021-01-15T17:36:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509482 -lt 64424509482 2021-01-15T17:36:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:10.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345962 2021-01-15T17:36:10.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:10.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:36:10.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345962 2021-01-15T17:36:10.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:10.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345962 2021-01-15T17:36:10.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345962' 2021-01-15T17:36:10.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:36:10.047 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345962 2021-01-15T17:36:10.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345962 -lt 85899345962 2021-01-15T17:36:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:10.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182439 2021-01-15T17:36:10.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:10.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:36:10.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182439 2021-01-15T17:36:10.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182439 2021-01-15T17:36:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182439' 2021-01-15T17:36:10.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:36:10.447 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182439 2021-01-15T17:36:10.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182440 -lt 107374182439 2021-01-15T17:36:10.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:10.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018917 2021-01-15T17:36:10.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:10.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:36:10.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018917 2021-01-15T17:36:10.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:10.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018917 2021-01-15T17:36:10.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018917' 2021-01-15T17:36:10.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:36:10.863 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018917 2021-01-15T17:36:11.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018918 -lt 128849018917 2021-01-15T17:36:11.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:36:11.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:36:11.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:36:11.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2021-01-15T17:36:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:36:11.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:36:11.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:36:11.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:36:11.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:36:11.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:36:11.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:36:12.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:36:12.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:36:12.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' -1 2021-01-15T17:36:12.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:68: wait_for_not_state: loop=0 2021-01-15T17:36:12.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:69: wait_for_not_state: num_in_state=1 2021-01-15T17:36:12.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 5 2021-01-15T17:36:17.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:36:17.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:36:17.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:36:17.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:36:17.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:36:17.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:36:17.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:36:17.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:36:17.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=1 2021-01-15T17:36:17.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 1 = 0 2021-01-15T17:36:17.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:67: wait_for_not_state: test 1 '!=' 1 2021-01-15T17:36:17.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:70: wait_for_not_state: (( 1 >= 17 )) 2021-01-15T17:36:17.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:74: wait_for_not_state: sleep 10 2021-01-15T17:36:27.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:75: wait_for_not_state: loop+=1 2021-01-15T17:36:27.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:36:27.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:36:27.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:36:27.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:36:27.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:36:27.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:36:27.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:36:27.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:36:27.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:36:27.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:36:27.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:36:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:823: TEST_ec_backfill_multi: wait_for_not_activating 60 2021-01-15T17:36:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:36:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:36:27.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:36:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:36:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:36:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:36:27.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:36:27.967 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:36:27.968 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:36:27.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:36:27.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:36:27.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:36:28.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:36:28.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:36:28.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:36:28.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:36:28.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:36:28.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:36:28.397 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:36:28.398 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:36:28.398 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:36:28.398 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:36:28.398 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:36:28.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:36:28.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:28.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:36:28.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836532 2021-01-15T17:36:28.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836532 2021-01-15T17:36:28.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532' 2021-01-15T17:36:28.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:28.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:36:28.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673010 2021-01-15T17:36:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673010 2021-01-15T17:36:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532 1-42949673010' 2021-01-15T17:36:28.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:28.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:36:28.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509489 2021-01-15T17:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509489 2021-01-15T17:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509489' 2021-01-15T17:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:36:29.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345968 2021-01-15T17:36:29.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345968 2021-01-15T17:36:29.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509489 3-85899345968' 2021-01-15T17:36:29.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:29.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:36:29.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182445 2021-01-15T17:36:29.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182445 2021-01-15T17:36:29.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509489 3-85899345968 4-107374182445' 2021-01-15T17:36:29.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:36:29.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:36:29.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018924 2021-01-15T17:36:29.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018924 2021-01-15T17:36:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836532 1-42949673010 2-64424509489 3-85899345968 4-107374182445 5-128849018924' 2021-01-15T17:36:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:29.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836532 2021-01-15T17:36:29.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:29.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836532 2021-01-15T17:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836532 2021-01-15T17:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2021-01-15T17:36:29.300 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836532 2021-01-15T17:36:29.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:36:29.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836530 -lt 21474836532 2021-01-15T17:36:29.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:36:30.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:36:30.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:36:31.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836532 -lt 21474836532 2021-01-15T17:36:31.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:31.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673010 2021-01-15T17:36:31.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:31.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:36:31.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673010 2021-01-15T17:36:31.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:31.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673010 2021-01-15T17:36:31.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2021-01-15T17:36:31.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:36:31.122 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673010 2021-01-15T17:36:31.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673010 -lt 42949673010 2021-01-15T17:36:31.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:31.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509489 2021-01-15T17:36:31.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:31.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:36:31.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509489 2021-01-15T17:36:31.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:31.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509489 2021-01-15T17:36:31.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2021-01-15T17:36:31.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:36:31.529 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509489 2021-01-15T17:36:31.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509489 -lt 64424509489 2021-01-15T17:36:31.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:31.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345968 2021-01-15T17:36:31.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:31.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:36:31.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345968 2021-01-15T17:36:31.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:31.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345968 2021-01-15T17:36:31.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345968' 2021-01-15T17:36:31.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:36:31.965 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345968 2021-01-15T17:36:32.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345968 -lt 85899345968 2021-01-15T17:36:32.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:32.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182445 2021-01-15T17:36:32.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:32.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:36:32.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182445 2021-01-15T17:36:32.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:32.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182445 2021-01-15T17:36:32.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182445' 2021-01-15T17:36:32.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:36:32.375 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182445 2021-01-15T17:36:32.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182446 -lt 107374182445 2021-01-15T17:36:32.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:36:32.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018924 2021-01-15T17:36:32.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:36:32.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:36:32.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018924 2021-01-15T17:36:32.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:36:32.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018924 2021-01-15T17:36:32.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018924' 2021-01-15T17:36:32.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:36:32.797 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018924 2021-01-15T17:36:33.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018924 -lt 128849018924 2021-01-15T17:36:33.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:36:33.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:36:33.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:36:33.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 3 == 0 2021-01-15T17:36:33.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:36:33.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:36:33.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:36:33.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:36:33.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:36:33.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:36:33.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:36:34.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:36:34.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:36:34.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:36:34.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:36:34.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:825: TEST_ec_backfill_multi: ceph pg dump pgs 2021-01-15T17:36:34.474 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:36:34.474 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 200 0 0 200 0 2457600 0 0 100 100 active+remapped+backfill_toofull 2021-01-15T17:36:07.052640+0000 58'200 67:246 [0,2,3,4,1] 0 [0,2,3,4,5] 0 0'0 2021-01-15T17:33:36.872099+0000 0'0 2021-01-15T17:33:36.872099+0000 0 2021-01-15T17:36:34.474 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 1843200 0 0 100 100 active+clean 2021-01-15T17:33:17.349256+0000 40'200 67:241 [1] 1 [1] 1 0'0 2021-01-15T17:33:14.912061+0000 0'0 2021-01-15T17:33:14.912061+0000 0 2021-01-15T17:36:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 200 0 0 0 0 2457600 0 0 100 100 active+clean 2021-01-15T17:36:19.799926+0000 58'200 68:659 [0,2,3,4,1] 0 [0,2,3,4,1] 0 0'0 2021-01-15T17:33:35.212946+0000 0'0 2021-01-15T17:33:35.212946+0000 0 2021-01-15T17:36:34.475 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:36:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:36:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:36:34.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:827: TEST_ec_backfill_multi: ERRORS=0 2021-01-15T17:36:34.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: ceph pg dump pgs 2021-01-15T17:36:34.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: grep -v '^1.0' 2021-01-15T17:36:34.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: grep +backfill_toofull 2021-01-15T17:36:34.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: wc -l 2021-01-15T17:36:34.942 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:36:34.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:828: TEST_ec_backfill_multi: '[' 1 '!=' 1 ']' 2021-01-15T17:36:34.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: ceph pg dump pgs 2021-01-15T17:36:34.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: grep -v '^1.0' 2021-01-15T17:36:34.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: grep active+clean 2021-01-15T17:36:34.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: wc -l 2021-01-15T17:36:35.326 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:36:35.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:834: TEST_ec_backfill_multi: '[' 1 '!=' 1 ']' 2021-01-15T17:36:35.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:840: TEST_ec_backfill_multi: '[' 0 '!=' 0 ']' 2021-01-15T17:36:35.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:845: TEST_ec_backfill_multi: delete_pool fillpool 2021-01-15T17:36:35.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2021-01-15T17:36:35.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2021-01-15T17:36:35.872 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' does not exist 2021-01-15T17:36:35.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: seq 1 2 2021-01-15T17:36:35.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: for i in $(seq 1 $pools) 2021-01-15T17:36:35.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:848: TEST_ec_backfill_multi: delete_pool test1 2021-01-15T17:36:35.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:36:35.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:36:36.437 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:36:36.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:846: TEST_ec_backfill_multi: for i in $(seq 1 $pools) 2021-01-15T17:36:36.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:848: TEST_ec_backfill_multi: delete_pool test2 2021-01-15T17:36:36.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T17:36:36.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T17:36:36.955 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T17:36:36.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:850: TEST_ec_backfill_multi: kill_daemons td/osd-backfill-space 2021-01-15T17:36:36.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:36:36.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:36:36.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:36:36.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:36:36.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:36:37.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:36:37.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:36:37.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:36:37.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:36:37.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:36:37.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:36:37.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:36:37.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:36:37.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:36:37.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:36:37.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:36:37.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:36:37.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:36:37.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:36:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:36:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:36:37.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:36:37.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:36:37.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:36:37.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:36:37.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:36:37.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:36:37.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:36:37.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:36:37.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:36:37.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:36:37.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:36:37.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:36:37.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:35: run: for func in $funcs 2021-01-15T17:36:37.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:36: run: setup td/osd-backfill-space 2021-01-15T17:36:37.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-space 2021-01-15T17:36:37.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-space 2021-01-15T17:36:37.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:36:37.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:36:37.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:36:37.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:36:37.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:36:37.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:36:37.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:36:37.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:36:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:36:37.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:36:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:36:37.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:36:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:36:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:36:37.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:36:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:36:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:36:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:36:37.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:36:37.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:36:37.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:36:37.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:36:37.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:36:37.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:36:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:36:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:36:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-space 2021-01-15T17:36:37.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:36:37.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173017 2021-01-15T17:36:37.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:36:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:36:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:36:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2021-01-15T17:36:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:37: run: TEST_ec_backfill_simple td/osd-backfill-space 2021-01-15T17:36:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:623: TEST_ec_backfill_simple: local dir=td/osd-backfill-space 2021-01-15T17:36:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:624: TEST_ec_backfill_simple: local EC= 2021-01-15T17:36:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:625: TEST_ec_backfill_simple: local pools=1 2021-01-15T17:36:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:626: TEST_ec_backfill_simple: local OSDS=6 2021-01-15T17:36:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:627: TEST_ec_backfill_simple: local k=3 2021-01-15T17:36:37.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:628: TEST_ec_backfill_simple: local m=2 2021-01-15T17:36:37.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:629: TEST_ec_backfill_simple: expr 600 / 3 2021-01-15T17:36:37.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:629: TEST_ec_backfill_simple: local ecobjects=200 2021-01-15T17:36:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:631: TEST_ec_backfill_simple: run_mon td/osd-backfill-space a 2021-01-15T17:36:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-space 2021-01-15T17:36:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:36:37.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:36:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:36:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-space/a 2021-01-15T17:36:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2021-01-15T17:36:37.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:36:37.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:36:37.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:36:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:36:37.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:37.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:36:37.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:36:37.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:36:37.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:36:37.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:36:37.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:36:37.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:36:37.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:36:37.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:36:37.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:36:37.610 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:36:37.610 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.610 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:36:37.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:36:37.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get fsid 2021-01-15T17:36:37.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:36:37.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:36:37.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:36:37.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:36:37.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:36:37.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:36:37.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:36:37.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:36:37.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:36:37.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:37.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:37.712 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173017/ceph-mon.a.asok 2021-01-15T17:36:37.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:36:37.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.173017/ceph-mon.a.asok config get mon_host 2021-01-15T17:36:37.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:632: TEST_ec_backfill_simple: run_mgr td/osd-backfill-space x 2021-01-15T17:36:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-space 2021-01-15T17:36:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:36:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:36:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:36:37.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-space/x 2021-01-15T17:36:37.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:36:38.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:36:38.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:36:38.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:36:38.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:36:38.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:38.302 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:38.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:38.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:36:38.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:36:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:633: TEST_ec_backfill_simple: export CEPH_ARGS 2021-01-15T17:36:38.326 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: expr 6 - 1 2021-01-15T17:36:38.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: seq 0 5 2021-01-15T17:36:38.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:36:38.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 0 2021-01-15T17:36:38.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:36:38.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:36:38.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:36:38.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:36:38.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/0 2021-01-15T17:36:38.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:36:38.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:36:38.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:36:38.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:36:38.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2021-01-15T17:36:38.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2021-01-15T17:36:38.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:36:38.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:36:38.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:36:38.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:36:38.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:36:38.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:36:38.335 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:36:38.335 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:38.336 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:38.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:38.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:38.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:36:38.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:36:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:36:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:36:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:36:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:36:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:36:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:36:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:36:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/0 2021-01-15T17:36:38.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:36:38.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=06f94a1e-edf5-4ebd-b553-23efb7cb2383 2021-01-15T17:36:38.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 06f94a1e-edf5-4ebd-b553-23efb7cb2383' 2021-01-15T17:36:38.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:36:38.340 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 06f94a1e-edf5-4ebd-b553-23efb7cb2383 2021-01-15T17:36:38.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCm0gFgfuTLFBAAOpP/vVcveJbUeMdm5y1v9Q== 2021-01-15T17:36:38.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCm0gFgfuTLFBAAOpP/vVcveJbUeMdm5y1v9Q=="}' 2021-01-15T17:36:38.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 06f94a1e-edf5-4ebd-b553-23efb7cb2383 -i td/osd-backfill-space/0/new.json 2021-01-15T17:36:38.724 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:36:38.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/0/new.json 2021-01-15T17:36:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCm0gFgfuTLFBAAOpP/vVcveJbUeMdm5y1v9Q== --osd-uuid 06f94a1e-edf5-4ebd-b553-23efb7cb2383 2021-01-15T17:36:38.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.763+0000 7f80d74eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:38.765 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.765+0000 7f80d74eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:38.767 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.767+0000 7f80d74eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:38.767 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.767+0000 7f80d74eb080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:36:38.768 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.767+0000 7f80d74eb080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:36:38.768 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.767+0000 7f80d74eb080 -1 bluestore(td/osd-backfill-space/0/block) _read_bdev_label failed to open td/osd-backfill-space/0/block: (2) No such file or directory 2021-01-15T17:36:38.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:38.830+0000 7f80d74eb080 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2021-01-15T17:36:43.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2021-01-15T17:36:43.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:36:43.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:36:43.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:36:43.417 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:36:43.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:36:43.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:43.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:36:43.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:36:43.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:36:43.943 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:36:43.968 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:43.967+0000 7fb861146080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:43.969 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:43.969+0000 7fb861146080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:43.971 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:43.971+0000 7fb861146080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:44.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:36:44.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:36:44.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:36:44.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:36:44.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:36:44.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:36:44.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:36:44.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:36:44.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:36:44.344 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:36:44.491 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:44.490+0000 7fb861146080 -1 Falling back to public interface 2021-01-15T17:36:44.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:36:45.289 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:45.288+0000 7fb861146080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:36:45.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:36:45.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:36:45.740 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:36:46.167 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/499269,v1:127.0.0.1:6803/499269] [v2:127.0.0.1:6804/499269,v1:127.0.0.1:6805/499269] exists,up 06f94a1e-edf5-4ebd-b553-23efb7cb2383 2021-01-15T17:36:46.167 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 e1fad7e3-86cc-48b3-8793-c85d93a4f2fd 2021-01-15T17:36:46.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:36:46.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:36:46.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:36:46.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:36:46.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 1 2021-01-15T17:36:46.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:36:46.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:36:46.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:36:46.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:36:46.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:36:46.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:36:46.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:36:46.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:36:46.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:36:46.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:36:46.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:36:46.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:36:46.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:36:46.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:36:46.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:36:46.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:36:46.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:36:46.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:36:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:46.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:36:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:36:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:36:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:36:46.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:36:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:36:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:36:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:36:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:36:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:36:46.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:36:46.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e1fad7e3-86cc-48b3-8793-c85d93a4f2fd 2021-01-15T17:36:46.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 e1fad7e3-86cc-48b3-8793-c85d93a4f2fd' 2021-01-15T17:36:46.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:36:46.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCu0gFgt2/wChAAwhoER/kRxXoJvMCpiwvtpg== 2021-01-15T17:36:46.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCu0gFgt2/wChAAwhoER/kRxXoJvMCpiwvtpg=="}' 2021-01-15T17:36:46.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e1fad7e3-86cc-48b3-8793-c85d93a4f2fd -i td/osd-backfill-space/1/new.json 2021-01-15T17:36:46.770 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:36:46.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/1/new.json 2021-01-15T17:36:46.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCu0gFgt2/wChAAwhoER/kRxXoJvMCpiwvtpg== --osd-uuid e1fad7e3-86cc-48b3-8793-c85d93a4f2fd 2021-01-15T17:36:46.810 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.810+0000 7f88b5d43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:46.812 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.812+0000 7f88b5d43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:46.814 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.813+0000 7f88b5d43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:46.814 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.814+0000 7f88b5d43080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:36:46.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.814+0000 7f88b5d43080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:36:46.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.814+0000 7f88b5d43080 -1 bluestore(td/osd-backfill-space/1/block) _read_bdev_label failed to open td/osd-backfill-space/1/block: (2) No such file or directory 2021-01-15T17:36:46.853 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:46.852+0000 7f88b5d43080 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2021-01-15T17:36:51.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2021-01-15T17:36:51.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:36:51.356 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:36:51.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:36:51.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:36:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:36:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:36:51.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:36:51.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:36:51.884 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:36:51.909 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:51.909+0000 7f52b95d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:51.942 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:51.942+0000 7f52b95d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:51.943+0000 7f52b95d1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:52.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:36:52.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:36:52.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:36:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:36:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:36:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:36:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:36:52.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:36:52.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:36:52.319 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:36:52.461 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:52.461+0000 7f52b95d1080 -1 Falling back to public interface 2021-01-15T17:36:52.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:36:53.268 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:53.267+0000 7f52b95d1080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:36:53.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:36:53.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:36:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:36:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:36:53.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:36:53.740 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:36:54.179 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/500533,v1:127.0.0.1:6811/500533] [v2:127.0.0.1:6812/500533,v1:127.0.0.1:6813/500533] exists,up e1fad7e3-86cc-48b3-8793-c85d93a4f2fd 2021-01-15T17:36:54.179 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 61b9d727-af67-47bd-96fc-72f61fcc6c65 2021-01-15T17:36:54.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:36:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:36:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:36:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:36:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 2 2021-01-15T17:36:54.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:36:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:36:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:36:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:36:54.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/2 2021-01-15T17:36:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:36:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:36:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:36:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:36:54.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2021-01-15T17:36:54.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2021-01-15T17:36:54.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:36:54.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:36:54.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:36:54.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:36:54.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:36:54.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:36:54.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:36:54.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:36:54.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:36:54.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:54.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:36:54.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:36:54.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:36:54.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:36:54.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:36:54.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:36:54.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:36:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:36:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:36:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:36:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/2 2021-01-15T17:36:54.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:36:54.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=61b9d727-af67-47bd-96fc-72f61fcc6c65 2021-01-15T17:36:54.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 61b9d727-af67-47bd-96fc-72f61fcc6c65' 2021-01-15T17:36:54.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:36:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC20gFgHauoCxAAITxB/1CK86UJGqDYJxTsfw== 2021-01-15T17:36:54.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC20gFgHauoCxAAITxB/1CK86UJGqDYJxTsfw=="}' 2021-01-15T17:36:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 61b9d727-af67-47bd-96fc-72f61fcc6c65 -i td/osd-backfill-space/2/new.json 2021-01-15T17:36:54.718 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:36:54.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/2/new.json 2021-01-15T17:36:54.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC20gFgHauoCxAAITxB/1CK86UJGqDYJxTsfw== --osd-uuid 61b9d727-af67-47bd-96fc-72f61fcc6c65 2021-01-15T17:36:54.759 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.758+0000 7fc33e6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:54.760 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.760+0000 7fc33e6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:54.762 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.762+0000 7fc33e6f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:54.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.763+0000 7fc33e6f2080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:36:54.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.763+0000 7fc33e6f2080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:36:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.763+0000 7fc33e6f2080 -1 bluestore(td/osd-backfill-space/2/block) _read_bdev_label failed to open td/osd-backfill-space/2/block: (2) No such file or directory 2021-01-15T17:36:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:54.808+0000 7fc33e6f2080 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2021-01-15T17:36:59.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2021-01-15T17:36:59.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:36:59.417 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:36:59.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:36:59.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:36:59.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:36:59.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:36:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:36:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:36:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:36:59.948 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:36:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:59.972+0000 7f82f2f98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:59.974 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:59.974+0000 7f82f2f98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:36:59.975 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:36:59.975+0000 7f82f2f98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:37:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:37:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:37:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:37:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:37:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:37:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:37:00.349 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:37:00.497 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:00.497+0000 7f82f2f98080 -1 Falling back to public interface 2021-01-15T17:37:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:37:01.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:01.292+0000 7f82f2f98080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:37:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:37:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:37:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:37:01.741 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:37:02.176 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/501797,v1:127.0.0.1:6819/501797] [v2:127.0.0.1:6820/501797,v1:127.0.0.1:6821/501797] exists,up 61b9d727-af67-47bd-96fc-72f61fcc6c65 2021-01-15T17:37:02.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:37:02.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:37:02.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:37:02.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:37:02.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 3 2021-01-15T17:37:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:37:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:37:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:37:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:37:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/3 2021-01-15T17:37:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:37:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:37:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:37:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:37:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/3' 2021-01-15T17:37:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/3/journal' 2021-01-15T17:37:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:37:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:37:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:37:02.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:37:02.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:37:02.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:37:02.181 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:37:02.181 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:37:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:37:02.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:37:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:37:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:37:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:37:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:37:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:37:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:37:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:37:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:37:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/3 2021-01-15T17:37:02.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:37:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f7afe973-b5cd-4d11-949f-a886a513de19 2021-01-15T17:37:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 f7afe973-b5cd-4d11-949f-a886a513de19' 2021-01-15T17:37:02.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:37:02.186 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 f7afe973-b5cd-4d11-949f-a886a513de19 2021-01-15T17:37:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC+0gFg8QCLCxAArqJrxHQaYTu0qWvcndnKDg== 2021-01-15T17:37:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC+0gFg8QCLCxAArqJrxHQaYTu0qWvcndnKDg=="}' 2021-01-15T17:37:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f7afe973-b5cd-4d11-949f-a886a513de19 -i td/osd-backfill-space/3/new.json 2021-01-15T17:37:02.709 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:37:02.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/3/new.json 2021-01-15T17:37:02.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC+0gFg8QCLCxAArqJrxHQaYTu0qWvcndnKDg== --osd-uuid f7afe973-b5cd-4d11-949f-a886a513de19 2021-01-15T17:37:02.748 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.748+0000 7f1b9ae44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:02.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.749+0000 7f1b9ae44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:02.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.751+0000 7f1b9ae44080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.751+0000 7f1b9ae44080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:37:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.751+0000 7f1b9ae44080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:37:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.751+0000 7f1b9ae44080 -1 bluestore(td/osd-backfill-space/3/block) _read_bdev_label failed to open td/osd-backfill-space/3/block: (2) No such file or directory 2021-01-15T17:37:02.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:02.848+0000 7f1b9ae44080 -1 bluestore(td/osd-backfill-space/3) _read_fsid unparsable uuid 2021-01-15T17:37:07.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/3/keyring 2021-01-15T17:37:07.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:37:07.383 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:37:07.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:37:07.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:37:07.907 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:37:07.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:37:07.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/3 --osd-journal=td/osd-backfill-space/3/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:07.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:37:07.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:37:07.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:37:07.933 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:07.933+0000 7f76d9250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:07.935 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:07.934+0000 7f76d9250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:07.936 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:07.935+0000 7f76d9250080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:08.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:37:08.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:37:08.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:37:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:37:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:37:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:37:08.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:08.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:37:08.346 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:37:08.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:08.458+0000 7f76d9250080 -1 Falling back to public interface 2021-01-15T17:37:08.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:37:09.249 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:09.249+0000 7f76d9250080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:37:09.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:37:09.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:09.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:37:09.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:09.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:37:09.740 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:37:10.147 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/503061,v1:127.0.0.1:6827/503061] [v2:127.0.0.1:6828/503061,v1:127.0.0.1:6829/503061] exists,up f7afe973-b5cd-4d11-949f-a886a513de19 2021-01-15T17:37:10.147 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 a9b63ccf-8556-435a-ba76-43961599f9f6 2021-01-15T17:37:10.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:37:10.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:37:10.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:37:10.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:37:10.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 4 2021-01-15T17:37:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:37:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:37:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:37:10.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:37:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/4 2021-01-15T17:37:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:37:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:37:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:37:10.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:37:10.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/4' 2021-01-15T17:37:10.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/4/journal' 2021-01-15T17:37:10.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:37:10.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:37:10.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:37:10.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:37:10.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:37:10.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:37:10.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:37:10.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:37:10.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:37:10.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:10.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:10.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:37:10.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:37:10.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:37:10.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:37:10.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:37:10.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:37:10.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:37:10.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:10.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:37:10.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:37:10.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/4 2021-01-15T17:37:10.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:37:10.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a9b63ccf-8556-435a-ba76-43961599f9f6 2021-01-15T17:37:10.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 a9b63ccf-8556-435a-ba76-43961599f9f6' 2021-01-15T17:37:10.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:37:10.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDG0gFgh5PICRAAacwXd+vIJjvSzBb23Ld1Jw== 2021-01-15T17:37:10.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDG0gFgh5PICRAAacwXd+vIJjvSzBb23Ld1Jw=="}' 2021-01-15T17:37:10.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a9b63ccf-8556-435a-ba76-43961599f9f6 -i td/osd-backfill-space/4/new.json 2021-01-15T17:37:10.834 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:37:10.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/4/new.json 2021-01-15T17:37:10.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDG0gFgh5PICRAAacwXd+vIJjvSzBb23Ld1Jw== --osd-uuid a9b63ccf-8556-435a-ba76-43961599f9f6 2021-01-15T17:37:10.873 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.873+0000 7f4e9fcb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:10.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.875+0000 7f4e9fcb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:10.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.875+0000 7f4e9fcb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:10.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.876+0000 7f4e9fcb1080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:37:10.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.876+0000 7f4e9fcb1080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:37:10.877 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.876+0000 7f4e9fcb1080 -1 bluestore(td/osd-backfill-space/4/block) _read_bdev_label failed to open td/osd-backfill-space/4/block: (2) No such file or directory 2021-01-15T17:37:10.983 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:10.982+0000 7f4e9fcb1080 -1 bluestore(td/osd-backfill-space/4) _read_fsid unparsable uuid 2021-01-15T17:37:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/4/keyring 2021-01-15T17:37:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:37:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:37:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:37:15.533 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:37:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:37:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/4 --osd-journal=td/osd-backfill-space/4/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:37:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:37:16.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:37:16.202 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:37:16.203 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:16.087+0000 7fb65e19e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:16.203 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:16.131+0000 7fb65e19e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:16.203 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:16.133+0000 7fb65e19e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:16.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:37:16.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:37:16.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:37:16.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:37:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:37:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:37:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:16.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:37:16.468 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:37:16.648 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:16.648+0000 7fb65e19e080 -1 Falling back to public interface 2021-01-15T17:37:16.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:37:17.434 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:17.433+0000 7fb65e19e080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:37:17.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:37:17.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:37:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:37:17.873 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:37:18.273 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/504325,v1:127.0.0.1:6835/504325] [v2:127.0.0.1:6836/504325,v1:127.0.0.1:6837/504325] exists,up a9b63ccf-8556-435a-ba76-43961599f9f6 2021-01-15T17:37:18.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:37:18.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:37:18.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:37:18.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:635: TEST_ec_backfill_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T17:37:18.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:637: TEST_ec_backfill_simple: run_osd td/osd-backfill-space 5 2021-01-15T17:37:18.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-space 2021-01-15T17:37:18.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:37:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:37:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:37:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-space/5 2021-01-15T17:37:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:37:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:37:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:37:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:37:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/5' 2021-01-15T17:37:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/5/journal' 2021-01-15T17:37:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:37:18.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:37:18.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:37:18.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:37:18.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:18.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:37:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:37:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:37:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:37:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:37:18.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:37:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:37:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:37:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:37:18.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:37:18.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-space/5 2021-01-15T17:37:18.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:37:18.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4b822a3a-6820-47a1-beaa-25ea8148076b 2021-01-15T17:37:18.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 4b822a3a-6820-47a1-beaa-25ea8148076b' 2021-01-15T17:37:18.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:37:18.283 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 4b822a3a-6820-47a1-beaa-25ea8148076b 2021-01-15T17:37:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDO0gFgHY5bERAAm+p0pRvDfSZ3XHycGmjj1Q== 2021-01-15T17:37:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDO0gFgHY5bERAAm+p0pRvDfSZ3XHycGmjj1Q=="}' 2021-01-15T17:37:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4b822a3a-6820-47a1-beaa-25ea8148076b -i td/osd-backfill-space/5/new.json 2021-01-15T17:37:18.846 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:37:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-space/5/new.json 2021-01-15T17:37:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDO0gFgHY5bERAAm+p0pRvDfSZ3XHycGmjj1Q== --osd-uuid 4b822a3a-6820-47a1-beaa-25ea8148076b 2021-01-15T17:37:18.885 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.884+0000 7f29e6390080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:18.886 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.886+0000 7f29e6390080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:18.888 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.887+0000 7f29e6390080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:18.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.889+0000 7f29e6390080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:37:18.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.889+0000 7f29e6390080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:37:18.890 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.889+0000 7f29e6390080 -1 bluestore(td/osd-backfill-space/5/block) _read_bdev_label failed to open td/osd-backfill-space/5/block: (2) No such file or directory 2021-01-15T17:37:18.965 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:18.965+0000 7f29e6390080 -1 bluestore(td/osd-backfill-space/5) _read_fsid unparsable uuid 2021-01-15T17:37:23.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-space/5/keyring 2021-01-15T17:37:23.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:37:23.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:37:23.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-space/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:37:23.533 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:37:24.065 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:37:24.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:37:24.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/5 --osd-journal=td/osd-backfill-space/5/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:37:24.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:37:24.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:37:24.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:37:24.090 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:24.090+0000 7f72e684e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:24.123 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:24.123+0000 7f72e684e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:24.126 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:24.125+0000 7f72e684e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:37:24.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:37:24.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:37:24.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:37:24.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:37:24.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:37:24.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:24.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:37:24.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:24.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:37:24.466 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:37:24.641 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:24.641+0000 7f72e684e080 -1 Falling back to public interface 2021-01-15T17:37:24.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:37:25.441 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:37:25.440+0000 7f72e684e080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:37:25.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:37:25.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:37:25.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:37:25.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:37:25.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:37:25.880 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:37:26.343 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/505589,v1:127.0.0.1:6843/505589] [v2:127.0.0.1:6844/505589,v1:127.0.0.1:6845/505589] exists,up 4b822a3a-6820-47a1-beaa-25ea8148076b 2021-01-15T17:37:26.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:37:26.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:37:26.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:37:26.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:640: TEST_ec_backfill_simple: ceph osd set-backfillfull-ratio .85 2021-01-15T17:37:27.145 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.85 2021-01-15T17:37:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:641: TEST_ec_backfill_simple: create_pool fillpool 1 1 2021-01-15T17:37:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create fillpool 1 1 2021-01-15T17:37:27.691 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' already exists 2021-01-15T17:37:27.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:37:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:642: TEST_ec_backfill_simple: ceph osd pool set fillpool size 1 --yes-i-really-mean-it 2021-01-15T17:37:29.457 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:37:29.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:651: TEST_ec_backfill_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=13 2021-01-15T17:37:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:13+0 records in 2021-01-15T17:37:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:13+0 records out 2021-01-15T17:37:29.467 INFO:tasks.workunit.client.0.smithi145.stderr:13312 bytes (13 kB, 13 KiB) copied, 0.000140992 s, 94.4 MB/s 2021-01-15T17:37:29.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: seq 1 200 2021-01-15T17:37:29.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:29.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj1 td/osd-backfill-space/datafile 2021-01-15T17:37:29.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:29.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj2 td/osd-backfill-space/datafile 2021-01-15T17:37:29.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:29.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj3 td/osd-backfill-space/datafile 2021-01-15T17:37:29.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:29.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj4 td/osd-backfill-space/datafile 2021-01-15T17:37:30.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj5 td/osd-backfill-space/datafile 2021-01-15T17:37:30.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj6 td/osd-backfill-space/datafile 2021-01-15T17:37:30.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj7 td/osd-backfill-space/datafile 2021-01-15T17:37:30.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj8 td/osd-backfill-space/datafile 2021-01-15T17:37:30.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj9 td/osd-backfill-space/datafile 2021-01-15T17:37:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj10 td/osd-backfill-space/datafile 2021-01-15T17:37:30.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj11 td/osd-backfill-space/datafile 2021-01-15T17:37:30.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj12 td/osd-backfill-space/datafile 2021-01-15T17:37:30.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj13 td/osd-backfill-space/datafile 2021-01-15T17:37:30.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:30.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj14 td/osd-backfill-space/datafile 2021-01-15T17:37:31.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj15 td/osd-backfill-space/datafile 2021-01-15T17:37:31.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj16 td/osd-backfill-space/datafile 2021-01-15T17:37:31.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj17 td/osd-backfill-space/datafile 2021-01-15T17:37:31.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj18 td/osd-backfill-space/datafile 2021-01-15T17:37:31.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj19 td/osd-backfill-space/datafile 2021-01-15T17:37:31.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj20 td/osd-backfill-space/datafile 2021-01-15T17:37:31.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj21 td/osd-backfill-space/datafile 2021-01-15T17:37:31.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj22 td/osd-backfill-space/datafile 2021-01-15T17:37:31.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj23 td/osd-backfill-space/datafile 2021-01-15T17:37:31.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj24 td/osd-backfill-space/datafile 2021-01-15T17:37:31.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj25 td/osd-backfill-space/datafile 2021-01-15T17:37:31.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj26 td/osd-backfill-space/datafile 2021-01-15T17:37:31.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:31.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj27 td/osd-backfill-space/datafile 2021-01-15T17:37:32.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj28 td/osd-backfill-space/datafile 2021-01-15T17:37:32.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj29 td/osd-backfill-space/datafile 2021-01-15T17:37:32.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj30 td/osd-backfill-space/datafile 2021-01-15T17:37:32.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj31 td/osd-backfill-space/datafile 2021-01-15T17:37:32.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj32 td/osd-backfill-space/datafile 2021-01-15T17:37:32.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj33 td/osd-backfill-space/datafile 2021-01-15T17:37:32.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj34 td/osd-backfill-space/datafile 2021-01-15T17:37:32.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj35 td/osd-backfill-space/datafile 2021-01-15T17:37:32.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj36 td/osd-backfill-space/datafile 2021-01-15T17:37:32.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj37 td/osd-backfill-space/datafile 2021-01-15T17:37:32.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:32.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj38 td/osd-backfill-space/datafile 2021-01-15T17:37:33.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj39 td/osd-backfill-space/datafile 2021-01-15T17:37:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj40 td/osd-backfill-space/datafile 2021-01-15T17:37:33.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj41 td/osd-backfill-space/datafile 2021-01-15T17:37:33.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj42 td/osd-backfill-space/datafile 2021-01-15T17:37:33.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj43 td/osd-backfill-space/datafile 2021-01-15T17:37:33.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj44 td/osd-backfill-space/datafile 2021-01-15T17:37:33.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj45 td/osd-backfill-space/datafile 2021-01-15T17:37:33.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj46 td/osd-backfill-space/datafile 2021-01-15T17:37:33.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj47 td/osd-backfill-space/datafile 2021-01-15T17:37:33.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj48 td/osd-backfill-space/datafile 2021-01-15T17:37:33.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj49 td/osd-backfill-space/datafile 2021-01-15T17:37:33.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj50 td/osd-backfill-space/datafile 2021-01-15T17:37:33.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:33.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj51 td/osd-backfill-space/datafile 2021-01-15T17:37:34.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj52 td/osd-backfill-space/datafile 2021-01-15T17:37:34.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj53 td/osd-backfill-space/datafile 2021-01-15T17:37:34.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj54 td/osd-backfill-space/datafile 2021-01-15T17:37:34.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj55 td/osd-backfill-space/datafile 2021-01-15T17:37:34.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj56 td/osd-backfill-space/datafile 2021-01-15T17:37:34.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj57 td/osd-backfill-space/datafile 2021-01-15T17:37:34.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj58 td/osd-backfill-space/datafile 2021-01-15T17:37:34.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj59 td/osd-backfill-space/datafile 2021-01-15T17:37:34.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj60 td/osd-backfill-space/datafile 2021-01-15T17:37:34.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj61 td/osd-backfill-space/datafile 2021-01-15T17:37:34.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj62 td/osd-backfill-space/datafile 2021-01-15T17:37:34.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj63 td/osd-backfill-space/datafile 2021-01-15T17:37:34.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:34.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj64 td/osd-backfill-space/datafile 2021-01-15T17:37:35.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj65 td/osd-backfill-space/datafile 2021-01-15T17:37:35.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj66 td/osd-backfill-space/datafile 2021-01-15T17:37:35.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj67 td/osd-backfill-space/datafile 2021-01-15T17:37:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj68 td/osd-backfill-space/datafile 2021-01-15T17:37:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj69 td/osd-backfill-space/datafile 2021-01-15T17:37:35.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj70 td/osd-backfill-space/datafile 2021-01-15T17:37:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj71 td/osd-backfill-space/datafile 2021-01-15T17:37:35.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj72 td/osd-backfill-space/datafile 2021-01-15T17:37:35.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj73 td/osd-backfill-space/datafile 2021-01-15T17:37:35.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj74 td/osd-backfill-space/datafile 2021-01-15T17:37:35.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj75 td/osd-backfill-space/datafile 2021-01-15T17:37:35.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj76 td/osd-backfill-space/datafile 2021-01-15T17:37:35.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:35.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj77 td/osd-backfill-space/datafile 2021-01-15T17:37:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj78 td/osd-backfill-space/datafile 2021-01-15T17:37:36.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj79 td/osd-backfill-space/datafile 2021-01-15T17:37:36.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj80 td/osd-backfill-space/datafile 2021-01-15T17:37:36.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj81 td/osd-backfill-space/datafile 2021-01-15T17:37:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj82 td/osd-backfill-space/datafile 2021-01-15T17:37:36.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj83 td/osd-backfill-space/datafile 2021-01-15T17:37:36.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj84 td/osd-backfill-space/datafile 2021-01-15T17:37:36.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj85 td/osd-backfill-space/datafile 2021-01-15T17:37:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj86 td/osd-backfill-space/datafile 2021-01-15T17:37:36.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj87 td/osd-backfill-space/datafile 2021-01-15T17:37:36.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj88 td/osd-backfill-space/datafile 2021-01-15T17:37:36.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:36.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj89 td/osd-backfill-space/datafile 2021-01-15T17:37:37.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj90 td/osd-backfill-space/datafile 2021-01-15T17:37:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj91 td/osd-backfill-space/datafile 2021-01-15T17:37:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj92 td/osd-backfill-space/datafile 2021-01-15T17:37:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj93 td/osd-backfill-space/datafile 2021-01-15T17:37:37.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj94 td/osd-backfill-space/datafile 2021-01-15T17:37:37.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj95 td/osd-backfill-space/datafile 2021-01-15T17:37:37.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj96 td/osd-backfill-space/datafile 2021-01-15T17:37:37.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj97 td/osd-backfill-space/datafile 2021-01-15T17:37:37.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj98 td/osd-backfill-space/datafile 2021-01-15T17:37:37.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj99 td/osd-backfill-space/datafile 2021-01-15T17:37:37.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj100 td/osd-backfill-space/datafile 2021-01-15T17:37:37.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:37.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj101 td/osd-backfill-space/datafile 2021-01-15T17:37:38.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj102 td/osd-backfill-space/datafile 2021-01-15T17:37:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj103 td/osd-backfill-space/datafile 2021-01-15T17:37:38.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj104 td/osd-backfill-space/datafile 2021-01-15T17:37:38.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj105 td/osd-backfill-space/datafile 2021-01-15T17:37:38.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj106 td/osd-backfill-space/datafile 2021-01-15T17:37:38.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj107 td/osd-backfill-space/datafile 2021-01-15T17:37:38.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj108 td/osd-backfill-space/datafile 2021-01-15T17:37:38.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj109 td/osd-backfill-space/datafile 2021-01-15T17:37:38.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj110 td/osd-backfill-space/datafile 2021-01-15T17:37:38.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj111 td/osd-backfill-space/datafile 2021-01-15T17:37:38.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj112 td/osd-backfill-space/datafile 2021-01-15T17:37:38.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:38.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj113 td/osd-backfill-space/datafile 2021-01-15T17:37:39.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj114 td/osd-backfill-space/datafile 2021-01-15T17:37:39.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj115 td/osd-backfill-space/datafile 2021-01-15T17:37:39.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj116 td/osd-backfill-space/datafile 2021-01-15T17:37:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj117 td/osd-backfill-space/datafile 2021-01-15T17:37:39.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj118 td/osd-backfill-space/datafile 2021-01-15T17:37:39.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj119 td/osd-backfill-space/datafile 2021-01-15T17:37:39.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj120 td/osd-backfill-space/datafile 2021-01-15T17:37:39.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj121 td/osd-backfill-space/datafile 2021-01-15T17:37:39.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj122 td/osd-backfill-space/datafile 2021-01-15T17:37:39.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj123 td/osd-backfill-space/datafile 2021-01-15T17:37:39.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj124 td/osd-backfill-space/datafile 2021-01-15T17:37:39.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj125 td/osd-backfill-space/datafile 2021-01-15T17:37:39.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj126 td/osd-backfill-space/datafile 2021-01-15T17:37:39.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:39.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj127 td/osd-backfill-space/datafile 2021-01-15T17:37:40.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj128 td/osd-backfill-space/datafile 2021-01-15T17:37:40.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj129 td/osd-backfill-space/datafile 2021-01-15T17:37:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj130 td/osd-backfill-space/datafile 2021-01-15T17:37:40.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj131 td/osd-backfill-space/datafile 2021-01-15T17:37:40.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj132 td/osd-backfill-space/datafile 2021-01-15T17:37:40.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj133 td/osd-backfill-space/datafile 2021-01-15T17:37:40.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj134 td/osd-backfill-space/datafile 2021-01-15T17:37:40.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj135 td/osd-backfill-space/datafile 2021-01-15T17:37:40.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj136 td/osd-backfill-space/datafile 2021-01-15T17:37:40.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj137 td/osd-backfill-space/datafile 2021-01-15T17:37:40.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:40.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj138 td/osd-backfill-space/datafile 2021-01-15T17:37:41.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj139 td/osd-backfill-space/datafile 2021-01-15T17:37:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj140 td/osd-backfill-space/datafile 2021-01-15T17:37:41.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj141 td/osd-backfill-space/datafile 2021-01-15T17:37:41.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj142 td/osd-backfill-space/datafile 2021-01-15T17:37:41.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj143 td/osd-backfill-space/datafile 2021-01-15T17:37:41.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj144 td/osd-backfill-space/datafile 2021-01-15T17:37:41.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj145 td/osd-backfill-space/datafile 2021-01-15T17:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj146 td/osd-backfill-space/datafile 2021-01-15T17:37:41.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj147 td/osd-backfill-space/datafile 2021-01-15T17:37:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:41.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj148 td/osd-backfill-space/datafile 2021-01-15T17:37:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj149 td/osd-backfill-space/datafile 2021-01-15T17:37:42.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj150 td/osd-backfill-space/datafile 2021-01-15T17:37:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj151 td/osd-backfill-space/datafile 2021-01-15T17:37:42.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj152 td/osd-backfill-space/datafile 2021-01-15T17:37:42.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj153 td/osd-backfill-space/datafile 2021-01-15T17:37:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj154 td/osd-backfill-space/datafile 2021-01-15T17:37:42.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj155 td/osd-backfill-space/datafile 2021-01-15T17:37:42.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj156 td/osd-backfill-space/datafile 2021-01-15T17:37:42.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj157 td/osd-backfill-space/datafile 2021-01-15T17:37:42.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj158 td/osd-backfill-space/datafile 2021-01-15T17:37:42.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj159 td/osd-backfill-space/datafile 2021-01-15T17:37:42.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj160 td/osd-backfill-space/datafile 2021-01-15T17:37:42.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:42.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj161 td/osd-backfill-space/datafile 2021-01-15T17:37:43.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj162 td/osd-backfill-space/datafile 2021-01-15T17:37:43.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj163 td/osd-backfill-space/datafile 2021-01-15T17:37:43.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj164 td/osd-backfill-space/datafile 2021-01-15T17:37:43.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj165 td/osd-backfill-space/datafile 2021-01-15T17:37:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj166 td/osd-backfill-space/datafile 2021-01-15T17:37:43.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj167 td/osd-backfill-space/datafile 2021-01-15T17:37:43.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj168 td/osd-backfill-space/datafile 2021-01-15T17:37:43.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj169 td/osd-backfill-space/datafile 2021-01-15T17:37:43.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj170 td/osd-backfill-space/datafile 2021-01-15T17:37:43.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj171 td/osd-backfill-space/datafile 2021-01-15T17:37:43.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj172 td/osd-backfill-space/datafile 2021-01-15T17:37:43.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:43.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj173 td/osd-backfill-space/datafile 2021-01-15T17:37:44.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj174 td/osd-backfill-space/datafile 2021-01-15T17:37:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj175 td/osd-backfill-space/datafile 2021-01-15T17:37:44.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj176 td/osd-backfill-space/datafile 2021-01-15T17:37:44.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj177 td/osd-backfill-space/datafile 2021-01-15T17:37:44.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj178 td/osd-backfill-space/datafile 2021-01-15T17:37:44.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj179 td/osd-backfill-space/datafile 2021-01-15T17:37:44.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj180 td/osd-backfill-space/datafile 2021-01-15T17:37:44.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj181 td/osd-backfill-space/datafile 2021-01-15T17:37:44.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj182 td/osd-backfill-space/datafile 2021-01-15T17:37:44.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj183 td/osd-backfill-space/datafile 2021-01-15T17:37:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj184 td/osd-backfill-space/datafile 2021-01-15T17:37:44.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:44.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj185 td/osd-backfill-space/datafile 2021-01-15T17:37:45.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj186 td/osd-backfill-space/datafile 2021-01-15T17:37:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj187 td/osd-backfill-space/datafile 2021-01-15T17:37:45.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj188 td/osd-backfill-space/datafile 2021-01-15T17:37:45.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj189 td/osd-backfill-space/datafile 2021-01-15T17:37:45.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj190 td/osd-backfill-space/datafile 2021-01-15T17:37:45.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj191 td/osd-backfill-space/datafile 2021-01-15T17:37:45.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj192 td/osd-backfill-space/datafile 2021-01-15T17:37:45.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj193 td/osd-backfill-space/datafile 2021-01-15T17:37:45.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj194 td/osd-backfill-space/datafile 2021-01-15T17:37:45.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj195 td/osd-backfill-space/datafile 2021-01-15T17:37:45.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj196 td/osd-backfill-space/datafile 2021-01-15T17:37:45.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj197 td/osd-backfill-space/datafile 2021-01-15T17:37:45.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:45.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj198 td/osd-backfill-space/datafile 2021-01-15T17:37:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj199 td/osd-backfill-space/datafile 2021-01-15T17:37:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:652: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:37:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:654: TEST_ec_backfill_simple: rados -p fillpool put obj200 td/osd-backfill-space/datafile 2021-01-15T17:37:46.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:657: TEST_ec_backfill_simple: get_primary fillpool obj1 2021-01-15T17:37:46.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=fillpool 2021-01-15T17:37:46.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:37:46.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map fillpool obj1 2021-01-15T17:37:46.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:37:46.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:657: TEST_ec_backfill_simple: local fillosd=1 2021-01-15T17:37:46.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:658: TEST_ec_backfill_simple: expr 1 + 1 2021-01-15T17:37:46.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:658: TEST_ec_backfill_simple: osd=2 2021-01-15T17:37:46.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:659: TEST_ec_backfill_simple: '[' 2 = 6 ']' 2021-01-15T17:37:46.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:663: TEST_ec_backfill_simple: sleep 5 2021-01-15T17:37:51.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:664: TEST_ec_backfill_simple: kill_daemon td/osd-backfill-space/osd.1.pid TERM 2021-01-15T17:37:51.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.1.pid 2021-01-15T17:37:51.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=500533 2021-01-15T17:37:51.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2021-01-15T17:37:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2021-01-15T17:37:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2021-01-15T17:37:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:37:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 500533 2021-01-15T17:37:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2021-01-15T17:37:51.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2021-01-15T17:37:51.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2021-01-15T17:37:51.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:37:51.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 500533 2021-01-15T17:37:51.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2021-01-15T17:37:51.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2021-01-15T17:37:51.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2021-01-15T17:37:51.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:665: TEST_ec_backfill_simple: ceph osd out osd.1 2021-01-15T17:37:52.291 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:37:52.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:666: TEST_ec_backfill_simple: sleep 2 2021-01-15T17:37:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:667: TEST_ec_backfill_simple: ceph osd erasure-code-profile set ec-profile k=3 m=2 crush-failure-domain=osd technique=reed_sol_van plugin=jerasure 2021-01-15T17:37:54.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:669: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:37:54.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:669: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:37:54.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:671: TEST_ec_backfill_simple: ceph osd pool create test1 1 1 erasure ec-profile 2021-01-15T17:37:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T17:37:55.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:676: TEST_ec_backfill_simple: sleep 5 2021-01-15T17:38:00.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:678: TEST_ec_backfill_simple: ceph pg dump pgs 2021-01-15T17:38:00.962 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:38:00.963 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 unknown 2021-01-15T17:37:55.350496+0000 0'0 0:0 [] -1 [] -1 0'0 2021-01-15T17:37:55.350496+0000 0'0 2021-01-15T17:37:55.350496+0000 0 2021-01-15T17:38:00.963 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 stale+active+clean 2021-01-15T17:37:29.638742+0000 38'200 38:218 [1] 1 [1] 1 0'0 2021-01-15T17:37:27.496397+0000 0'0 2021-01-15T17:37:27.496397+0000 0 2021-01-15T17:38:00.963 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:38:00.963 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:38:00.964 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:38:00.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:680: TEST_ec_backfill_simple: dd if=/dev/urandom of=td/osd-backfill-space/datafile bs=1024 count=18 2021-01-15T17:38:00.974 INFO:tasks.workunit.client.0.smithi145.stderr:18+0 records in 2021-01-15T17:38:00.975 INFO:tasks.workunit.client.0.smithi145.stderr:18+0 records out 2021-01-15T17:38:00.975 INFO:tasks.workunit.client.0.smithi145.stderr:18432 bytes (18 kB, 18 KiB) copied, 0.000201015 s, 91.7 MB/s 2021-01-15T17:38:00.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: seq 1 200 2021-01-15T17:38:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:00.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:00.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj1 td/osd-backfill-space/datafile 2021-01-15T17:38:01.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:01.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:01.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:01.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj2 td/osd-backfill-space/datafile 2021-01-15T17:38:01.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:01.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:01.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:01.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj3 td/osd-backfill-space/datafile 2021-01-15T17:38:01.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:01.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:01.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:01.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj4 td/osd-backfill-space/datafile 2021-01-15T17:38:01.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:01.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:01.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:01.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj5 td/osd-backfill-space/datafile 2021-01-15T17:38:02.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:02.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:02.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:02.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj6 td/osd-backfill-space/datafile 2021-01-15T17:38:02.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:02.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:02.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:02.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj7 td/osd-backfill-space/datafile 2021-01-15T17:38:02.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:02.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:02.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:02.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj8 td/osd-backfill-space/datafile 2021-01-15T17:38:02.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:02.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:02.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:02.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj9 td/osd-backfill-space/datafile 2021-01-15T17:38:03.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:03.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:03.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj10 td/osd-backfill-space/datafile 2021-01-15T17:38:03.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:03.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:03.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:03.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj11 td/osd-backfill-space/datafile 2021-01-15T17:38:03.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:03.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:03.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:03.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj12 td/osd-backfill-space/datafile 2021-01-15T17:38:03.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:03.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:03.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:03.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj13 td/osd-backfill-space/datafile 2021-01-15T17:38:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:04.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:04.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:04.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj14 td/osd-backfill-space/datafile 2021-01-15T17:38:04.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:04.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj15 td/osd-backfill-space/datafile 2021-01-15T17:38:04.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:04.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:04.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:04.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj16 td/osd-backfill-space/datafile 2021-01-15T17:38:05.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:05.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj17 td/osd-backfill-space/datafile 2021-01-15T17:38:05.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:05.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:05.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:05.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj18 td/osd-backfill-space/datafile 2021-01-15T17:38:05.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:05.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:05.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:05.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj19 td/osd-backfill-space/datafile 2021-01-15T17:38:05.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:05.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:05.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:05.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj20 td/osd-backfill-space/datafile 2021-01-15T17:38:06.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:06.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj21 td/osd-backfill-space/datafile 2021-01-15T17:38:06.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:06.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:06.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:06.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj22 td/osd-backfill-space/datafile 2021-01-15T17:38:06.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:06.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:06.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:06.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj23 td/osd-backfill-space/datafile 2021-01-15T17:38:06.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:06.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:06.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:06.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj24 td/osd-backfill-space/datafile 2021-01-15T17:38:07.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:07.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:07.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj25 td/osd-backfill-space/datafile 2021-01-15T17:38:07.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:07.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj26 td/osd-backfill-space/datafile 2021-01-15T17:38:07.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:07.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:07.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:07.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj27 td/osd-backfill-space/datafile 2021-01-15T17:38:07.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:07.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:07.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:07.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj28 td/osd-backfill-space/datafile 2021-01-15T17:38:08.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:08.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:08.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:08.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj29 td/osd-backfill-space/datafile 2021-01-15T17:38:08.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:08.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:08.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:08.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj30 td/osd-backfill-space/datafile 2021-01-15T17:38:08.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:08.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:08.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:08.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj31 td/osd-backfill-space/datafile 2021-01-15T17:38:09.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:09.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:09.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:09.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj32 td/osd-backfill-space/datafile 2021-01-15T17:38:09.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:09.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:09.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:09.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj33 td/osd-backfill-space/datafile 2021-01-15T17:38:09.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:09.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:09.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:09.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj34 td/osd-backfill-space/datafile 2021-01-15T17:38:09.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:09.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:09.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:09.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj35 td/osd-backfill-space/datafile 2021-01-15T17:38:09.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:09.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:09.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj36 td/osd-backfill-space/datafile 2021-01-15T17:38:10.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:10.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:10.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:10.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj37 td/osd-backfill-space/datafile 2021-01-15T17:38:10.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:10.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:10.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:10.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj38 td/osd-backfill-space/datafile 2021-01-15T17:38:10.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:10.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:10.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj39 td/osd-backfill-space/datafile 2021-01-15T17:38:11.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:11.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:11.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:11.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj40 td/osd-backfill-space/datafile 2021-01-15T17:38:11.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:11.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:11.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:11.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj41 td/osd-backfill-space/datafile 2021-01-15T17:38:11.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:11.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:11.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:11.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj42 td/osd-backfill-space/datafile 2021-01-15T17:38:11.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:11.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:11.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:11.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj43 td/osd-backfill-space/datafile 2021-01-15T17:38:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:11.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj44 td/osd-backfill-space/datafile 2021-01-15T17:38:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:12.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:12.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj45 td/osd-backfill-space/datafile 2021-01-15T17:38:12.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:12.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:12.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:12.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj46 td/osd-backfill-space/datafile 2021-01-15T17:38:12.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:12.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:12.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:12.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj47 td/osd-backfill-space/datafile 2021-01-15T17:38:13.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:13.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:13.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:13.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj48 td/osd-backfill-space/datafile 2021-01-15T17:38:13.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:13.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj49 td/osd-backfill-space/datafile 2021-01-15T17:38:13.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:13.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:13.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:13.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj50 td/osd-backfill-space/datafile 2021-01-15T17:38:13.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:13.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:13.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:13.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj51 td/osd-backfill-space/datafile 2021-01-15T17:38:14.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:14.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:14.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:14.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj52 td/osd-backfill-space/datafile 2021-01-15T17:38:14.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:14.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:14.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:14.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj53 td/osd-backfill-space/datafile 2021-01-15T17:38:14.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:14.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:14.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:14.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj54 td/osd-backfill-space/datafile 2021-01-15T17:38:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:15.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj55 td/osd-backfill-space/datafile 2021-01-15T17:38:15.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:15.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj56 td/osd-backfill-space/datafile 2021-01-15T17:38:15.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:15.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:15.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:15.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj57 td/osd-backfill-space/datafile 2021-01-15T17:38:15.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:15.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:15.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:15.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj58 td/osd-backfill-space/datafile 2021-01-15T17:38:15.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:15.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:15.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:15.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj59 td/osd-backfill-space/datafile 2021-01-15T17:38:16.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:16.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:16.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:16.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj60 td/osd-backfill-space/datafile 2021-01-15T17:38:16.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:16.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:16.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:16.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj61 td/osd-backfill-space/datafile 2021-01-15T17:38:16.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:16.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:16.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:16.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj62 td/osd-backfill-space/datafile 2021-01-15T17:38:17.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:17.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:17.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:17.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj63 td/osd-backfill-space/datafile 2021-01-15T17:38:17.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:17.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:17.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:17.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj64 td/osd-backfill-space/datafile 2021-01-15T17:38:17.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:17.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:17.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:17.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj65 td/osd-backfill-space/datafile 2021-01-15T17:38:17.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:17.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:17.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:17.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj66 td/osd-backfill-space/datafile 2021-01-15T17:38:17.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:17.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:17.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:17.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj67 td/osd-backfill-space/datafile 2021-01-15T17:38:18.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:18.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:18.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:18.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj68 td/osd-backfill-space/datafile 2021-01-15T17:38:18.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:18.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:18.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj69 td/osd-backfill-space/datafile 2021-01-15T17:38:18.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:18.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:18.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj70 td/osd-backfill-space/datafile 2021-01-15T17:38:19.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:19.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:19.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:19.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj71 td/osd-backfill-space/datafile 2021-01-15T17:38:19.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:19.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:19.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:19.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj72 td/osd-backfill-space/datafile 2021-01-15T17:38:19.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:19.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:19.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:19.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj73 td/osd-backfill-space/datafile 2021-01-15T17:38:19.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:19.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:19.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:19.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj74 td/osd-backfill-space/datafile 2021-01-15T17:38:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:20.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:20.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:20.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj75 td/osd-backfill-space/datafile 2021-01-15T17:38:20.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:20.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:20.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:20.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj76 td/osd-backfill-space/datafile 2021-01-15T17:38:20.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:20.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:20.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:20.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj77 td/osd-backfill-space/datafile 2021-01-15T17:38:20.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:20.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:20.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:20.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj78 td/osd-backfill-space/datafile 2021-01-15T17:38:21.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:21.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:21.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:21.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj79 td/osd-backfill-space/datafile 2021-01-15T17:38:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:21.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:21.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:21.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj80 td/osd-backfill-space/datafile 2021-01-15T17:38:21.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:21.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:21.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj81 td/osd-backfill-space/datafile 2021-01-15T17:38:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:22.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:22.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:22.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj82 td/osd-backfill-space/datafile 2021-01-15T17:38:22.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:22.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:22.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj83 td/osd-backfill-space/datafile 2021-01-15T17:38:22.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:22.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:22.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:22.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj84 td/osd-backfill-space/datafile 2021-01-15T17:38:22.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:22.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:22.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:22.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj85 td/osd-backfill-space/datafile 2021-01-15T17:38:22.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:22.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj86 td/osd-backfill-space/datafile 2021-01-15T17:38:23.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:23.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:23.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:23.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj87 td/osd-backfill-space/datafile 2021-01-15T17:38:23.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:23.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj88 td/osd-backfill-space/datafile 2021-01-15T17:38:23.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:23.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:23.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:23.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj89 td/osd-backfill-space/datafile 2021-01-15T17:38:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:24.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj90 td/osd-backfill-space/datafile 2021-01-15T17:38:24.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:24.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:24.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:24.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj91 td/osd-backfill-space/datafile 2021-01-15T17:38:24.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:24.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:24.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:24.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj92 td/osd-backfill-space/datafile 2021-01-15T17:38:24.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:24.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:24.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:24.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj93 td/osd-backfill-space/datafile 2021-01-15T17:38:25.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:25.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:25.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:25.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj94 td/osd-backfill-space/datafile 2021-01-15T17:38:25.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:25.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:25.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:25.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj95 td/osd-backfill-space/datafile 2021-01-15T17:38:25.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:25.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:25.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:25.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj96 td/osd-backfill-space/datafile 2021-01-15T17:38:25.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:25.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:25.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:25.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj97 td/osd-backfill-space/datafile 2021-01-15T17:38:26.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:26.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:26.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:26.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj98 td/osd-backfill-space/datafile 2021-01-15T17:38:26.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:26.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:26.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:26.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj99 td/osd-backfill-space/datafile 2021-01-15T17:38:26.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:26.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:26.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:26.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj100 td/osd-backfill-space/datafile 2021-01-15T17:38:26.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:26.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:26.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:26.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj101 td/osd-backfill-space/datafile 2021-01-15T17:38:27.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:27.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:27.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:27.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj102 td/osd-backfill-space/datafile 2021-01-15T17:38:27.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:27.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:27.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:27.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj103 td/osd-backfill-space/datafile 2021-01-15T17:38:27.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:27.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:27.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj104 td/osd-backfill-space/datafile 2021-01-15T17:38:27.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:27.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:27.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:27.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj105 td/osd-backfill-space/datafile 2021-01-15T17:38:28.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:28.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:28.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:28.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj106 td/osd-backfill-space/datafile 2021-01-15T17:38:28.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:28.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:28.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:28.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj107 td/osd-backfill-space/datafile 2021-01-15T17:38:28.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:28.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:28.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:28.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj108 td/osd-backfill-space/datafile 2021-01-15T17:38:28.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:28.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj109 td/osd-backfill-space/datafile 2021-01-15T17:38:29.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:29.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj110 td/osd-backfill-space/datafile 2021-01-15T17:38:29.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:29.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:29.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:29.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj111 td/osd-backfill-space/datafile 2021-01-15T17:38:29.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:29.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:29.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:29.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj112 td/osd-backfill-space/datafile 2021-01-15T17:38:29.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:29.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:29.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:29.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj113 td/osd-backfill-space/datafile 2021-01-15T17:38:30.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:30.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:30.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:30.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj114 td/osd-backfill-space/datafile 2021-01-15T17:38:30.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:30.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:30.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:30.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj115 td/osd-backfill-space/datafile 2021-01-15T17:38:30.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:30.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:30.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:30.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj116 td/osd-backfill-space/datafile 2021-01-15T17:38:30.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:30.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:30.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:30.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj117 td/osd-backfill-space/datafile 2021-01-15T17:38:31.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:31.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:31.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:31.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj118 td/osd-backfill-space/datafile 2021-01-15T17:38:31.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:31.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:31.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj119 td/osd-backfill-space/datafile 2021-01-15T17:38:31.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:31.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:31.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:31.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj120 td/osd-backfill-space/datafile 2021-01-15T17:38:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:31.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj121 td/osd-backfill-space/datafile 2021-01-15T17:38:32.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:32.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:32.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:32.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj122 td/osd-backfill-space/datafile 2021-01-15T17:38:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:32.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:32.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:32.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj123 td/osd-backfill-space/datafile 2021-01-15T17:38:32.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:32.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:32.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:32.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj124 td/osd-backfill-space/datafile 2021-01-15T17:38:32.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:32.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:32.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:32.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj125 td/osd-backfill-space/datafile 2021-01-15T17:38:33.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:33.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj126 td/osd-backfill-space/datafile 2021-01-15T17:38:33.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:33.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:33.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:33.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj127 td/osd-backfill-space/datafile 2021-01-15T17:38:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:33.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:33.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:33.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj128 td/osd-backfill-space/datafile 2021-01-15T17:38:33.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:33.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:33.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:33.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj129 td/osd-backfill-space/datafile 2021-01-15T17:38:34.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:34.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:34.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:34.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj130 td/osd-backfill-space/datafile 2021-01-15T17:38:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:34.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:34.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:34.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj131 td/osd-backfill-space/datafile 2021-01-15T17:38:34.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:34.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:34.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:34.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj132 td/osd-backfill-space/datafile 2021-01-15T17:38:35.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:35.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:35.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:35.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj133 td/osd-backfill-space/datafile 2021-01-15T17:38:35.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:35.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:35.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:35.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj134 td/osd-backfill-space/datafile 2021-01-15T17:38:35.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:35.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:35.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:35.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj135 td/osd-backfill-space/datafile 2021-01-15T17:38:35.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:35.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:35.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:35.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj136 td/osd-backfill-space/datafile 2021-01-15T17:38:36.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:36.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:36.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:36.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj137 td/osd-backfill-space/datafile 2021-01-15T17:38:36.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:36.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:36.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:36.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj138 td/osd-backfill-space/datafile 2021-01-15T17:38:36.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:36.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:36.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:36.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj139 td/osd-backfill-space/datafile 2021-01-15T17:38:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:36.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:36.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj140 td/osd-backfill-space/datafile 2021-01-15T17:38:37.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:37.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:37.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:37.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj141 td/osd-backfill-space/datafile 2021-01-15T17:38:37.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:37.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:37.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj142 td/osd-backfill-space/datafile 2021-01-15T17:38:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:37.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj143 td/osd-backfill-space/datafile 2021-01-15T17:38:37.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:37.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:37.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:37.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj144 td/osd-backfill-space/datafile 2021-01-15T17:38:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:38.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj145 td/osd-backfill-space/datafile 2021-01-15T17:38:38.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:38.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:38.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:38.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj146 td/osd-backfill-space/datafile 2021-01-15T17:38:38.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:38.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:38.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:38.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj147 td/osd-backfill-space/datafile 2021-01-15T17:38:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:38.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj148 td/osd-backfill-space/datafile 2021-01-15T17:38:39.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:39.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:39.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:39.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj149 td/osd-backfill-space/datafile 2021-01-15T17:38:39.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:39.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:39.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:39.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj150 td/osd-backfill-space/datafile 2021-01-15T17:38:39.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:39.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:39.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:39.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj151 td/osd-backfill-space/datafile 2021-01-15T17:38:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:39.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj152 td/osd-backfill-space/datafile 2021-01-15T17:38:40.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:40.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:40.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:40.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj153 td/osd-backfill-space/datafile 2021-01-15T17:38:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:40.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj154 td/osd-backfill-space/datafile 2021-01-15T17:38:40.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:40.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:40.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:40.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj155 td/osd-backfill-space/datafile 2021-01-15T17:38:40.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:40.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:40.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:40.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj156 td/osd-backfill-space/datafile 2021-01-15T17:38:40.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:40.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:40.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:40.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj157 td/osd-backfill-space/datafile 2021-01-15T17:38:41.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:41.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:41.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:41.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj158 td/osd-backfill-space/datafile 2021-01-15T17:38:41.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:41.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:41.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:41.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj159 td/osd-backfill-space/datafile 2021-01-15T17:38:41.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:41.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:41.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:41.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj160 td/osd-backfill-space/datafile 2021-01-15T17:38:41.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:41.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:41.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:41.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj161 td/osd-backfill-space/datafile 2021-01-15T17:38:42.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:42.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:42.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:42.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj162 td/osd-backfill-space/datafile 2021-01-15T17:38:42.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:42.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj163 td/osd-backfill-space/datafile 2021-01-15T17:38:42.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:42.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:42.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:42.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj164 td/osd-backfill-space/datafile 2021-01-15T17:38:42.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:42.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:42.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:42.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj165 td/osd-backfill-space/datafile 2021-01-15T17:38:43.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:43.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:43.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:43.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj166 td/osd-backfill-space/datafile 2021-01-15T17:38:43.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:43.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:43.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:43.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj167 td/osd-backfill-space/datafile 2021-01-15T17:38:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:43.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:43.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:43.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj168 td/osd-backfill-space/datafile 2021-01-15T17:38:44.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:44.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:44.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:44.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj169 td/osd-backfill-space/datafile 2021-01-15T17:38:44.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:44.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:44.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj170 td/osd-backfill-space/datafile 2021-01-15T17:38:44.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:44.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:44.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:44.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj171 td/osd-backfill-space/datafile 2021-01-15T17:38:44.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:44.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:44.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:44.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj172 td/osd-backfill-space/datafile 2021-01-15T17:38:45.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:45.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj173 td/osd-backfill-space/datafile 2021-01-15T17:38:45.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:45.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:45.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:45.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj174 td/osd-backfill-space/datafile 2021-01-15T17:38:45.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:45.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:45.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:45.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj175 td/osd-backfill-space/datafile 2021-01-15T17:38:45.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:45.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:45.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:45.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj176 td/osd-backfill-space/datafile 2021-01-15T17:38:46.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:46.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:46.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:46.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj177 td/osd-backfill-space/datafile 2021-01-15T17:38:46.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:46.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:46.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:46.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj178 td/osd-backfill-space/datafile 2021-01-15T17:38:46.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:46.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:46.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj179 td/osd-backfill-space/datafile 2021-01-15T17:38:46.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:46.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:46.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj180 td/osd-backfill-space/datafile 2021-01-15T17:38:47.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:47.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:47.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:47.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj181 td/osd-backfill-space/datafile 2021-01-15T17:38:47.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:47.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:47.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:47.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj182 td/osd-backfill-space/datafile 2021-01-15T17:38:47.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:47.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:47.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:47.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj183 td/osd-backfill-space/datafile 2021-01-15T17:38:47.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:47.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:47.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:47.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj184 td/osd-backfill-space/datafile 2021-01-15T17:38:47.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:47.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:47.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:47.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj185 td/osd-backfill-space/datafile 2021-01-15T17:38:48.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:48.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:48.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:48.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj186 td/osd-backfill-space/datafile 2021-01-15T17:38:48.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:48.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:48.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:48.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj187 td/osd-backfill-space/datafile 2021-01-15T17:38:48.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:48.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:48.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:48.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj188 td/osd-backfill-space/datafile 2021-01-15T17:38:48.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:48.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:48.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:48.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj189 td/osd-backfill-space/datafile 2021-01-15T17:38:49.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:49.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj190 td/osd-backfill-space/datafile 2021-01-15T17:38:49.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:49.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:49.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:49.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj191 td/osd-backfill-space/datafile 2021-01-15T17:38:49.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:49.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:49.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:49.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj192 td/osd-backfill-space/datafile 2021-01-15T17:38:49.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:49.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj193 td/osd-backfill-space/datafile 2021-01-15T17:38:50.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:50.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:50.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:50.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj194 td/osd-backfill-space/datafile 2021-01-15T17:38:50.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:50.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:50.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj195 td/osd-backfill-space/datafile 2021-01-15T17:38:50.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:50.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:50.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:50.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj196 td/osd-backfill-space/datafile 2021-01-15T17:38:50.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:50.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:50.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:50.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj197 td/osd-backfill-space/datafile 2021-01-15T17:38:51.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:51.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:51.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:51.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj198 td/osd-backfill-space/datafile 2021-01-15T17:38:51.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:51.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:51.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj199 td/osd-backfill-space/datafile 2021-01-15T17:38:51.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:681: TEST_ec_backfill_simple: for o in $(seq 1 $ecobjects) 2021-01-15T17:38:51.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:38:51.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:683: TEST_ec_backfill_simple: for p in $(seq 1 $pools) 2021-01-15T17:38:51.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:685: TEST_ec_backfill_simple: rados -p test1 put obj200 td/osd-backfill-space/datafile 2021-01-15T17:38:51.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:689: TEST_ec_backfill_simple: kill_daemon td/osd-backfill-space/osd.2.pid TERM 2021-01-15T17:38:51.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/osd-backfill-space/osd.2.pid 2021-01-15T17:38:51.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=501797 2021-01-15T17:38:51.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2021-01-15T17:38:51.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2021-01-15T17:38:51.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2021-01-15T17:38:51.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:38:51.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 501797 2021-01-15T17:38:51.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2021-01-15T17:38:51.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2021-01-15T17:38:51.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2021-01-15T17:38:51.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2021-01-15T17:38:51.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 501797 2021-01-15T17:38:51.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2021-01-15T17:38:51.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2021-01-15T17:38:51.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2021-01-15T17:38:51.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:690: TEST_ec_backfill_simple: ceph osd out osd.2 2021-01-15T17:38:52.842 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T17:38:52.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:692: TEST_ec_backfill_simple: activate_osd td/osd-backfill-space 1 2021-01-15T17:38:52.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-backfill-space 2021-01-15T17:38:52.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T17:38:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T17:38:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T17:38:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-backfill-space/1 2021-01-15T17:38:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 ' 2021-01-15T17:38:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:38:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:38:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:38:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2021-01-15T17:38:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2021-01-15T17:38:52.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T17:38:52.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T17:38:52.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2021-01-15T17:38:52.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T17:38:52.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:38:52.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:38:52.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:38:52.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:38:52.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:38:52.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:38:52.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' 2021-01-15T17:38:52.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:38:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2021-01-15T17:38:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2021-01-15T17:38:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:38:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:38:52.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:38:52.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T17:38:52.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T17:38:52.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-backfill-space/1 2021-01-15T17:38:52.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T17:38:52.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=83bc673c-2d0b-48f5-bd28-415cc33826c5 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.173017/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:38:52.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-backfill-space/1/whoami 2021-01-15T17:38:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T17:38:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T17:38:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T17:38:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T17:38:52.864 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:38:52.880 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:38:52.879+0000 7f6ebbd26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:38:52.881 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:38:52.881+0000 7f6ebbd26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:38:52.883 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:38:52.883+0000 7f6ebbd26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:38:53.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T17:38:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:38:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:38:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:38:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:38:53.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:38:53.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:38:53.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:38:53.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:38:53.261 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:38:53.408 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:38:53.407+0000 7f6ebbd26080 -1 Falling back to public interface 2021-01-15T17:38:53.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:38:54.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:38:54.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:38:54.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:38:54.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:38:54.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:38:54.708 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:38:54.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:38:54.844+0000 7f6ebbd26080 -1 osd.1 38 log_to_monitors {default=true} 2021-01-15T17:38:55.118 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up out weight 0 up_from 49 up_thru 37 down_at 39 last_clean_interval [10,38) [v2:127.0.0.1:6810/514531,v1:127.0.0.1:6811/514531] [v2:127.0.0.1:6812/514531,v1:127.0.0.1:6813/514531] exists,up e1fad7e3-86cc-48b3-8793-c85d93a4f2fd 2021-01-15T17:38:55.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:38:55.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:38:55.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:38:55.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:693: TEST_ec_backfill_simple: ceph osd in osd.1 2021-01-15T17:38:55.679 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already in. 2021-01-15T17:38:55.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:694: TEST_ec_backfill_simple: sleep 30 2021-01-15T17:39:25.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:696: TEST_ec_backfill_simple: ceph pg dump pgs 2021-01-15T17:39:26.067 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:39:26.067 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 200 0 200 200 0 3686400 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:38:57.390670+0000 45'200 54:243 [3,5,1,0,4] 3 [3,5,4,0,NONE] 3 0'0 2021-01-15T17:37:55.350496+0000 0'0 2021-01-15T17:37:55.350496+0000 0 2021-01-15T17:39:26.067 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 active+clean 2021-01-15T17:38:56.105095+0000 38'200 53:228 [1] 1 [1] 1 0'0 2021-01-15T17:37:27.496397+0000 0'0 2021-01-15T17:37:27.496397+0000 0 2021-01-15T17:39:26.067 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:39:26.068 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:39:26.068 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:39:26.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:698: TEST_ec_backfill_simple: wait_for_not_backfilling 240 2021-01-15T17:39:26.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:82: wait_for_not_backfilling: local timeout=240 2021-01-15T17:39:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:83: wait_for_not_backfilling: wait_for_not_state backfilling 240 2021-01-15T17:39:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=backfilling 2021-01-15T17:39:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:39:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:39:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:39:26.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 240 5 2021-01-15T17:39:26.080 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:39:26.081 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:39:26.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:39:26.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:39:26.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:39:26.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:39:26.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:39:26.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:39:26.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:39:26.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:39:26.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:39:26.602 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:39:26.603 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:39:26.603 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:39:26.603 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:39:26.603 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:39:26.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:39:26.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:26.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:39:26.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836515 2021-01-15T17:39:26.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836515 2021-01-15T17:39:26.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836515' 2021-01-15T17:39:26.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:26.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:39:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397513 2021-01-15T17:39:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397513 2021-01-15T17:39:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836515 1-210453397513' 2021-01-15T17:39:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:26.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:39:26.980 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-01-15T17:39:26.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:39:26.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:39:26.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:39:26.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:26.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:39:27.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345950 2021-01-15T17:39:27.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345950 2021-01-15T17:39:27.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836515 1-210453397513 3-85899345950' 2021-01-15T17:39:27.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:27.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:39:27.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182429 2021-01-15T17:39:27.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182429 2021-01-15T17:39:27.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836515 1-210453397513 3-85899345950 4-107374182429' 2021-01-15T17:39:27.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:27.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:39:27.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018907 2021-01-15T17:39:27.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018907 2021-01-15T17:39:27.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836515 1-210453397513 3-85899345950 4-107374182429 5-128849018907' 2021-01-15T17:39:27.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:27.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836515 2021-01-15T17:39:27.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:27.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:39:27.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836515 2021-01-15T17:39:27.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:27.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836515 2021-01-15T17:39:27.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2021-01-15T17:39:27.403 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836515 2021-01-15T17:39:27.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:39:27.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836512 -lt 21474836515 2021-01-15T17:39:27.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:39:28.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:39:28.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:39:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836515 -lt 21474836515 2021-01-15T17:39:29.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:29.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397513 2021-01-15T17:39:29.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:39:29.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397513 2021-01-15T17:39:29.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:29.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397513 2021-01-15T17:39:29.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397513' 2021-01-15T17:39:29.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:39:29.196 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 210453397513 2021-01-15T17:39:29.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397513 -lt 210453397513 2021-01-15T17:39:29.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:29.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345950 2021-01-15T17:39:29.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:29.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:39:29.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345950 2021-01-15T17:39:29.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:29.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345950 2021-01-15T17:39:29.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2021-01-15T17:39:29.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:39:29.594 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345950 2021-01-15T17:39:30.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345950 -lt 85899345950 2021-01-15T17:39:30.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:30.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182429 2021-01-15T17:39:30.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:30.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:39:30.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182429 2021-01-15T17:39:30.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:30.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182429 2021-01-15T17:39:30.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182429' 2021-01-15T17:39:30.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:39:30.006 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182429 2021-01-15T17:39:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182430 -lt 107374182429 2021-01-15T17:39:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:30.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018907 2021-01-15T17:39:30.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:30.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:39:30.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018907 2021-01-15T17:39:30.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018907 2021-01-15T17:39:30.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018907' 2021-01-15T17:39:30.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:39:30.404 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018907 2021-01-15T17:39:30.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018907 -lt 128849018907 2021-01-15T17:39:30.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:39:30.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:39:30.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:39:31.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:39:31.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:39:31.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state backfilling 2021-01-15T17:39:31.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=backfilling 2021-01-15T17:39:31.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:39:31.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("backfilling"))' 2021-01-15T17:39:31.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:39:31.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("backfilling"))] | length' 2021-01-15T17:39:31.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:39:31.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:39:31.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:39:31.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:39:31.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:699: TEST_ec_backfill_simple: wait_for_not_activating 60 2021-01-15T17:39:31.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:88: wait_for_not_activating: local timeout=60 2021-01-15T17:39:31.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:89: wait_for_not_activating: wait_for_not_state activating 60 2021-01-15T17:39:31.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:53: wait_for_not_state: local state=activating 2021-01-15T17:39:31.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:54: wait_for_not_state: local num_in_state=-1 2021-01-15T17:39:31.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:55: wait_for_not_state: local cur_in_state 2021-01-15T17:39:31.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T17:39:31.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: get_timeout_delays 60 5 2021-01-15T17:39:31.724 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:39:31.724 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:39:31.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:39:31.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:39:31.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:39:31.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:56: wait_for_not_state: local -a delays 2021-01-15T17:39:31.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:57: wait_for_not_state: local -i loop=0 2021-01-15T17:39:31.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:59: wait_for_not_state: flush_pg_stats 2021-01-15T17:39:31.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:39:31.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:39:32.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:39:32.156 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:39:32.156 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:39:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:39:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:39:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:39:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:39:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:39:32.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836518 2021-01-15T17:39:32.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836518 2021-01-15T17:39:32.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836518' 2021-01-15T17:39:32.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:39:32.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=210453397516 2021-01-15T17:39:32.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 210453397516 2021-01-15T17:39:32.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836518 1-210453397516' 2021-01-15T17:39:32.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:39:32.559 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-01-15T17:39:32.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:39:32.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:39:32.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:39:32.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:39:32.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345953 2021-01-15T17:39:32.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345953 2021-01-15T17:39:32.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836518 1-210453397516 3-85899345953' 2021-01-15T17:39:32.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:39:32.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182432 2021-01-15T17:39:32.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182432 2021-01-15T17:39:32.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836518 1-210453397516 3-85899345953 4-107374182432' 2021-01-15T17:39:32.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:39:32.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:39:32.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018910 2021-01-15T17:39:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018910 2021-01-15T17:39:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836518 1-210453397516 3-85899345953 4-107374182432 5-128849018910' 2021-01-15T17:39:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:32.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836518 2021-01-15T17:39:32.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:39:32.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836518 2021-01-15T17:39:32.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836518 2021-01-15T17:39:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2021-01-15T17:39:32.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:39:32.982 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836518 2021-01-15T17:39:33.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836516 -lt 21474836518 2021-01-15T17:39:33.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:39:34.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:39:34.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:39:34.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836518 -lt 21474836518 2021-01-15T17:39:34.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:34.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-210453397516 2021-01-15T17:39:34.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:34.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:39:34.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-210453397516 2021-01-15T17:39:34.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=210453397516 2021-01-15T17:39:34.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 210453397516' 2021-01-15T17:39:34.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:39:34.782 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 210453397516 2021-01-15T17:39:35.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 210453397516 -lt 210453397516 2021-01-15T17:39:35.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:35.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345953 2021-01-15T17:39:35.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:35.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:39:35.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345953 2021-01-15T17:39:35.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:35.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345953 2021-01-15T17:39:35.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2021-01-15T17:39:35.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:39:35.190 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345953 2021-01-15T17:39:35.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345953 -lt 85899345953 2021-01-15T17:39:35.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:35.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182432 2021-01-15T17:39:35.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:35.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:39:35.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182432 2021-01-15T17:39:35.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:35.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182432 2021-01-15T17:39:35.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2021-01-15T17:39:35.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:39:35.589 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182432 2021-01-15T17:39:35.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182433 -lt 107374182432 2021-01-15T17:39:35.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:39:35.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018910 2021-01-15T17:39:35.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:39:35.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:39:35.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018910 2021-01-15T17:39:35.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:39:35.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018910 2021-01-15T17:39:35.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018910' 2021-01-15T17:39:35.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:39:35.985 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018910 2021-01-15T17:39:36.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018910 -lt 128849018910 2021-01-15T17:39:36.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: get_num_pgs 2021-01-15T17:39:36.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:39:36.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:39:36.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:60: wait_for_not_state: test 2 == 0 2021-01-15T17:39:36.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:64: wait_for_not_state: true 2021-01-15T17:39:36.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: get_num_in_state activating 2021-01-15T17:39:36.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:44: get_num_in_state: local state=activating 2021-01-15T17:39:36.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:45: get_num_in_state: local expression 2021-01-15T17:39:36.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:46: get_num_in_state: expression+='select(contains("activating"))' 2021-01-15T17:39:36.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:47: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T17:39:36.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:48: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("activating"))] | length' 2021-01-15T17:39:37.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:65: wait_for_not_state: cur_in_state=0 2021-01-15T17:39:37.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: test 0 = 0 2021-01-15T17:39:37.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:66: wait_for_not_state: break 2021-01-15T17:39:37.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:77: wait_for_not_state: return 0 2021-01-15T17:39:37.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:701: TEST_ec_backfill_simple: ceph pg dump pgs 2021-01-15T17:39:37.654 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:39:37.654 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 200 0 200 200 0 3686400 0 0 100 100 active+undersized+degraded+remapped+backfill_toofull 2021-01-15T17:39:27.391549+0000 45'200 54:247 [3,5,1,0,4] 3 [3,5,4,0,NONE] 3 0'0 2021-01-15T17:37:55.350496+0000 0'0 2021-01-15T17:37:55.350496+0000 0 2021-01-15T17:39:37.655 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2662400 0 0 100 100 active+clean 2021-01-15T17:38:56.105095+0000 38'200 53:228 [1] 1 [1] 1 0'0 2021-01-15T17:37:27.496397+0000 0'0 2021-01-15T17:37:27.496397+0000 0 2021-01-15T17:39:37.655 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:39:37.655 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:39:37.656 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:39:37.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:703: TEST_ec_backfill_simple: ERRORS=0 2021-01-15T17:39:37.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: ceph pg dump pgs 2021-01-15T17:39:37.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: grep -v '^1.0' 2021-01-15T17:39:37.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: grep +backfill_toofull 2021-01-15T17:39:37.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: wc -l 2021-01-15T17:39:38.037 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:39:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:704: TEST_ec_backfill_simple: '[' 1 '!=' 1 ']' 2021-01-15T17:39:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:709: TEST_ec_backfill_simple: '[' 0 '!=' 0 ']' 2021-01-15T17:39:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:714: TEST_ec_backfill_simple: delete_pool fillpool 2021-01-15T17:39:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=fillpool 2021-01-15T17:39:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete fillpool fillpool --yes-i-really-really-mean-it 2021-01-15T17:39:38.586 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'fillpool' does not exist 2021-01-15T17:39:38.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:715: TEST_ec_backfill_simple: seq 1 1 2021-01-15T17:39:38.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:715: TEST_ec_backfill_simple: for i in $(seq 1 $pools) 2021-01-15T17:39:38.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:717: TEST_ec_backfill_simple: delete_pool test1 2021-01-15T17:39:38.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T17:39:38.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T17:39:39.135 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T17:39:39.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:719: TEST_ec_backfill_simple: kill_daemons td/osd-backfill-space 2021-01-15T17:39:39.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:39:39.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:39:39.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:39:39.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:39:39.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:39:39.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:39:39.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-space.sh:38: run: teardown td/osd-backfill-space 2021-01-15T17:39:39.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:39:39.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:39:39.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:39:39.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:39:39.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:39:39.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:39:39.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:39:39.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:39:39.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:39:39.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:39:39.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:39:39.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:39:39.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:39:39.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:39:39.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:39:39.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:39:39.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:39:39.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:39:39.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:39:39.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:39:39.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:39:39.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:39:39.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:39:39.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:39:39.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:39:39.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T17:39:39.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-space 0 2021-01-15T17:39:39.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-space 2021-01-15T17:39:39.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T17:39:39.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-space KILL 2021-01-15T17:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:39:39.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:39:39.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:39:39.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:39:39.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:39:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:39:39.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:39:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:39:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:39:39.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:39:39.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T17:39:39.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-space 2021-01-15T17:39:39.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:39:39.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173017 2021-01-15T17:39:39.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173017 2021-01-15T17:39:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:39:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:39:39.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T17:39:39.300 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T17:39:39.301 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T17:39:39.335 INFO:tasks.workunit:Running workunit osd/osd-backfill-stats.sh... 2021-01-15T17:39:39.336 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-backfill-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh 2021-01-15T17:39:39.395 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T17:39:39.405 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T17:39:39.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T17:39:39.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T17:39:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T17:39:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T17:39:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T17:39:39.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T17:39:39.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T17:39:39.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T17:39:39.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-backfill-stats 2021-01-15T17:39:39.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:21: run: local dir=td/osd-backfill-stats 2021-01-15T17:39:39.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:22: run: shift 2021-01-15T17:39:39.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-01-15T17:39:39.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-01-15T17:39:39.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:26: run: export CEPH_ARGS 2021-01-15T17:39:39.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:27: run: uuidgen 2021-01-15T17:39:39.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:27: run: CEPH_ARGS+='--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none ' 2021-01-15T17:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 ' 2021-01-15T17:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:30: run: export margin=10 2021-01-15T17:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:30: run: margin=10 2021-01-15T17:39:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:31: run: export objects=200 2021-01-15T17:39:39.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:31: run: objects=200 2021-01-15T17:39:39.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:32: run: export poolname=test 2021-01-15T17:39:39.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:32: run: poolname=test 2021-01-15T17:39:39.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: set 2021-01-15T17:39:39.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T17:39:39.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:34: run: local 'funcs=TEST_backfill_down_out 2021-01-15T17:39:39.413 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_ec_all_out 2021-01-15T17:39:39.414 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_ec_down_all_out 2021-01-15T17:39:39.414 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_ec_down_out 2021-01-15T17:39:39.414 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_ec_prim_out 2021-01-15T17:39:39.414 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_out 2021-01-15T17:39:39.414 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_out2 2021-01-15T17:39:39.415 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_remapped 2021-01-15T17:39:39.415 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_sizeup 2021-01-15T17:39:39.415 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_sizeup4_allout 2021-01-15T17:39:39.415 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_backfill_sizeup_out' 2021-01-15T17:39:39.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:39:39.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:39:39.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:39:39.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:39:39.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:39:39.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:39:39.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:39:39.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:39:39.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:39:39.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:39:39.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:39:39.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:39:39.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:39:39.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:39:39.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:39:39.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:39:39.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:39:39.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:39:39.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:39:39.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:39:39.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:39:39.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:39:39.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:39:39.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:39:39.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:39:39.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:39:39.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:39.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:39:39.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:39:39.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:39:39.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:39:39.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:39:39.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:39.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:39:39.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:39:39.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T17:39:39.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T17:39:39.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:39:39.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:39:39.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_down_out td/osd-backfill-stats 2021-01-15T17:39:39.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:278: TEST_backfill_down_out: local dir=td/osd-backfill-stats 2021-01-15T17:39:39.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:280: TEST_backfill_down_out: run_mon td/osd-backfill-stats a 2021-01-15T17:39:39.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:39:39.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:39:39.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:39:39.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:39:39.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:39:39.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:39:39.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:39:39.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:39:39.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:39:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:39:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:39.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:39:39.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:39:39.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:39:39.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:39:39.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:39:39.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:39:39.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:39:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:39:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:39:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:39:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:39:39.886 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.886 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:39.886 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:39:39.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:39:39.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:39:39.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:39:39.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:39:39.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:39:39.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:39:39.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:39:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:39:39.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:39:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:39:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:39:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:39.988 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:39.989 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:39:39.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:39:39.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:39:40.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:281: TEST_backfill_down_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:39:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:39:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:39:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:39:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:39:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:39:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:39:40.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:39:40.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:39:40.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:39:40.577 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:39:40.578 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:40.578 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:40.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:40.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:39:40.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:39:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:282: TEST_backfill_down_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:39:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:39:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:39:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:39:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:39:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:39:40.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:39:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:39:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:39:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:39:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:39:40.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:39:40.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:39:40.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:39:40.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:39:40.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:39:40.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:39:40.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:39:40.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:39:40.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:40.607 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:40.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:40.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:40.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:39:40.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:39:40.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:39:40.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:39:40.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:39:40.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:39:40.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:39:40.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:39:40.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:39:40.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:39:40.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:39:40.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:39:40.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=09c2ba60-45ae-45da-8516-50b9f971a5e5 2021-01-15T17:39:40.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 09c2ba60-45ae-45da-8516-50b9f971a5e5' 2021-01-15T17:39:40.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:39:40.611 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 09c2ba60-45ae-45da-8516-50b9f971a5e5 2021-01-15T17:39:40.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBc0wFgOKbgJBAAfCAlLPHp2ct0EAcdnrltag== 2021-01-15T17:39:40.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBc0wFgOKbgJBAAfCAlLPHp2ct0EAcdnrltag=="}' 2021-01-15T17:39:40.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 09c2ba60-45ae-45da-8516-50b9f971a5e5 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:39:40.958 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:39:40.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:39:40.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBc0wFgOKbgJBAAfCAlLPHp2ct0EAcdnrltag== --osd-uuid 09c2ba60-45ae-45da-8516-50b9f971a5e5 2021-01-15T17:39:40.994 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.994+0000 7fd40379e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:40.996 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.996+0000 7fd40379e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:40.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.997+0000 7fd40379e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:40.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.998+0000 7fd40379e080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:39:40.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.998+0000 7fd40379e080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:39:40.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:40.998+0000 7fd40379e080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:39:41.040 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:41.039+0000 7fd40379e080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:39:45.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:39:45.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:39:45.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:39:45.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:39:45.608 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:39:46.134 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:39:46.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:39:46.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:39:46.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:39:46.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:39:46.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:39:46.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:46.157+0000 7f0752400080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:46.160 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:46.159+0000 7f0752400080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:46.160 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:46.160+0000 7f0752400080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:46.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:39:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:39:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:39:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:39:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:39:46.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:39:46.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:39:46.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:39:46.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:39:46.532 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:39:46.675 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:46.675+0000 7f0752400080 -1 Falling back to public interface 2021-01-15T17:39:46.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:39:47.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:47.464+0000 7f0752400080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:39:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:39:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:39:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:39:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:39:47.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:39:47.926 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:39:48.342 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/517306,v1:127.0.0.1:6803/517306] [v2:127.0.0.1:6804/517306,v1:127.0.0.1:6805/517306] exists,up 09c2ba60-45ae-45da-8516-50b9f971a5e5 2021-01-15T17:39:48.342 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 b8ae333d-2efb-487c-8a38-6ace0dd10ee3 2021-01-15T17:39:48.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:39:48.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:39:48.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:39:48.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:283: TEST_backfill_down_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:39:48.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:39:48.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:39:48.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:39:48.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:39:48.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:39:48.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:39:48.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:39:48.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:39:48.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:39:48.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:39:48.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:39:48.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:39:48.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:39:48.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:39:48.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:39:48.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:39:48.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:39:48.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:39:48.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:48.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:48.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:48.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:39:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:39:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:39:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:39:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:39:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:39:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:39:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:39:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:39:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:39:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:39:48.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:39:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b8ae333d-2efb-487c-8a38-6ace0dd10ee3 2021-01-15T17:39:48.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 b8ae333d-2efb-487c-8a38-6ace0dd10ee3' 2021-01-15T17:39:48.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:39:48.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBk0wFgdQlIFRAApzDSBDZJIYHt3VlZcuvVsA== 2021-01-15T17:39:48.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBk0wFgdQlIFRAApzDSBDZJIYHt3VlZcuvVsA=="}' 2021-01-15T17:39:48.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b8ae333d-2efb-487c-8a38-6ace0dd10ee3 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:39:48.850 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:39:48.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:39:48.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBk0wFgdQlIFRAApzDSBDZJIYHt3VlZcuvVsA== --osd-uuid b8ae333d-2efb-487c-8a38-6ace0dd10ee3 2021-01-15T17:39:48.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.887+0000 7f9921392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:48.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.888+0000 7f9921392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:48.889 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.889+0000 7f9921392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:48.890 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.890+0000 7f9921392080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:39:48.891 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.890+0000 7f9921392080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:39:48.891 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.890+0000 7f9921392080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:39:48.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:48.938+0000 7f9921392080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:39:53.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:39:53.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:39:53.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:39:53.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:39:53.439 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:39:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:39:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:39:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:39:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:39:53.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:39:53.968 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:39:53.991 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:53.991+0000 7f09a3d3e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:54.019 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:54.018+0000 7f09a3d3e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:54.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:54.020+0000 7f09a3d3e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:54.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:39:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:39:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:39:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:39:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:39:54.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:39:54.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:39:54.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:39:54.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:39:54.359 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:39:54.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:54.534+0000 7f09a3d3e080 -1 Falling back to public interface 2021-01-15T17:39:54.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:39:55.529 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:55.528+0000 7f09a3d3e080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:39:55.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:39:55.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:39:55.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:39:55.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:39:55.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:39:55.755 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:39:56.216 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/518570,v1:127.0.0.1:6811/518570] [v2:127.0.0.1:6812/518570,v1:127.0.0.1:6813/518570] exists,up b8ae333d-2efb-487c-8a38-6ace0dd10ee3 2021-01-15T17:39:56.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:39:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:39:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:39:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:284: TEST_backfill_down_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:39:56.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:39:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:39:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:39:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:39:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:39:56.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:39:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:39:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:39:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:39:56.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:39:56.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:39:56.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:39:56.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:39:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:39:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:39:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:39:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:39:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:39:56.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:39:56.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=37a97990-9e24-4411-b7f1-a72d41fe6208 2021-01-15T17:39:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 37a97990-9e24-4411-b7f1-a72d41fe6208' 2021-01-15T17:39:56.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:39:56.229 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 37a97990-9e24-4411-b7f1-a72d41fe6208 2021-01-15T17:39:56.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBs0wFgOhThDRAAWWsOvUypkgBWMMsptMLN+w== 2021-01-15T17:39:56.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBs0wFgOhThDRAAWWsOvUypkgBWMMsptMLN+w=="}' 2021-01-15T17:39:56.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 37a97990-9e24-4411-b7f1-a72d41fe6208 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:39:56.785 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:39:56.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:39:56.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBs0wFgOhThDRAAWWsOvUypkgBWMMsptMLN+w== --osd-uuid 37a97990-9e24-4411-b7f1-a72d41fe6208 2021-01-15T17:39:56.823 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.822+0000 7f405ed53080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:56.844 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.843+0000 7f405ed53080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:56.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.844+0000 7f405ed53080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:39:56.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.845+0000 7f405ed53080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:39:56.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.845+0000 7f405ed53080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:39:56.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.845+0000 7f405ed53080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:39:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:39:56.902+0000 7f405ed53080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:40:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:40:01.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:40:01.271 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:40:01.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:40:01.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:40:01.807 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:40:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:40:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:40:01.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:01.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:40:01.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:40:01.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:01.830+0000 7f44bfa6e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:01.832 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:01.832+0000 7f44bfa6e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:01.833 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:01.833+0000 7f44bfa6e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:02.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:40:02.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:40:02.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:40:02.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:40:02.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:40:02.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:02.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:40:02.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:02.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:40:02.202 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:40:02.349 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:02.349+0000 7f44bfa6e080 -1 Falling back to public interface 2021-01-15T17:40:02.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:40:03.156 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:03.156+0000 7f44bfa6e080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:40:03.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:40:03.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:03.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:40:03.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:03.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:40:03.610 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:40:04.023 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/519847,v1:127.0.0.1:6819/519847] [v2:127.0.0.1:6820/519847,v1:127.0.0.1:6821/519847] exists,up 37a97990-9e24-4411-b7f1-a72d41fe6208 2021-01-15T17:40:04.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:40:04.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:40:04.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:40:04.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:285: TEST_backfill_down_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:40:04.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:40:04.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:40:04.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:40:04.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:40:04.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:40:04.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:40:04.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:40:04.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:40:04.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:40:04.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:40:04.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:40:04.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:40:04.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:40:04.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:40:04.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:40:04.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:40:04.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:40:04.028 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:40:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:40:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:40:04.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:40:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:40:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:40:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:40:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:40:04.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:40:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:40:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:40:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:40:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:40:04.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:40:04.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=57dc1f06-2dce-4b44-99c9-e6a12d40d104 2021-01-15T17:40:04.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 57dc1f06-2dce-4b44-99c9-e6a12d40d104' 2021-01-15T17:40:04.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:40:04.033 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 57dc1f06-2dce-4b44-99c9-e6a12d40d104 2021-01-15T17:40:04.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB00wFgMYJrAhAAhU3RDIL1yKcCouN7Vysq+g== 2021-01-15T17:40:04.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB00wFgMYJrAhAAhU3RDIL1yKcCouN7Vysq+g=="}' 2021-01-15T17:40:04.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 57dc1f06-2dce-4b44-99c9-e6a12d40d104 -i td/osd-backfill-stats/3/new.json 2021-01-15T17:40:04.569 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:40:04.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:40:04.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB00wFgMYJrAhAAhU3RDIL1yKcCouN7Vysq+g== --osd-uuid 57dc1f06-2dce-4b44-99c9-e6a12d40d104 2021-01-15T17:40:04.606 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.606+0000 7f143252a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:04.608 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.608+0000 7f143252a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:04.609 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.609+0000 7f143252a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:04.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.609+0000 7f143252a080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:40:04.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.609+0000 7f143252a080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:40:04.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.609+0000 7f143252a080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:40:04.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:04.650+0000 7f143252a080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:40:09.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:40:09.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:40:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:40:09.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:40:09.206 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:40:09.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:40:09.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:09.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:40:09.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:40:09.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:40:09.732 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:40:09.754 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:09.753+0000 7f7f28c1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:09.779 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:09.778+0000 7f7f28c1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:09.780 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:09.780+0000 7f7f28c1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:10.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:40:10.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:40:10.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:40:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:40:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:40:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:40:10.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:10.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:40:10.143 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:40:10.296 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:10.295+0000 7f7f28c1c080 -1 Falling back to public interface 2021-01-15T17:40:10.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:40:11.079 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:11.079+0000 7f7f28c1c080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:40:11.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:40:11.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:11.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:40:11.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:11.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:40:11.539 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:40:11.969 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/521111,v1:127.0.0.1:6827/521111] [v2:127.0.0.1:6828/521111,v1:127.0.0.1:6829/521111] exists,up 57dc1f06-2dce-4b44-99c9-e6a12d40d104 2021-01-15T17:40:11.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:40:11.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:40:11.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:40:11.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:286: TEST_backfill_down_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:40:11.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:40:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:40:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:40:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:40:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:40:11.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:40:11.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:40:11.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:40:11.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:40:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:40:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:40:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:40:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:40:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:40:11.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:40:11.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:40:11.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:40:11.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:40:11.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:40:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:40:11.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:40:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:40:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:40:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:40:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:40:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:40:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:40:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:40:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:40:11.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:40:11.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:40:11.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=72105524-c275-42cc-9ab7-0252c996637c 2021-01-15T17:40:11.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 72105524-c275-42cc-9ab7-0252c996637c' 2021-01-15T17:40:11.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:40:11.979 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 72105524-c275-42cc-9ab7-0252c996637c 2021-01-15T17:40:11.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB70wFgpm3OOhAANVhC4WUrqk7IRVw7XrfVMQ== 2021-01-15T17:40:11.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB70wFgpm3OOhAANVhC4WUrqk7IRVw7XrfVMQ=="}' 2021-01-15T17:40:11.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 72105524-c275-42cc-9ab7-0252c996637c -i td/osd-backfill-stats/4/new.json 2021-01-15T17:40:12.501 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:40:12.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:40:12.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB70wFgpm3OOhAANVhC4WUrqk7IRVw7XrfVMQ== --osd-uuid 72105524-c275-42cc-9ab7-0252c996637c 2021-01-15T17:40:12.538 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.537+0000 7f879f95e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:12.539 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.539+0000 7f879f95e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:12.541 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.541+0000 7f879f95e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.542+0000 7f879f95e080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:40:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.542+0000 7f879f95e080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:40:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.542+0000 7f879f95e080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:40:12.595 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:12.595+0000 7f879f95e080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:40:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:40:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:40:17.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:40:17.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:40:17.349 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:40:17.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:40:17.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:40:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:40:17.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:40:17.869 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:40:17.892 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:17.891+0000 7f3762e64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:17.913 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:17.912+0000 7f3762e64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:17.914 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:17.913+0000 7f3762e64080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:18.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:40:18.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:40:18.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:40:18.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:40:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:40:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:40:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:40:18.268 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:40:18.438 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:18.438+0000 7f3762e64080 -1 Falling back to public interface 2021-01-15T17:40:18.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:40:19.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:19.240+0000 7f3762e64080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:40:19.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:40:19.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:19.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:40:19.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:19.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:40:19.667 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:40:20.073 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/522375,v1:127.0.0.1:6835/522375] [v2:127.0.0.1:6836/522375,v1:127.0.0.1:6837/522375] exists,up 72105524-c275-42cc-9ab7-0252c996637c 2021-01-15T17:40:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:40:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:40:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:40:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:287: TEST_backfill_down_out: run_osd td/osd-backfill-stats 5 2021-01-15T17:40:20.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:40:20.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:40:20.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:40:20.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:40:20.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:40:20.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:40:20.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:40:20.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:40:20.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:40:20.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:40:20.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:40:20.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:40:20.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:40:20.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:40:20.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:40:20.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:40:20.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:40:20.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:40:20.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:40:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:40:20.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:40:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:40:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:40:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:40:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:40:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:40:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:40:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:40:20.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:20.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:40:20.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:40:20.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:40:20.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:40:20.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=af1cc91b-de2e-40b8-b649-478d954aec61 2021-01-15T17:40:20.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 af1cc91b-de2e-40b8-b649-478d954aec61' 2021-01-15T17:40:20.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:40:20.084 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 af1cc91b-de2e-40b8-b649-478d954aec61 2021-01-15T17:40:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCE0wFgjelkBRAA78gBNZNhjVbNrBxAjrUzhw== 2021-01-15T17:40:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCE0wFgjelkBRAA78gBNZNhjVbNrBxAjrUzhw=="}' 2021-01-15T17:40:20.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new af1cc91b-de2e-40b8-b649-478d954aec61 -i td/osd-backfill-stats/5/new.json 2021-01-15T17:40:20.679 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:40:20.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:40:20.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCE0wFgjelkBRAA78gBNZNhjVbNrBxAjrUzhw== --osd-uuid af1cc91b-de2e-40b8-b649-478d954aec61 2021-01-15T17:40:20.716 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.716+0000 7f89225f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:20.718 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.718+0000 7f89225f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:20.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.719+0000 7f89225f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:20.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.720+0000 7f89225f9080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:40:20.721 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.720+0000 7f89225f9080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:40:20.721 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.720+0000 7f89225f9080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:40:20.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:20.763+0000 7f89225f9080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:40:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:40:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:40:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:40:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:40:25.299 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:40:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:40:25.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:40:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:40:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:40:25.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:40:25.825 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:40:25.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:25.844+0000 7f5b6504c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:25.864 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:25.863+0000 7f5b6504c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:25.865 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:25.865+0000 7f5b6504c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:40:26.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:40:26.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:40:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:40:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:40:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:40:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:40:26.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:26.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:40:26.221 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:40:26.390 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:26.388+0000 7f5b6504c080 -1 Falling back to public interface 2021-01-15T17:40:26.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:40:27.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:40:27.191+0000 7f5b6504c080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:40:27.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:40:27.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:40:27.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:40:27.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:40:27.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:40:27.629 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:40:28.053 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/523639,v1:127.0.0.1:6843/523639] [v2:127.0.0.1:6844/523639,v1:127.0.0.1:6845/523639] exists,up af1cc91b-de2e-40b8-b649-478d954aec61 2021-01-15T17:40:28.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:40:28.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:40:28.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:40:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:289: TEST_backfill_down_out: create_pool test 1 1 2021-01-15T17:40:28.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T17:40:28.656 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:40:28.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:40:29.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:290: TEST_backfill_down_out: ceph osd pool set test size 2 2021-01-15T17:40:30.434 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:40:30.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:291: TEST_backfill_down_out: sleep 5 2021-01-15T17:40:35.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:293: TEST_backfill_down_out: wait_for_clean 2021-01-15T17:40:35.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:40:35.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:40:35.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:40:35.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:40:35.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:40:35.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:40:35.448 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:40:35.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:40:35.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:40:35.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:40:35.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:40:35.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:40:35.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:40:35.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:40:35.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:40:35.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:40:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:40:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:40:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:40:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:40:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:40:35.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:40:35.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:35.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:40:36.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2021-01-15T17:40:36.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2021-01-15T17:40:36.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2021-01-15T17:40:36.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:36.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:40:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T17:40:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T17:40:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2021-01-15T17:40:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:36.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:40:36.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2021-01-15T17:40:36.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2021-01-15T17:40:36.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449' 2021-01-15T17:40:36.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:36.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:40:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:40:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:40:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345927' 2021-01-15T17:40:36.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:36.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:40:36.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T17:40:36.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T17:40:36.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345927 4-107374182406' 2021-01-15T17:40:36.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:40:36.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:40:36.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018885 2021-01-15T17:40:36.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018885 2021-01-15T17:40:36.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509449 3-85899345927 4-107374182406 5-128849018885' 2021-01-15T17:40:36.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:36.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2021-01-15T17:40:36.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:36.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:40:36.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2021-01-15T17:40:36.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:36.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2021-01-15T17:40:36.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-01-15T17:40:36.779 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836492 2021-01-15T17:40:36.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:40:37.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2021-01-15T17:40:37.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:37.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T17:40:37.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:37.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:40:37.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T17:40:37.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:37.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T17:40:37.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T17:40:37.172 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T17:40:37.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:40:37.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T17:40:37.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:37.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2021-01-15T17:40:37.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:37.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:40:37.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-01-15T17:40:37.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:37.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2021-01-15T17:40:37.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-01-15T17:40:37.573 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2021-01-15T17:40:37.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:40:37.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509449 -lt 64424509449 2021-01-15T17:40:37.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:37.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:40:37.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:37.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:40:37.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:40:37.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:37.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:40:37.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:40:37.970 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:40:37.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:40:38.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345925 -lt 85899345927 2021-01-15T17:40:38.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:40:39.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:40:39.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:40:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2021-01-15T17:40:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:39.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182406 2021-01-15T17:40:39.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:39.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:40:39.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182406 2021-01-15T17:40:39.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:39.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T17:40:39.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-01-15T17:40:39.779 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182406 2021-01-15T17:40:39.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:40:40.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2021-01-15T17:40:40.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:40:40.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018885 2021-01-15T17:40:40.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:40:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:40:40.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018885 2021-01-15T17:40:40.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:40:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018885 2021-01-15T17:40:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2021-01-15T17:40:40.177 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018885 2021-01-15T17:40:40.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:40:40.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018886 -lt 128849018885 2021-01-15T17:40:40.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:40:40.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:40:40.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:40:41.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:40:41.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:40:41.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:40:41.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:40:41.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:40:41.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:40:41.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:40:41.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:40:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:40:41.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:40:41.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:40:41.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:40:42.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:40:42.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:40:42.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:40:42.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: seq 1 200 2021-01-15T17:40:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj1 /dev/null 2021-01-15T17:40:42.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj2 /dev/null 2021-01-15T17:40:42.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj3 /dev/null 2021-01-15T17:40:42.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj4 /dev/null 2021-01-15T17:40:42.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj5 /dev/null 2021-01-15T17:40:42.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj6 /dev/null 2021-01-15T17:40:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj7 /dev/null 2021-01-15T17:40:42.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj8 /dev/null 2021-01-15T17:40:42.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj9 /dev/null 2021-01-15T17:40:42.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:42.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj10 /dev/null 2021-01-15T17:40:43.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj11 /dev/null 2021-01-15T17:40:43.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj12 /dev/null 2021-01-15T17:40:43.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj13 /dev/null 2021-01-15T17:40:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj14 /dev/null 2021-01-15T17:40:43.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj15 /dev/null 2021-01-15T17:40:43.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj16 /dev/null 2021-01-15T17:40:43.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj17 /dev/null 2021-01-15T17:40:43.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj18 /dev/null 2021-01-15T17:40:43.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj19 /dev/null 2021-01-15T17:40:43.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:43.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj20 /dev/null 2021-01-15T17:40:44.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj21 /dev/null 2021-01-15T17:40:44.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj22 /dev/null 2021-01-15T17:40:44.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj23 /dev/null 2021-01-15T17:40:44.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj24 /dev/null 2021-01-15T17:40:44.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj25 /dev/null 2021-01-15T17:40:44.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj26 /dev/null 2021-01-15T17:40:44.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj27 /dev/null 2021-01-15T17:40:44.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj28 /dev/null 2021-01-15T17:40:44.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj29 /dev/null 2021-01-15T17:40:45.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj30 /dev/null 2021-01-15T17:40:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj31 /dev/null 2021-01-15T17:40:45.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj32 /dev/null 2021-01-15T17:40:45.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj33 /dev/null 2021-01-15T17:40:45.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj34 /dev/null 2021-01-15T17:40:45.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj35 /dev/null 2021-01-15T17:40:45.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj36 /dev/null 2021-01-15T17:40:45.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj37 /dev/null 2021-01-15T17:40:45.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj38 /dev/null 2021-01-15T17:40:45.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj39 /dev/null 2021-01-15T17:40:45.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:45.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj40 /dev/null 2021-01-15T17:40:46.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj41 /dev/null 2021-01-15T17:40:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj42 /dev/null 2021-01-15T17:40:46.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj43 /dev/null 2021-01-15T17:40:46.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj44 /dev/null 2021-01-15T17:40:46.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj45 /dev/null 2021-01-15T17:40:46.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj46 /dev/null 2021-01-15T17:40:46.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj47 /dev/null 2021-01-15T17:40:46.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj48 /dev/null 2021-01-15T17:40:46.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj49 /dev/null 2021-01-15T17:40:46.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:46.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj50 /dev/null 2021-01-15T17:40:47.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj51 /dev/null 2021-01-15T17:40:47.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj52 /dev/null 2021-01-15T17:40:47.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj53 /dev/null 2021-01-15T17:40:47.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj54 /dev/null 2021-01-15T17:40:47.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj55 /dev/null 2021-01-15T17:40:47.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj56 /dev/null 2021-01-15T17:40:47.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj57 /dev/null 2021-01-15T17:40:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj58 /dev/null 2021-01-15T17:40:47.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj59 /dev/null 2021-01-15T17:40:47.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj60 /dev/null 2021-01-15T17:40:47.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:47.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj61 /dev/null 2021-01-15T17:40:48.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj62 /dev/null 2021-01-15T17:40:48.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj63 /dev/null 2021-01-15T17:40:48.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj64 /dev/null 2021-01-15T17:40:48.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj65 /dev/null 2021-01-15T17:40:48.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj66 /dev/null 2021-01-15T17:40:48.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj67 /dev/null 2021-01-15T17:40:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj68 /dev/null 2021-01-15T17:40:48.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj69 /dev/null 2021-01-15T17:40:48.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj70 /dev/null 2021-01-15T17:40:48.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:48.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj71 /dev/null 2021-01-15T17:40:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj72 /dev/null 2021-01-15T17:40:49.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj73 /dev/null 2021-01-15T17:40:49.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj74 /dev/null 2021-01-15T17:40:49.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj75 /dev/null 2021-01-15T17:40:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj76 /dev/null 2021-01-15T17:40:49.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj77 /dev/null 2021-01-15T17:40:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj78 /dev/null 2021-01-15T17:40:49.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj79 /dev/null 2021-01-15T17:40:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj80 /dev/null 2021-01-15T17:40:49.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:49.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj81 /dev/null 2021-01-15T17:40:50.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj82 /dev/null 2021-01-15T17:40:50.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj83 /dev/null 2021-01-15T17:40:50.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj84 /dev/null 2021-01-15T17:40:50.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj85 /dev/null 2021-01-15T17:40:50.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj86 /dev/null 2021-01-15T17:40:50.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj87 /dev/null 2021-01-15T17:40:50.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj88 /dev/null 2021-01-15T17:40:50.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj89 /dev/null 2021-01-15T17:40:50.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj90 /dev/null 2021-01-15T17:40:50.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:50.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj91 /dev/null 2021-01-15T17:40:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj92 /dev/null 2021-01-15T17:40:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj93 /dev/null 2021-01-15T17:40:51.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj94 /dev/null 2021-01-15T17:40:51.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj95 /dev/null 2021-01-15T17:40:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj96 /dev/null 2021-01-15T17:40:51.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj97 /dev/null 2021-01-15T17:40:51.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj98 /dev/null 2021-01-15T17:40:51.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj99 /dev/null 2021-01-15T17:40:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj100 /dev/null 2021-01-15T17:40:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj101 /dev/null 2021-01-15T17:40:52.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj102 /dev/null 2021-01-15T17:40:52.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj103 /dev/null 2021-01-15T17:40:52.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj104 /dev/null 2021-01-15T17:40:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj105 /dev/null 2021-01-15T17:40:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj106 /dev/null 2021-01-15T17:40:52.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj107 /dev/null 2021-01-15T17:40:52.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj108 /dev/null 2021-01-15T17:40:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj109 /dev/null 2021-01-15T17:40:52.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj110 /dev/null 2021-01-15T17:40:52.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:52.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj111 /dev/null 2021-01-15T17:40:53.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj112 /dev/null 2021-01-15T17:40:53.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj113 /dev/null 2021-01-15T17:40:53.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj114 /dev/null 2021-01-15T17:40:53.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj115 /dev/null 2021-01-15T17:40:53.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj116 /dev/null 2021-01-15T17:40:53.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj117 /dev/null 2021-01-15T17:40:53.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj118 /dev/null 2021-01-15T17:40:53.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj119 /dev/null 2021-01-15T17:40:53.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj120 /dev/null 2021-01-15T17:40:53.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj121 /dev/null 2021-01-15T17:40:54.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj122 /dev/null 2021-01-15T17:40:54.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj123 /dev/null 2021-01-15T17:40:54.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj124 /dev/null 2021-01-15T17:40:54.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj125 /dev/null 2021-01-15T17:40:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj126 /dev/null 2021-01-15T17:40:54.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj127 /dev/null 2021-01-15T17:40:54.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj128 /dev/null 2021-01-15T17:40:54.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj129 /dev/null 2021-01-15T17:40:54.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj130 /dev/null 2021-01-15T17:40:54.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:54.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj131 /dev/null 2021-01-15T17:40:55.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj132 /dev/null 2021-01-15T17:40:55.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj133 /dev/null 2021-01-15T17:40:55.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj134 /dev/null 2021-01-15T17:40:55.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj135 /dev/null 2021-01-15T17:40:55.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj136 /dev/null 2021-01-15T17:40:55.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj137 /dev/null 2021-01-15T17:40:55.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj138 /dev/null 2021-01-15T17:40:55.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj139 /dev/null 2021-01-15T17:40:55.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj140 /dev/null 2021-01-15T17:40:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:55.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj141 /dev/null 2021-01-15T17:40:56.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj142 /dev/null 2021-01-15T17:40:56.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj143 /dev/null 2021-01-15T17:40:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj144 /dev/null 2021-01-15T17:40:56.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj145 /dev/null 2021-01-15T17:40:56.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj146 /dev/null 2021-01-15T17:40:56.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj147 /dev/null 2021-01-15T17:40:56.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj148 /dev/null 2021-01-15T17:40:56.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj149 /dev/null 2021-01-15T17:40:56.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj150 /dev/null 2021-01-15T17:40:56.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:56.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj151 /dev/null 2021-01-15T17:40:57.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj152 /dev/null 2021-01-15T17:40:57.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj153 /dev/null 2021-01-15T17:40:57.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj154 /dev/null 2021-01-15T17:40:57.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj155 /dev/null 2021-01-15T17:40:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj156 /dev/null 2021-01-15T17:40:57.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj157 /dev/null 2021-01-15T17:40:57.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:57.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj158 /dev/null 2021-01-15T17:40:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj159 /dev/null 2021-01-15T17:40:58.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj160 /dev/null 2021-01-15T17:40:58.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj161 /dev/null 2021-01-15T17:40:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj162 /dev/null 2021-01-15T17:40:58.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj163 /dev/null 2021-01-15T17:40:58.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj164 /dev/null 2021-01-15T17:40:58.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj165 /dev/null 2021-01-15T17:40:58.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj166 /dev/null 2021-01-15T17:40:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj167 /dev/null 2021-01-15T17:40:58.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj168 /dev/null 2021-01-15T17:40:58.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj169 /dev/null 2021-01-15T17:40:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj170 /dev/null 2021-01-15T17:40:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj171 /dev/null 2021-01-15T17:40:59.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj172 /dev/null 2021-01-15T17:40:59.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj173 /dev/null 2021-01-15T17:40:59.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj174 /dev/null 2021-01-15T17:40:59.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj175 /dev/null 2021-01-15T17:40:59.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj176 /dev/null 2021-01-15T17:40:59.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj177 /dev/null 2021-01-15T17:40:59.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj178 /dev/null 2021-01-15T17:40:59.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:40:59.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj179 /dev/null 2021-01-15T17:41:00.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj180 /dev/null 2021-01-15T17:41:00.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj181 /dev/null 2021-01-15T17:41:00.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj182 /dev/null 2021-01-15T17:41:00.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj183 /dev/null 2021-01-15T17:41:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj184 /dev/null 2021-01-15T17:41:00.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj185 /dev/null 2021-01-15T17:41:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj186 /dev/null 2021-01-15T17:41:00.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj187 /dev/null 2021-01-15T17:41:00.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj188 /dev/null 2021-01-15T17:41:00.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:00.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj189 /dev/null 2021-01-15T17:41:01.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj190 /dev/null 2021-01-15T17:41:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj191 /dev/null 2021-01-15T17:41:01.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj192 /dev/null 2021-01-15T17:41:01.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj193 /dev/null 2021-01-15T17:41:01.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj194 /dev/null 2021-01-15T17:41:01.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj195 /dev/null 2021-01-15T17:41:01.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj196 /dev/null 2021-01-15T17:41:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj197 /dev/null 2021-01-15T17:41:01.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:01.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj198 /dev/null 2021-01-15T17:41:02.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:02.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj199 /dev/null 2021-01-15T17:41:02.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:295: TEST_backfill_down_out: for i in $(seq 1 $objects) 2021-01-15T17:41:02.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:297: TEST_backfill_down_out: rados -p test put obj200 /dev/null 2021-01-15T17:41:02.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:300: TEST_backfill_down_out: get_pg test obj1 2021-01-15T17:41:02.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:41:02.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:41:02.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:41:02.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:41:02.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:300: TEST_backfill_down_out: local PG=1.0 2021-01-15T17:41:02.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:302: TEST_backfill_down_out: get_primary test obj1 2021-01-15T17:41:02.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:41:02.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:41:02.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:41:02.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:41:02.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:302: TEST_backfill_down_out: local primary=1 2021-01-15T17:41:02.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:303: TEST_backfill_down_out: get_not_primary test obj1 2021-01-15T17:41:02.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:41:02.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:41:02.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:41:02.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:41:02.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:41:02.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:41:02.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:41:03.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:41:03.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:41:03.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:41:03.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:303: TEST_backfill_down_out: local otherosd=0 2021-01-15T17:41:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:305: TEST_backfill_down_out: ceph osd set nobackfill 2021-01-15T17:41:04.641 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:41:04.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:306: TEST_backfill_down_out: cat td/osd-backfill-stats/osd.0.pid 2021-01-15T17:41:04.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:306: TEST_backfill_down_out: kill 517306 2021-01-15T17:41:04.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:307: TEST_backfill_down_out: ceph osd down osd.0 2021-01-15T17:41:05.152 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T17:41:05.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:308: TEST_backfill_down_out: ceph osd out osd.0 2021-01-15T17:41:05.727 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:41:05.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:309: TEST_backfill_down_out: sleep 2 2021-01-15T17:41:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:310: TEST_backfill_down_out: ceph osd unset nobackfill 2021-01-15T17:41:08.445 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:41:08.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:312: TEST_backfill_down_out: wait_for_clean 2021-01-15T17:41:08.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:41:08.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:41:08.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:41:08.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:41:08.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:41:08.459 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:41:08.460 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:41:08.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:41:08.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:41:08.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:41:08.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:41:08.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:41:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:41:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:41:08.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:41:08.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:41:08.942 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:41:08.942 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:41:08.942 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:41:08.942 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:41:08.943 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:41:08.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:41:08.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:08.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:41:09.040 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T17:41:09.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:41:09.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:41:09.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:41:09.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:09.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:41:09.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672980 2021-01-15T17:41:09.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672980 2021-01-15T17:41:09.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980' 2021-01-15T17:41:09.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:09.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:41:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T17:41:09.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T17:41:09.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457' 2021-01-15T17:41:09.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:09.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:41:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2021-01-15T17:41:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2021-01-15T17:41:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457 3-85899345936' 2021-01-15T17:41:09.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:09.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:41:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T17:41:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T17:41:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457 3-85899345936 4-107374182414' 2021-01-15T17:41:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:41:09.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:41:09.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018894 2021-01-15T17:41:09.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018894 2021-01-15T17:41:09.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457 3-85899345936 4-107374182414 5-128849018894' 2021-01-15T17:41:09.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:41:09.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672980 2021-01-15T17:41:09.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:41:09.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:41:09.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672980 2021-01-15T17:41:09.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:41:09.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672980 2021-01-15T17:41:09.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-01-15T17:41:09.738 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672980 2021-01-15T17:41:09.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:41:10.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672980 2021-01-15T17:41:10.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:41:11.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:41:11.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:41:11.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672980 2021-01-15T17:41:11.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:41:11.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T17:41:11.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:41:11.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:41:11.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T17:41:11.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:41:11.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T17:41:11.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T17:41:11.551 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T17:41:11.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:41:11.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509458 -lt 64424509457 2021-01-15T17:41:11.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:41:11.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2021-01-15T17:41:11.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:41:11.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:41:11.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2021-01-15T17:41:11.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:41:11.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2021-01-15T17:41:11.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-01-15T17:41:11.947 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345936 2021-01-15T17:41:11.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:41:12.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345936 2021-01-15T17:41:12.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:41:12.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T17:41:12.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:41:12.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:41:12.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T17:41:12.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:41:12.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T17:41:12.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T17:41:12.364 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T17:41:12.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:41:12.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182415 -lt 107374182414 2021-01-15T17:41:12.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:41:12.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018894 2021-01-15T17:41:12.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:41:12.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018894 2021-01-15T17:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:41:12.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018894 2021-01-15T17:41:12.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2021-01-15T17:41:12.790 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018894 2021-01-15T17:41:12.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:41:13.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018894 2021-01-15T17:41:13.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:41:13.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:13.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:13.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:41:13.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:13.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:13.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:13.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:13.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:13.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:13.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:14.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:14.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:14.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:14.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:14.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:14.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:41:14.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:41:14.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:41:14.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:14.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:14.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:14.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:14.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:14.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:14.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:14.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:14.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:14.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:15.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:15.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:15.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:16.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:41:16.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:41:16.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:16.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:16.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:16.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:16.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:16.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:16.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:16.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:16.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:16.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:16.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:16.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:16.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:16.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:16.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:17.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:17.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:17.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:17.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:17.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:17.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:17.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:17.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:17.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:17.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:41:17.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:41:17.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:17.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:17.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:17.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:17.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:17.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:17.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:17.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:17.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:17.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:18.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:18.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:18.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:18.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:18.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:18.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:18.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:18.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:18.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:18.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:18.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:18.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:18.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:19.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:19.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:19.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:19.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:19.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:19.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:19.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:19.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:19.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:19.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:19.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:19.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:19.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:19.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:20.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:20.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:20.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:20.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:20.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:20.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:41:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:41:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:20.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:20.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:20.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:20.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:20.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:20.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:20.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:20.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:21.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:21.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:21.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:21.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:21.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:21.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:21.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:22.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:41:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:41:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:22.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:22.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:22.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:22.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:22.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:22.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:22.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:22.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:22.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:22.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:22.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:22.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:22.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:23.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:23.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:23.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:23.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:23.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:23.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:23.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:23.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:23.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2021-01-15T17:41:23.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2021-01-15T17:41:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:23.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:23.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:23.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:23.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:23.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:23.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:23.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:23.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:24.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:24.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:24.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:24.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:24.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:24.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:24.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:24.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:24.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:25.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:25.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:25.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:25.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:25.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:25.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:25.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:25.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:25.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:25.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:25.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:25.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:25.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:25.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:25.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:25.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:25.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:26.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:26.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:26.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:26.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:26.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:26.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:26.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:26.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:26.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:26.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:27.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:27.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:27.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:27.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:27.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:27.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:27.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:27.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:27.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:27.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:27.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:27.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:27.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:27.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:28.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:28.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:28.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:28.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:28.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:28.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:28.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:28.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:28.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:28.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:28.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:28.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:28.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:28.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:28.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:28.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:29.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:29.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:29.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:29.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:29.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:29.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:29.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:29.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:29.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:29.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:41:29.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:41:29.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:29.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:29.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:30.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:30.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:30.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:30.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:30.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:30.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:30.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:30.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:30.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:30.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:30.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:31.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:31.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:31.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:31.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:31.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:31.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:31.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:31.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:31.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:31.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:31.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:31.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:32.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:32.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:32.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:32.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:32.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:32.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:32.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:32.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:32.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:41:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:41:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:33.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:33.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:33.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:33.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:33.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:33.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:33.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:33.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:33.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:33.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:33.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:33.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:34.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:34.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:34.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:34.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:34.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:34.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:34.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:34.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:34.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:34.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:34.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:34.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:34.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:34.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:34.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:34.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:34.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:34.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:35.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:35.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:35.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:35.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:35.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:35.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:35.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:35.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:35.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:35.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:36.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:36.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:36.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:36.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:36.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:36.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:36.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:36.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:36.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:36.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:36.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:36.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:37.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:37.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:37.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:37.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:37.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:37.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:37.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:41:37.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:41:37.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:37.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:37.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:37.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:37.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:37.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:37.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:37.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:37.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:37.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:37.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:38.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:38.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:38.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:38.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:38.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:38.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:38.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:38.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:38.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:38.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:38.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:39.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:41:39.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:41:39.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:39.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:39.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:39.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:39.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:39.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:39.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:39.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:40.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:40.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:40.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:40.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:40.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:41:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:41:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:40.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:40.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:40.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:40.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:40.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:40.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:40.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:40.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:40.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:40.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:41.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:41:41.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:41.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:41.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:41.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:41:41.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:41:41.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:41:41.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:41:41.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:41:41.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:41:41.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:41:41.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:41:41.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:41:42.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:41:42.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:41:42.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:41:42.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:41:42.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:41:42.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:41:42.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:41:42.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:41:42.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:41:42.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:41:42.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:41:42.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:41:42.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:41:42.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:41:42.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:41:42.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:41:42.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:41:43.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:41:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:41:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:41:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:314: TEST_backfill_down_out: check td/osd-backfill-stats 1.0 1 replicated 200 0 0 0 2021-01-15T17:41:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:41:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:41:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:41:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T17:41:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2021-01-15T17:41:43.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:41:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2021-01-15T17:41:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:41:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:41:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:41:43.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:41:43.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:41:43.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T17:41:43.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:41:43.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:41:43.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:41:43.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T17:41:43.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:41:43.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:41:43.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:41:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,2]/[1]' 2021-01-15T17:41:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,2]/[1]' ']' 2021-01-15T17:41:43.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:41:43.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:41:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,2]/[1] ' 2021-01-15T17:41:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:41:43.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:41:43.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:41:43.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2021-01-15T17:41:43.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2021-01-15T17:41:43.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2021-01-15T17:41:43.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:41:43.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:41:43.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:41:43.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:41:43.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:41:43.350 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:41:43.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:41:43.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2021-01-15T17:41:43.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2021-01-15T17:41:43.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2021-01-15T17:41:43.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:41:43.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:41:43.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:41:43.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:41:43.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:41:43.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:41:43.359 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,2]/[1] ' 2021-01-15T17:41:43.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:41:43.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:41:43.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:41:43.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2021-01-15T17:41:43.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2021-01-15T17:41:43.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2021-01-15T17:41:43.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:41:43.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:41:43.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:41:43.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:41:43.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:41:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:41:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:41:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:41:43.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2021-01-15T17:41:43.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2021-01-15T17:41:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2021-01-15T17:41:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:41:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:41:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:41:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:41:43.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:41:43.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:316: TEST_backfill_down_out: delete_pool test 2021-01-15T17:41:43.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:41:43.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:41:43.928 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:41:43.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:317: TEST_backfill_down_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:41:43.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:41:43.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:41:43.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:41:43.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:41:43.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:41:44.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:41:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:41:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:41:44.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:41:44.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:41:44.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:41:44.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:41:44.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:41:44.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:41:44.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:41:44.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:41:44.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:41:44.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:41:44.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:41:44.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:41:44.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:41:44.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:41:44.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:41:44.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:41:44.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:41:44.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:41:44.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:41:44.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:41:44.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:41:44.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:41:44.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:41:44.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:41:44.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:41:44.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:41:44.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:41:44.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:41:44.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:41:44.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:41:44.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:41:44.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:41:44.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:41:44.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:41:44.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:41:44.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:41:44.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:41:44.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:41:44.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:41:44.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:41:44.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:41:44.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:41:44.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:41:44.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:41:44.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:41:44.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:41:44.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:41:44.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:41:44.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:41:44.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:41:44.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:41:44.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:41:44.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:41:44.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:41:44.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:41:44.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:41:44.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:41:44.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:41:44.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:41:44.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_all_out td/osd-backfill-stats 2021-01-15T17:41:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:515: TEST_backfill_ec_all_out: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:517: TEST_backfill_ec_all_out: run_mon td/osd-backfill-stats a 2021-01-15T17:41:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:41:44.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:41:44.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:41:44.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:41:44.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:41:44.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:41:44.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:41:44.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:41:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:41:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:44.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:41:44.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:41:44.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:41:44.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:41:44.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:41:44.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:41:44.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:41:44.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:41:44.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:41:44.753 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:41:44.754 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:41:44.754 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.754 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:41:44.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:41:44.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:41:44.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:41:44.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:41:44.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:41:44.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:41:44.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:41:44.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:41:44.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:41:44.857 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:41:44.857 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:41:44.857 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:44.857 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:44.857 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:41:44.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:41:44.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:41:44.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:518: TEST_backfill_ec_all_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:41:44.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:41:44.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:41:44.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:41:44.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:41:44.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:41:44.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:41:45.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:41:45.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:41:45.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:41:45.453 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:41:45.453 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:45.453 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:45.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:45.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:41:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:41:45.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:519: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:41:45.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:41:45.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:41:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:41:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:41:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:41:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:41:45.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:41:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:41:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:41:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:41:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:41:45.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:41:45.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:41:45.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:41:45.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:41:45.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:41:45.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:41:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:41:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:45.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:41:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:41:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:41:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:41:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:41:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:41:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:41:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:41:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:41:45.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:41:45.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:41:45.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:41:45.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=45d54f06-1683-495f-9e91-36956c03f273 2021-01-15T17:41:45.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 45d54f06-1683-495f-9e91-36956c03f273' 2021-01-15T17:41:45.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:41:45.486 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 45d54f06-1683-495f-9e91-36956c03f273 2021-01-15T17:41:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDZ0wFg1+l8HRAA4mQV7b6nW5xJJvdFuvW+Ug== 2021-01-15T17:41:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDZ0wFg1+l8HRAA4mQV7b6nW5xJJvdFuvW+Ug=="}' 2021-01-15T17:41:45.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 45d54f06-1683-495f-9e91-36956c03f273 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:41:45.835 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:41:45.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:41:45.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZ0wFg1+l8HRAA4mQV7b6nW5xJJvdFuvW+Ug== --osd-uuid 45d54f06-1683-495f-9e91-36956c03f273 2021-01-15T17:41:45.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.872+0000 7fc78ca5a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:45.874 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.874+0000 7fc78ca5a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:45.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.875+0000 7fc78ca5a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:45.876 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.876+0000 7fc78ca5a080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:41:45.877 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.876+0000 7fc78ca5a080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:41:45.877 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.876+0000 7fc78ca5a080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:41:45.916 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:45.916+0000 7fc78ca5a080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:41:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:41:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:41:50.441 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:41:50.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:41:50.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:41:50.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:41:50.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:41:50.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:41:50.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:41:50.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:41:50.965 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:41:50.986 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:50.986+0000 7f3f4ec06080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:51.007+0000 7f3f4ec06080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:51.009+0000 7f3f4ec06080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:51.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:41:51.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:41:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:41:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:41:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:41:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:41:51.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:41:51.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:41:51.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:41:51.369 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:41:51.525 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:51.525+0000 7f3f4ec06080 -1 Falling back to public interface 2021-01-15T17:41:51.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:41:52.325 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:52.325+0000 7f3f4ec06080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:41:52.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:41:52.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:41:52.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:41:52.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:41:52.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:41:52.764 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:41:53.179 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/532101,v1:127.0.0.1:6803/532101] [v2:127.0.0.1:6804/532101,v1:127.0.0.1:6805/532101] exists,up 45d54f06-1683-495f-9e91-36956c03f273 2021-01-15T17:41:53.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:41:53.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:41:53.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:41:53.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:520: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:41:53.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:41:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:41:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:41:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:41:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:41:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:41:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:41:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:41:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:41:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:41:53.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:41:53.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:41:53.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:41:53.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:41:53.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:41:53.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:41:53.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:41:53.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:41:53.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:41:53.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:41:53.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:53.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:41:53.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:41:53.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:41:53.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:41:53.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:41:53.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:41:53.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:41:53.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:41:53.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:41:53.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:41:53.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:41:53.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:41:53.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:41:53.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=84a0c843-4498-407b-b425-df093055b4d5 2021-01-15T17:41:53.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 84a0c843-4498-407b-b425-df093055b4d5' 2021-01-15T17:41:53.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:41:53.189 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 84a0c843-4498-407b-b425-df093055b4d5 2021-01-15T17:41:53.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDh0wFgFSugCxAA5df0uaCJ/LfsnSkr85vRtw== 2021-01-15T17:41:53.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDh0wFgFSugCxAA5df0uaCJ/LfsnSkr85vRtw=="}' 2021-01-15T17:41:53.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 84a0c843-4498-407b-b425-df093055b4d5 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:41:53.683 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:41:53.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:41:53.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDh0wFgFSugCxAA5df0uaCJ/LfsnSkr85vRtw== --osd-uuid 84a0c843-4498-407b-b425-df093055b4d5 2021-01-15T17:41:53.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.720+0000 7f3d55dfd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:53.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.722+0000 7f3d55dfd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:53.724 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.724+0000 7f3d55dfd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.725+0000 7f3d55dfd080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:41:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.725+0000 7f3d55dfd080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:41:53.725 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.725+0000 7f3d55dfd080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:41:53.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:53.772+0000 7f3d55dfd080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:41:58.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:41:58.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:41:58.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:41:58.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:41:58.305 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:41:58.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:41:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:41:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:41:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:41:58.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:41:58.835 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:41:58.857 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:58.856+0000 7fccc17b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:58.878 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:58.878+0000 7fccc17b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:58.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:58.879+0000 7fccc17b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:41:59.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:41:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:41:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:41:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:41:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:41:59.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:41:59.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:41:59.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:41:59.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:41:59.227 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:41:59.396 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:41:59.395+0000 7fccc17b2080 -1 Falling back to public interface 2021-01-15T17:41:59.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:42:00.181 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:00.180+0000 7fccc17b2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:42:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:42:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:42:00.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:00.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:42:00.636 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:42:01.043 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/533365,v1:127.0.0.1:6811/533365] [v2:127.0.0.1:6812/533365,v1:127.0.0.1:6813/533365] exists,up 84a0c843-4498-407b-b425-df093055b4d5 2021-01-15T17:42:01.043 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 07c7e849-6031-4f07-8fca-3b25aeec7a55 2021-01-15T17:42:01.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:42:01.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:42:01.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:42:01.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:521: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:42:01.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:42:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:42:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:42:01.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:42:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:42:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:42:01.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:42:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:42:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:42:01.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:42:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:42:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:42:01.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:42:01.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:42:01.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:42:01.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:42:01.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:42:01.051 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:42:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:42:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:42:01.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:42:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:42:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:42:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:42:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:42:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:42:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:42:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:01.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:42:01.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:42:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:42:01.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:42:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=07c7e849-6031-4f07-8fca-3b25aeec7a55 2021-01-15T17:42:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 07c7e849-6031-4f07-8fca-3b25aeec7a55' 2021-01-15T17:42:01.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:42:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDp0wFgee5RAxAAlW4dbeN5ok/7Q0N3ubmTMw== 2021-01-15T17:42:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDp0wFgee5RAxAAlW4dbeN5ok/7Q0N3ubmTMw=="}' 2021-01-15T17:42:01.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 07c7e849-6031-4f07-8fca-3b25aeec7a55 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:42:01.648 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:42:01.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:42:01.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDp0wFgee5RAxAAlW4dbeN5ok/7Q0N3ubmTMw== --osd-uuid 07c7e849-6031-4f07-8fca-3b25aeec7a55 2021-01-15T17:42:01.685 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.684+0000 7fb5fd3e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:01.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.686+0000 7fb5fd3e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:01.687 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.687+0000 7fb5fd3e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.688+0000 7fb5fd3e0080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:42:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.688+0000 7fb5fd3e0080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:42:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.688+0000 7fb5fd3e0080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:42:01.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:01.728+0000 7fb5fd3e0080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:42:06.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:42:06.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:42:06.230 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:42:06.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:42:06.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:42:06.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:42:06.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:06.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:42:06.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:42:06.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:42:06.757 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:42:06.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:06.778+0000 7f7f21a88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:06.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:06.801+0000 7f7f21a88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:06.803 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:06.803+0000 7f7f21a88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:07.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:42:07.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:42:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:42:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:42:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:42:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:42:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:42:07.149 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:42:07.320 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:07.319+0000 7f7f21a88080 -1 Falling back to public interface 2021-01-15T17:42:07.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:42:08.103 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:08.103+0000 7f7f21a88080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:42:08.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:42:08.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:08.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:42:08.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:08.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:42:08.544 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:42:08.968 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/534629,v1:127.0.0.1:6819/534629] [v2:127.0.0.1:6820/534629,v1:127.0.0.1:6821/534629] exists,up 07c7e849-6031-4f07-8fca-3b25aeec7a55 2021-01-15T17:42:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:42:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:42:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:42:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:522: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:42:08.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:42:08.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:42:08.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:42:08.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:42:08.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:42:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:42:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:42:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:42:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:42:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:42:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:42:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:42:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:42:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:42:08.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:42:08.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:42:08.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:42:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:42:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:42:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:42:08.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:42:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:42:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:42:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:42:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:42:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:42:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:42:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:42:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:42:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:42:08.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:42:08.977 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 09f3f203-4964-4aa4-87a9-83783c292b7b 2021-01-15T17:42:08.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=09f3f203-4964-4aa4-87a9-83783c292b7b 2021-01-15T17:42:08.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 09f3f203-4964-4aa4-87a9-83783c292b7b' 2021-01-15T17:42:08.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:42:08.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDw0wFgPH+uOhAAKbxe0ylF0vo3OJLAy3AFPA== 2021-01-15T17:42:08.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDw0wFgPH+uOhAAKbxe0ylF0vo3OJLAy3AFPA=="}' 2021-01-15T17:42:08.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 09f3f203-4964-4aa4-87a9-83783c292b7b -i td/osd-backfill-stats/3/new.json 2021-01-15T17:42:09.614 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:42:09.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:42:09.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDw0wFgPH+uOhAAKbxe0ylF0vo3OJLAy3AFPA== --osd-uuid 09f3f203-4964-4aa4-87a9-83783c292b7b 2021-01-15T17:42:09.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.652+0000 7f85ea464080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:09.655 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.654+0000 7f85ea464080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:09.655 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.655+0000 7f85ea464080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:09.656 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.655+0000 7f85ea464080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:42:09.656 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.655+0000 7f85ea464080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:42:09.656 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.655+0000 7f85ea464080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:42:09.692 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:09.693+0000 7f85ea464080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:42:14.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:42:14.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:42:14.504 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:42:14.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:42:14.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:42:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:42:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:42:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:42:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:42:15.038 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:42:15.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:15.059+0000 7fcde3c22080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:15.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:15.061+0000 7fcde3c22080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:15.063 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:15.062+0000 7fcde3c22080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:15.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:42:15.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:42:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:42:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:42:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:42:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:15.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:42:15.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:15.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:42:15.431 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:42:15.583 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:15.582+0000 7fcde3c22080 -1 Falling back to public interface 2021-01-15T17:42:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:42:16.403 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:16.402+0000 7fcde3c22080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:42:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:42:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:42:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:16.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:42:16.835 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:42:17.250 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/535893,v1:127.0.0.1:6827/535893] [v2:127.0.0.1:6828/535893,v1:127.0.0.1:6829/535893] exists,up 09f3f203-4964-4aa4-87a9-83783c292b7b 2021-01-15T17:42:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:42:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:42:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:42:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:523: TEST_backfill_ec_all_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:42:17.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:42:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:42:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:42:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:42:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:42:17.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:42:17.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:42:17.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:42:17.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:42:17.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:42:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:42:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:42:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:42:17.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:42:17.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:42:17.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:42:17.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:42:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:42:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:42:17.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:42:17.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:17.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:42:17.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:42:17.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:42:17.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:42:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:42:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:42:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:42:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:42:17.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:17.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:42:17.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:42:17.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:42:17.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:42:17.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7593f747-2e34-4e4d-ac7c-4358ee06a727 2021-01-15T17:42:17.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 7593f747-2e34-4e4d-ac7c-4358ee06a727' 2021-01-15T17:42:17.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:42:17.260 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 7593f747-2e34-4e4d-ac7c-4358ee06a727 2021-01-15T17:42:17.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD50wFglaHiDxAA29pL4QsTrI+7taOy8AF6ng== 2021-01-15T17:42:17.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD50wFglaHiDxAA29pL4QsTrI+7taOy8AF6ng=="}' 2021-01-15T17:42:17.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7593f747-2e34-4e4d-ac7c-4358ee06a727 -i td/osd-backfill-stats/4/new.json 2021-01-15T17:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:42:17.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:42:17.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD50wFglaHiDxAA29pL4QsTrI+7taOy8AF6ng== --osd-uuid 7593f747-2e34-4e4d-ac7c-4358ee06a727 2021-01-15T17:42:17.787 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.787+0000 7fb22a68d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:17.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.789+0000 7fb22a68d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:17.791 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.790+0000 7fb22a68d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:17.792 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.791+0000 7fb22a68d080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:42:17.792 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.791+0000 7fb22a68d080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:42:17.792 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.791+0000 7fb22a68d080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:42:17.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:17.851+0000 7fb22a68d080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:42:22.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:42:22.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:42:22.604 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:42:22.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:42:22.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:42:23.126 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:42:23.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:42:23.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:42:23.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:42:23.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:42:23.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:42:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:23.149+0000 7fc9a126b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:23.168 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:23.167+0000 7fc9a126b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:23.170 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:23.170+0000 7fc9a126b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:42:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:42:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:42:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:42:23.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:42:23.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:42:23.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:42:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:42:23.524 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:42:23.684 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:23.684+0000 7fc9a126b080 -1 Falling back to public interface 2021-01-15T17:42:23.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:42:24.494 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:42:24.494+0000 7fc9a126b080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:42:24.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:42:24.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:42:24.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:42:24.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:42:24.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:42:24.919 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:42:25.350 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/537157,v1:127.0.0.1:6835/537157] [v2:127.0.0.1:6836/537157,v1:127.0.0.1:6837/537157] exists,up 7593f747-2e34-4e4d-ac7c-4358ee06a727 2021-01-15T17:42:25.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:42:25.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:42:25.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:42:25.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:525: TEST_backfill_ec_all_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T17:42:25.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:526: TEST_backfill_ec_all_out: create_pool test 1 1 erasure myprofile 2021-01-15T17:42:25.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T17:42:26.670 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:42:26.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:42:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:528: TEST_backfill_ec_all_out: wait_for_clean 2021-01-15T17:42:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:42:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:42:27.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:42:27.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:42:27.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:42:27.683 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:42:27.683 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:42:27.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:42:27.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:42:27.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:42:27.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:42:27.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:42:27.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:42:27.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:42:27.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:42:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:42:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:42:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:42:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:42:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:42:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:42:28.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:42:28.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:42:28.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2021-01-15T17:42:28.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2021-01-15T17:42:28.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2021-01-15T17:42:28.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:42:28.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:42:28.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T17:42:28.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T17:42:28.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2021-01-15T17:42:28.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:42:28.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:42:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T17:42:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T17:42:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2021-01-15T17:42:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:42:28.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:42:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345925 2021-01-15T17:42:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345925 2021-01-15T17:42:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446 3-85899345925' 2021-01-15T17:42:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:42:28.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:42:28.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182403 2021-01-15T17:42:28.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182403 2021-01-15T17:42:28.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446 3-85899345925 4-107374182403' 2021-01-15T17:42:28.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:42:28.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2021-01-15T17:42:28.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:42:28.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:42:28.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2021-01-15T17:42:28.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:42:28.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2021-01-15T17:42:28.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-01-15T17:42:28.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:42:28.948 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2021-01-15T17:42:29.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836489 2021-01-15T17:42:29.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:42:30.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:42:30.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:42:30.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2021-01-15T17:42:30.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:42:30.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T17:42:30.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:42:30.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:42:30.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T17:42:30.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:42:30.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T17:42:30.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T17:42:30.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:42:30.756 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T17:42:31.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T17:42:31.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:42:31.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T17:42:31.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:42:31.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:42:31.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T17:42:31.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:42:31.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T17:42:31.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T17:42:31.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:42:31.152 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T17:42:31.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509447 -lt 64424509446 2021-01-15T17:42:31.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:42:31.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345925 2021-01-15T17:42:31.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:42:31.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:42:31.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345925 2021-01-15T17:42:31.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:42:31.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345925 2021-01-15T17:42:31.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2021-01-15T17:42:31.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:42:31.563 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345925 2021-01-15T17:42:31.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345925 -lt 85899345925 2021-01-15T17:42:31.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:42:31.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182403 2021-01-15T17:42:31.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:42:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:42:31.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182403 2021-01-15T17:42:31.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:42:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182403 2021-01-15T17:42:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182403' 2021-01-15T17:42:31.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:42:31.952 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182403 2021-01-15T17:42:32.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182404 -lt 107374182403 2021-01-15T17:42:32.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:42:32.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:42:32.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:42:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:42:32.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:42:32.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:42:32.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:42:32.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:42:32.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:42:32.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:42:32.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:42:33.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:42:33.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:42:33.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:42:33.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:42:33.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:42:33.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:42:33.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:42:33.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: seq 1 200 2021-01-15T17:42:33.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:33.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj1 /dev/null 2021-01-15T17:42:33.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:33.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj2 /dev/null 2021-01-15T17:42:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj3 /dev/null 2021-01-15T17:42:34.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:34.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj4 /dev/null 2021-01-15T17:42:34.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:34.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj5 /dev/null 2021-01-15T17:42:34.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:34.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj6 /dev/null 2021-01-15T17:42:34.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:34.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj7 /dev/null 2021-01-15T17:42:35.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:35.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj8 /dev/null 2021-01-15T17:42:35.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:35.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj9 /dev/null 2021-01-15T17:42:35.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:35.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj10 /dev/null 2021-01-15T17:42:35.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:35.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj11 /dev/null 2021-01-15T17:42:35.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:35.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj12 /dev/null 2021-01-15T17:42:36.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj13 /dev/null 2021-01-15T17:42:36.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj14 /dev/null 2021-01-15T17:42:36.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj15 /dev/null 2021-01-15T17:42:36.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj16 /dev/null 2021-01-15T17:42:36.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj17 /dev/null 2021-01-15T17:42:36.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:36.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj18 /dev/null 2021-01-15T17:42:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj19 /dev/null 2021-01-15T17:42:37.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:37.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj20 /dev/null 2021-01-15T17:42:37.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:37.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj21 /dev/null 2021-01-15T17:42:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj22 /dev/null 2021-01-15T17:42:37.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:37.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj23 /dev/null 2021-01-15T17:42:38.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:38.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj24 /dev/null 2021-01-15T17:42:38.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:38.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj25 /dev/null 2021-01-15T17:42:38.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:38.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj26 /dev/null 2021-01-15T17:42:38.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:38.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj27 /dev/null 2021-01-15T17:42:38.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:38.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj28 /dev/null 2021-01-15T17:42:39.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:39.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj29 /dev/null 2021-01-15T17:42:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj30 /dev/null 2021-01-15T17:42:39.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:39.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj31 /dev/null 2021-01-15T17:42:39.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:39.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj32 /dev/null 2021-01-15T17:42:39.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:39.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj33 /dev/null 2021-01-15T17:42:40.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:40.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj34 /dev/null 2021-01-15T17:42:40.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:40.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj35 /dev/null 2021-01-15T17:42:40.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:40.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj36 /dev/null 2021-01-15T17:42:40.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:40.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj37 /dev/null 2021-01-15T17:42:40.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:40.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj38 /dev/null 2021-01-15T17:42:41.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:41.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj39 /dev/null 2021-01-15T17:42:41.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:41.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj40 /dev/null 2021-01-15T17:42:41.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:41.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj41 /dev/null 2021-01-15T17:42:41.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:41.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj42 /dev/null 2021-01-15T17:42:41.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:41.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj43 /dev/null 2021-01-15T17:42:42.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj44 /dev/null 2021-01-15T17:42:42.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj45 /dev/null 2021-01-15T17:42:42.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj46 /dev/null 2021-01-15T17:42:42.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj47 /dev/null 2021-01-15T17:42:42.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj48 /dev/null 2021-01-15T17:42:42.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:42.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj49 /dev/null 2021-01-15T17:42:43.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:43.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj50 /dev/null 2021-01-15T17:42:43.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:43.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj51 /dev/null 2021-01-15T17:42:43.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:43.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj52 /dev/null 2021-01-15T17:42:43.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:43.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj53 /dev/null 2021-01-15T17:42:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj54 /dev/null 2021-01-15T17:42:44.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:44.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj55 /dev/null 2021-01-15T17:42:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:44.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj56 /dev/null 2021-01-15T17:42:44.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:44.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj57 /dev/null 2021-01-15T17:42:44.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:44.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj58 /dev/null 2021-01-15T17:42:44.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:44.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj59 /dev/null 2021-01-15T17:42:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:45.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj60 /dev/null 2021-01-15T17:42:45.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:45.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj61 /dev/null 2021-01-15T17:42:45.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:45.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj62 /dev/null 2021-01-15T17:42:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj63 /dev/null 2021-01-15T17:42:45.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:45.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj64 /dev/null 2021-01-15T17:42:46.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:46.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj65 /dev/null 2021-01-15T17:42:46.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:46.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj66 /dev/null 2021-01-15T17:42:46.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:46.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj67 /dev/null 2021-01-15T17:42:46.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:46.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj68 /dev/null 2021-01-15T17:42:46.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:46.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj69 /dev/null 2021-01-15T17:42:47.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:47.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj70 /dev/null 2021-01-15T17:42:47.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:47.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj71 /dev/null 2021-01-15T17:42:47.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:47.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj72 /dev/null 2021-01-15T17:42:47.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:47.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj73 /dev/null 2021-01-15T17:42:47.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:47.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj74 /dev/null 2021-01-15T17:42:48.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj75 /dev/null 2021-01-15T17:42:48.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj76 /dev/null 2021-01-15T17:42:48.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj77 /dev/null 2021-01-15T17:42:48.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj78 /dev/null 2021-01-15T17:42:48.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj79 /dev/null 2021-01-15T17:42:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:48.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj80 /dev/null 2021-01-15T17:42:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj81 /dev/null 2021-01-15T17:42:49.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:49.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj82 /dev/null 2021-01-15T17:42:49.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:49.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj83 /dev/null 2021-01-15T17:42:49.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:49.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj84 /dev/null 2021-01-15T17:42:50.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:50.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj85 /dev/null 2021-01-15T17:42:50.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:50.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj86 /dev/null 2021-01-15T17:42:50.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:50.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj87 /dev/null 2021-01-15T17:42:50.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:50.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj88 /dev/null 2021-01-15T17:42:51.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:51.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj89 /dev/null 2021-01-15T17:42:51.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:51.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj90 /dev/null 2021-01-15T17:42:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:51.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj91 /dev/null 2021-01-15T17:42:51.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:51.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj92 /dev/null 2021-01-15T17:42:51.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:51.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj93 /dev/null 2021-01-15T17:42:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj94 /dev/null 2021-01-15T17:42:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj95 /dev/null 2021-01-15T17:42:52.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:52.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj96 /dev/null 2021-01-15T17:42:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:52.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj97 /dev/null 2021-01-15T17:42:52.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:52.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj98 /dev/null 2021-01-15T17:42:53.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj99 /dev/null 2021-01-15T17:42:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj100 /dev/null 2021-01-15T17:42:53.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj101 /dev/null 2021-01-15T17:42:53.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj102 /dev/null 2021-01-15T17:42:53.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj103 /dev/null 2021-01-15T17:42:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj104 /dev/null 2021-01-15T17:42:54.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:54.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj105 /dev/null 2021-01-15T17:42:54.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:54.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj106 /dev/null 2021-01-15T17:42:54.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:54.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj107 /dev/null 2021-01-15T17:42:54.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:54.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj108 /dev/null 2021-01-15T17:42:54.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:54.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj109 /dev/null 2021-01-15T17:42:55.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:55.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj110 /dev/null 2021-01-15T17:42:55.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:55.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj111 /dev/null 2021-01-15T17:42:55.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:55.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj112 /dev/null 2021-01-15T17:42:55.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:55.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj113 /dev/null 2021-01-15T17:42:55.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:55.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj114 /dev/null 2021-01-15T17:42:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj115 /dev/null 2021-01-15T17:42:56.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:56.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj116 /dev/null 2021-01-15T17:42:56.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:56.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj117 /dev/null 2021-01-15T17:42:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj118 /dev/null 2021-01-15T17:42:56.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:56.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj119 /dev/null 2021-01-15T17:42:57.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:57.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj120 /dev/null 2021-01-15T17:42:57.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:57.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj121 /dev/null 2021-01-15T17:42:57.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:57.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj122 /dev/null 2021-01-15T17:42:57.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:57.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj123 /dev/null 2021-01-15T17:42:57.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:57.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj124 /dev/null 2021-01-15T17:42:58.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj125 /dev/null 2021-01-15T17:42:58.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:58.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj126 /dev/null 2021-01-15T17:42:58.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:58.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj127 /dev/null 2021-01-15T17:42:58.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:58.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj128 /dev/null 2021-01-15T17:42:58.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:58.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj129 /dev/null 2021-01-15T17:42:59.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:59.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj130 /dev/null 2021-01-15T17:42:59.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:59.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj131 /dev/null 2021-01-15T17:42:59.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:59.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj132 /dev/null 2021-01-15T17:42:59.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:59.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj133 /dev/null 2021-01-15T17:42:59.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:42:59.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj134 /dev/null 2021-01-15T17:43:00.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:00.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj135 /dev/null 2021-01-15T17:43:00.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:00.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj136 /dev/null 2021-01-15T17:43:00.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:00.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj137 /dev/null 2021-01-15T17:43:00.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:00.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj138 /dev/null 2021-01-15T17:43:00.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:00.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj139 /dev/null 2021-01-15T17:43:01.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:01.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj140 /dev/null 2021-01-15T17:43:01.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:01.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj141 /dev/null 2021-01-15T17:43:01.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:01.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj142 /dev/null 2021-01-15T17:43:01.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:01.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj143 /dev/null 2021-01-15T17:43:01.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:01.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj144 /dev/null 2021-01-15T17:43:02.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj145 /dev/null 2021-01-15T17:43:02.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj146 /dev/null 2021-01-15T17:43:02.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj147 /dev/null 2021-01-15T17:43:02.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj148 /dev/null 2021-01-15T17:43:02.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj149 /dev/null 2021-01-15T17:43:02.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:02.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj150 /dev/null 2021-01-15T17:43:03.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:03.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj151 /dev/null 2021-01-15T17:43:03.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:03.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj152 /dev/null 2021-01-15T17:43:03.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:03.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj153 /dev/null 2021-01-15T17:43:03.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:03.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj154 /dev/null 2021-01-15T17:43:03.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:03.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj155 /dev/null 2021-01-15T17:43:04.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:04.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj156 /dev/null 2021-01-15T17:43:04.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:04.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj157 /dev/null 2021-01-15T17:43:04.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj158 /dev/null 2021-01-15T17:43:04.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:04.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj159 /dev/null 2021-01-15T17:43:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:04.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj160 /dev/null 2021-01-15T17:43:05.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:05.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj161 /dev/null 2021-01-15T17:43:05.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:05.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj162 /dev/null 2021-01-15T17:43:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:05.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj163 /dev/null 2021-01-15T17:43:05.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:05.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj164 /dev/null 2021-01-15T17:43:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:05.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj165 /dev/null 2021-01-15T17:43:06.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:06.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj166 /dev/null 2021-01-15T17:43:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj167 /dev/null 2021-01-15T17:43:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj168 /dev/null 2021-01-15T17:43:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj169 /dev/null 2021-01-15T17:43:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj170 /dev/null 2021-01-15T17:43:07.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:07.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj171 /dev/null 2021-01-15T17:43:07.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:07.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj172 /dev/null 2021-01-15T17:43:07.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:07.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj173 /dev/null 2021-01-15T17:43:07.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:07.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj174 /dev/null 2021-01-15T17:43:07.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:07.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj175 /dev/null 2021-01-15T17:43:08.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:08.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj176 /dev/null 2021-01-15T17:43:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj177 /dev/null 2021-01-15T17:43:08.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:08.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj178 /dev/null 2021-01-15T17:43:08.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:08.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj179 /dev/null 2021-01-15T17:43:08.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:08.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj180 /dev/null 2021-01-15T17:43:09.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:09.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj181 /dev/null 2021-01-15T17:43:09.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:09.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj182 /dev/null 2021-01-15T17:43:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:09.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj183 /dev/null 2021-01-15T17:43:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:09.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj184 /dev/null 2021-01-15T17:43:09.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:09.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj185 /dev/null 2021-01-15T17:43:10.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:10.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj186 /dev/null 2021-01-15T17:43:10.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:10.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj187 /dev/null 2021-01-15T17:43:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj188 /dev/null 2021-01-15T17:43:10.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:10.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj189 /dev/null 2021-01-15T17:43:10.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:10.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj190 /dev/null 2021-01-15T17:43:11.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:11.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj191 /dev/null 2021-01-15T17:43:11.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:11.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj192 /dev/null 2021-01-15T17:43:11.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:11.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj193 /dev/null 2021-01-15T17:43:11.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:11.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj194 /dev/null 2021-01-15T17:43:11.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:11.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj195 /dev/null 2021-01-15T17:43:12.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:12.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj196 /dev/null 2021-01-15T17:43:12.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:12.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj197 /dev/null 2021-01-15T17:43:12.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:12.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj198 /dev/null 2021-01-15T17:43:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:12.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj199 /dev/null 2021-01-15T17:43:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:530: TEST_backfill_ec_all_out: for i in $(seq 1 $objects) 2021-01-15T17:43:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:532: TEST_backfill_ec_all_out: rados -p test put obj200 /dev/null 2021-01-15T17:43:13.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:535: TEST_backfill_ec_all_out: get_pg test obj1 2021-01-15T17:43:13.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:43:13.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:43:13.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:43:13.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:43:13.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:535: TEST_backfill_ec_all_out: local PG=1.0 2021-01-15T17:43:13.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:537: TEST_backfill_ec_all_out: get_primary test obj1 2021-01-15T17:43:13.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:43:13.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:43:13.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:43:13.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:43:13.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:537: TEST_backfill_ec_all_out: local primary=1 2021-01-15T17:43:13.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:539: TEST_backfill_ec_all_out: ceph osd set nobackfill 2021-01-15T17:43:14.597 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:43:14.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: get_osds test obj1 2021-01-15T17:43:14.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2021-01-15T17:43:14.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2021-01-15T17:43:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2021-01-15T17:43:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T17:43:15.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2021-01-15T17:43:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T17:43:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:43:15.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2021-01-15T17:43:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2021-01-15T17:43:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.1 2021-01-15T17:43:15.660 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:43:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2021-01-15T17:43:15.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.0 2021-01-15T17:43:16.285 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:43:16.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:540: TEST_backfill_ec_all_out: for o in $(get_osds $poolname obj1) 2021-01-15T17:43:16.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:542: TEST_backfill_ec_all_out: ceph osd out osd.2 2021-01-15T17:43:17.094 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T17:43:17.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:545: TEST_backfill_ec_all_out: sleep 2 2021-01-15T17:43:19.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:546: TEST_backfill_ec_all_out: get_primary test obj1 2021-01-15T17:43:19.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:43:19.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:43:19.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:43:19.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:43:19.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:546: TEST_backfill_ec_all_out: primary=1 2021-01-15T17:43:19.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:547: TEST_backfill_ec_all_out: ceph osd unset nobackfill 2021-01-15T17:43:20.368 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:43:20.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:548: TEST_backfill_ec_all_out: ceph tell osd.1 get_latest_osdmap 2021-01-15T17:43:20.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:549: TEST_backfill_ec_all_out: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:43:20.666 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:43:20.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:550: TEST_backfill_ec_all_out: sleep 2 2021-01-15T17:43:22.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:552: TEST_backfill_ec_all_out: wait_for_clean 2021-01-15T17:43:22.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:43:22.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:43:22.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:43:22.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:43:22.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:43:22.678 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:43:22.679 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:43:22.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:43:22.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:43:22.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:43:22.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:43:22.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:43:22.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:43:22.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:43:22.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:43:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:43:23.177 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:43:23.177 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:43:23.177 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:43:23.177 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:43:23.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:43:23.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:43:23.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:43:23.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2021-01-15T17:43:23.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2021-01-15T17:43:23.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2021-01-15T17:43:23.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:43:23.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:43:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672981 2021-01-15T17:43:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672981 2021-01-15T17:43:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981' 2021-01-15T17:43:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:43:23.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:43:23.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509459 2021-01-15T17:43:23.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509459 2021-01-15T17:43:23.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459' 2021-01-15T17:43:23.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:43:23.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:43:23.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345938 2021-01-15T17:43:23.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345938 2021-01-15T17:43:23.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459 3-85899345938' 2021-01-15T17:43:23.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:43:23.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:43:23.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182416 2021-01-15T17:43:23.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182416 2021-01-15T17:43:23.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459 3-85899345938 4-107374182416' 2021-01-15T17:43:23.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:43:23.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2021-01-15T17:43:23.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:43:23.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:43:23.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2021-01-15T17:43:23.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:43:23.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2021-01-15T17:43:23.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-01-15T17:43:23.871 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836502 2021-01-15T17:43:23.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:43:24.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836502 2021-01-15T17:43:24.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:43:25.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:43:25.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:43:25.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836502 2021-01-15T17:43:25.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:43:25.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672981 2021-01-15T17:43:25.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:43:25.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:43:25.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672981 2021-01-15T17:43:25.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:43:25.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672981 2021-01-15T17:43:25.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2021-01-15T17:43:25.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:43:25.709 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672981 2021-01-15T17:43:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672981 -lt 42949672981 2021-01-15T17:43:26.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:43:26.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509459 2021-01-15T17:43:26.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:43:26.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:43:26.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509459 2021-01-15T17:43:26.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:43:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509459 2021-01-15T17:43:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-01-15T17:43:26.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:43:26.108 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509459 2021-01-15T17:43:26.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509460 -lt 64424509459 2021-01-15T17:43:26.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:43:26.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345938 2021-01-15T17:43:26.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:43:26.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:43:26.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345938 2021-01-15T17:43:26.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:43:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345938 2021-01-15T17:43:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2021-01-15T17:43:26.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:43:26.506 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345938 2021-01-15T17:43:26.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345938 -lt 85899345938 2021-01-15T17:43:26.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:43:26.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182416 2021-01-15T17:43:26.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:43:26.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:43:26.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182416 2021-01-15T17:43:26.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:43:26.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182416 2021-01-15T17:43:26.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2021-01-15T17:43:26.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:43:26.901 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182416 2021-01-15T17:43:27.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182416 -lt 107374182416 2021-01-15T17:43:27.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:43:27.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:27.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:27.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:43:27.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:27.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:27.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:27.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:27.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:27.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:27.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:28.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:28.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:28.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:28.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:43:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:43:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:43:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:28.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:28.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:28.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:28.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:28.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:28.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:28.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:29.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:29.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:29.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:29.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:29.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:29.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:29.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:29.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:29.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:29.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:30.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:43:30.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:43:30.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:30.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:30.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:30.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:30.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:30.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:30.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:30.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:30.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:30.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:30.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:30.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:30.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:30.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:30.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:31.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:31.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:31.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:31.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:31.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:31.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:31.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:43:31.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:43:31.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:31.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:31.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:31.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:31.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:31.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:31.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:31.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:31.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:31.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:31.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:32.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:32.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:32.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:32.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:32.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:32.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:32.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:32.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:32.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:32.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:32.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:32.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:32.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:33.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:43:33.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:43:33.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:33.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:33.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:33.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:33.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:33.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:33.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:33.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:33.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:33.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:33.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:33.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:33.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:33.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:33.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:34.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:34.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:34.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:34.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:34.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:34.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:34.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:34.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:34.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:34.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:34.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:34.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:34.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:34.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:34.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:35.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:35.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:35.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:35.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:35.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:35.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:35.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:35.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:35.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:36.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:36.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:36.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:36.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:36.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:36.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:36.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:36.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:36.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:36.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:36.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:36.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:36.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:37.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:37.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:37.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:37.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:37.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:38.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:38.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:38.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:38.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:38.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:38.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:38.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:38.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:38.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:38.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:38.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:38.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:38.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:38.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:38.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:38.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:38.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:39.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:39.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:39.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:39.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:39.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:39.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:39.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:39.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:39.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:39.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:39.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:39.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:39.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:39.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:39.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:39.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:39.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:39.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:39.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:39.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:39.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:39.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:40.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:40.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:40.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:40.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:40.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:40.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:40.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:40.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:40.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:40.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:40.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:40.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:40.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:41.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:41.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:41.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:41.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:41.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:41.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:41.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:41.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:41.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:41.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:41.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:41.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:41.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:41.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:41.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:41.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:41.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:42.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:42.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:42.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:42.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:42.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:42.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:42.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:42.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:42.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:42.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:42.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:42.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:42.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:42.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:43.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:43.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:43.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:43.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:43.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:43.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:43.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:43.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:43.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:43.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:43.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:44.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:44.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:44.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:44.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:44.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:44.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:44.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:44.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:44.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:44.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:44.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:44.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:44.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:44.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:44.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:44.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:44.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:45.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:45.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:45.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:45.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:45.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:45.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:45.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:45.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:45.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:45.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:45.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:45.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:45.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:45.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:45.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:45.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:45.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:45.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:45.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:45.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:46.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:46.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:46.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:46.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:46.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:46.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:46.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:46.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:46.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:47.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:47.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:47.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:47.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:47.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:47.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:47.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:47.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:47.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:47.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:47.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:47.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:48.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:48.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:48.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:48.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:48.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:48.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:48.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:49.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:49.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:49.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:49.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:49.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:49.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:49.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:49.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:49.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:50.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:50.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:50.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:50.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:50.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:50.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:50.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:50.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:50.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:50.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:50.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:50.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:50.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:50.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:51.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:51.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:51.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:51.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:51.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:51.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:51.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:51.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:51.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:51.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:51.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:51.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:51.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:52.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:52.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:52.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:52.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:52.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:52.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:53.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:53.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:53.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:53.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:53.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:53.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:53.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:53.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:53.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:53.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:53.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:53.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:53.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:53.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:53.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:53.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:53.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:54.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:54.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:54.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:55.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:55.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:55.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:55.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:55.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:55.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:55.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:55.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:55.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:55.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:55.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:56.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:56.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:56.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:56.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:56.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:56.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:56.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:56.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:56.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:56.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:56.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:56.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:56.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:56.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:56.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:56.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:56.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:56.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:56.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:57.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:57.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:57.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:57.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:57.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:57.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:58.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:43:58.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:43:58.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:58.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:58.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:58.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:58.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:58.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:58.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:58.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:58.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:58.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:58.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:43:58.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:43:58.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:43:58.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:43:58.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:43:59.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:43:59.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:43:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:43:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:43:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:43:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:43:59.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:43:59.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:43:59.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:43:59.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:43:59.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:43:59.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:43:59.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:43:59.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:43:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:43:59.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:43:59.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:43:59.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:43:59.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:43:59.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:43:59.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:43:59.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:00.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:00.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:00.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:00.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:00.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:00.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:00.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:00.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:00.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:01.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:01.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:01.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:01.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:01.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:01.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:01.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:01.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:01.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:01.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:01.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:01.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:01.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:01.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:01.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:01.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:02.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:02.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:02.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:02.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:02.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:02.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:02.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:02.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:02.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:02.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:02.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:02.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:02.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:02.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:02.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:02.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:03.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:03.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:03.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:03.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:03.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:03.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:03.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:03.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:03.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:03.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:03.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:03.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:44:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:44:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:04.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:04.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:04.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:04.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:04.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:04.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:04.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:04.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:04.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:04.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:04.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:04.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:04.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:05.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:05.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:05.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:05.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:05.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:05.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:05.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:05.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:05.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:05.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:05.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:05.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:05.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:05.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:05.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:05.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:06.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:06.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:06.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:06.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:06.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:06.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:06.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:06.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:06.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:06.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:06.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:06.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:06.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:07.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:07.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:07.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:07.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:07.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:07.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:07.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:07.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:07.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:07.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:07.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:07.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:07.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:07.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:08.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:08.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:08.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:08.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:08.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:08.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:08.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:44:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:44:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:09.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:09.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:09.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:09.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:09.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:09.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:09.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:09.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:09.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:09.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:09.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:09.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:09.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:09.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:10.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:10.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:10.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:10.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:10.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:10.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:10.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:10.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:10.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:10.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:10.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:10.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:10.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:10.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:10.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:10.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:10.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:10.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:11.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:11.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:11.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:11.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:11.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:11.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:11.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:11.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:12.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:44:12.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:44:12.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:12.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:12.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:12.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:12.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:12.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:12.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:12.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:12.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:12.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:12.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:12.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:12.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:12.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:12.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:13.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:13.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:13.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:13.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:13.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:13.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:13.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:13.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:13.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:13.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:44:13.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:44:13.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:13.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:13.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:13.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:13.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:13.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:13.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:13.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:13.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:13.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:13.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:14.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:14.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:14.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:14.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:14.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:14.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:44:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:44:15.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:15.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:15.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:15.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:15.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:15.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:15.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:15.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:15.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:15.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:15.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:15.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:15.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:16.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:16.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:16.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:16.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:16.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:16.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:16.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:16.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:16.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:44:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:44:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:44:16.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:16.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:44:16.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:16.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:16.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:16.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:16.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:16.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:16.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:16.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:17.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:44:17.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:17.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:17.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:44:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:44:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:44:17.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:44:17.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:44:17.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:44:17.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:44:17.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:44:17.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:44:18.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T17:44:18.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T17:44:18.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2021-01-15T17:44:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:44:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T17:44:18.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:44:18.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:44:18.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:44:18.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:44:18.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:44:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:44:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:44:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:44:18.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:44:18.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:44:18.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:44:18.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:44:19.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:44:19.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:44:19.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:44:19.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:554: TEST_backfill_ec_all_out: expr 200 '*' 3 2021-01-15T17:44:19.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:554: TEST_backfill_ec_all_out: local misplaced=600 2021-01-15T17:44:19.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:555: TEST_backfill_ec_all_out: check td/osd-backfill-stats 1.0 1 erasure 0 0 600 200 2021-01-15T17:44:19.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:44:19.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:44:19.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:44:19.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2021-01-15T17:44:19.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2021-01-15T17:44:19.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:44:19.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=600 2021-01-15T17:44:19.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2021-01-15T17:44:19.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:44:19.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:44:19.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:44:19.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:44:19.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2021-01-15T17:44:19.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:44:19.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:44:19.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:44:19.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2021-01-15T17:44:19.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2021-01-15T17:44:19.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:44:19.436 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.436 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:44:19.437 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:44:19.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,3,NONE]/[1,0,2]' 2021-01-15T17:44:19.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,3,NONE]/[1,0,2]' ']' 2021-01-15T17:44:19.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:44:19.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:44:19.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,3,NONE]/[1,0,2]p' 2021-01-15T17:44:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:44:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:44:19.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:44:19.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2021-01-15T17:44:19.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2021-01-15T17:44:19.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2021-01-15T17:44:19.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:44:19.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:44:19.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:44:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:44:19.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:44:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:44:19.471 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:44:19.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2021-01-15T17:44:19.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2021-01-15T17:44:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2021-01-15T17:44:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:44:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:44:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:44:19.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:44:19.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:44:19.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:44:19.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,3,NONE]/[1,0,2]p' 2021-01-15T17:44:19.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:44:19.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:44:19.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:44:19.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=600 2021-01-15T17:44:19.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 600 ']' 2021-01-15T17:44:19.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 600 600 2021-01-15T17:44:19.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=600 2021-01-15T17:44:19.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:44:19.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=600 2021-01-15T17:44:19.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:44:19.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:44:19.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:44:19.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:44:19.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:44:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=202 2021-01-15T17:44:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 202 ']' 2021-01-15T17:44:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 202 200 2021-01-15T17:44:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=202 2021-01-15T17:44:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:44:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2021-01-15T17:44:19.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:44:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:44:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:557: TEST_backfill_ec_all_out: delete_pool test 2021-01-15T17:44:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:44:19.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:44:20.027 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:44:20.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:558: TEST_backfill_ec_all_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:44:20.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:44:20.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:44:20.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:44:20.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:44:20.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:44:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:44:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:44:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:44:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:44:20.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:44:20.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:44:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:44:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:44:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:44:20.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:44:20.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:44:20.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:44:20.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:44:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:44:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:44:20.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:44:20.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:44:20.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:44:20.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:44:20.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:44:20.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:44:20.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:44:20.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:44:20.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:44:20.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:44:20.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:44:20.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:44:20.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:44:20.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:44:20.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:44:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:44:20.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:44:20.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:44:20.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:44:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:44:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:44:20.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:44:20.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:44:20.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:44:20.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:44:20.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:44:20.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:44:20.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:44:20.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:44:20.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:44:20.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:44:20.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:44:20.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:44:20.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:44:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:44:20.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:44:20.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:44:20.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:44:20.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:44:20.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:44:20.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:44:20.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:44:20.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:44:20.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:44:20.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:44:20.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:44:20.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:44:20.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_down_all_out td/osd-backfill-stats 2021-01-15T17:44:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:619: TEST_backfill_ec_down_all_out: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:621: TEST_backfill_ec_down_all_out: run_mon td/osd-backfill-stats a 2021-01-15T17:44:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:44:20.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:44:20.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:44:20.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:44:20.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:44:20.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:44:20.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:20.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:20.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:20.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.489 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:20.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:44:20.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:44:20.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:44:20.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:44:20.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:44:20.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:44:20.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:44:20.675 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:44:20.675 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:44:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:44:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:44:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:44:20.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:44:20.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:44:20.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:44:20.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:44:20.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:44:20.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:44:20.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:44:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:44:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:44:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:44:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:44:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:44:20.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:44:20.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:44:20.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:622: TEST_backfill_ec_down_all_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:44:20.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:44:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:44:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:44:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:44:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:44:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:44:21.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:44:21.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:21.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:21.366 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:21.366 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:21.366 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:21.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:21.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:44:21.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:44:21.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:623: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:44:21.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:44:21.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:44:21.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:44:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:44:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:44:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:44:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:44:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:44:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:44:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:44:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:44:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:44:21.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:44:21.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:44:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:44:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:21.396 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:21.396 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:21.396 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:21.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:21.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:21.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:44:21.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:44:21.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:44:21.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:44:21.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:44:21.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:44:21.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:44:21.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:21.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:44:21.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:44:21.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:44:21.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:44:21.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fada5b77-4ff3-4a54-b7b8-2d2a66d965c0 2021-01-15T17:44:21.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 fada5b77-4ff3-4a54-b7b8-2d2a66d965c0' 2021-01-15T17:44:21.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:44:21.404 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 fada5b77-4ff3-4a54-b7b8-2d2a66d965c0 2021-01-15T17:44:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB11AFgFNpTGBAAvbUH05kzkTq+S+ApM7vahg== 2021-01-15T17:44:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB11AFgFNpTGBAAvbUH05kzkTq+S+ApM7vahg=="}' 2021-01-15T17:44:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fada5b77-4ff3-4a54-b7b8-2d2a66d965c0 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:44:21.749 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:21.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:44:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB11AFgFNpTGBAAvbUH05kzkTq+S+ApM7vahg== --osd-uuid fada5b77-4ff3-4a54-b7b8-2d2a66d965c0 2021-01-15T17:44:21.786 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.785+0000 7fe4ff175080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:21.787 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.787+0000 7fe4ff175080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:21.788 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.788+0000 7fe4ff175080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.789+0000 7fe4ff175080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:44:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.789+0000 7fe4ff175080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:44:21.790 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.789+0000 7fe4ff175080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:44:21.830 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:21.829+0000 7fe4ff175080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:44:26.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:44:26.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:44:26.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:44:26.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:44:26.364 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:44:26.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:44:26.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:26.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:44:26.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:44:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:44:26.891 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:44:26.913 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:26.913+0000 7f330b9a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:26.936 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:26.936+0000 7f330b9a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:26.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:26.937+0000 7f330b9a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:44:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:44:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:44:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:44:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:44:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:44:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:44:27.289 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:27.453 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:27.452+0000 7f330b9a6080 -1 Falling back to public interface 2021-01-15T17:44:27.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:44:28.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:28.229+0000 7f330b9a6080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:44:28.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:44:28.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:44:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:44:28.684 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:44:29.088 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/546715,v1:127.0.0.1:6803/546715] [v2:127.0.0.1:6804/546715,v1:127.0.0.1:6805/546715] exists,up fada5b77-4ff3-4a54-b7b8-2d2a66d965c0 2021-01-15T17:44:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:44:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:44:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:44:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:624: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:44:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:44:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:44:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:44:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:44:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:44:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:44:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:44:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:44:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:44:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:44:29.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:29.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:29.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:44:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:44:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:44:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:44:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:44:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:44:29.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:44:29.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=18df9754-7e30-4a50-a9cd-ea609300aa22 2021-01-15T17:44:29.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 18df9754-7e30-4a50-a9cd-ea609300aa22' 2021-01-15T17:44:29.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:44:29.097 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 18df9754-7e30-4a50-a9cd-ea609300aa22 2021-01-15T17:44:29.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB91AFgnwtJBhAApXZffwsfTsEqvf10u8m4Cg== 2021-01-15T17:44:29.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB91AFgnwtJBhAApXZffwsfTsEqvf10u8m4Cg=="}' 2021-01-15T17:44:29.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 18df9754-7e30-4a50-a9cd-ea609300aa22 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:44:29.643 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:44:29.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:44:29.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB91AFgnwtJBhAApXZffwsfTsEqvf10u8m4Cg== --osd-uuid 18df9754-7e30-4a50-a9cd-ea609300aa22 2021-01-15T17:44:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.681+0000 7ffb0f640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:29.683 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.683+0000 7ffb0f640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:29.684 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.684+0000 7ffb0f640080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:29.685 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.685+0000 7ffb0f640080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:44:29.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.685+0000 7ffb0f640080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:44:29.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.685+0000 7ffb0f640080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:44:29.735 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:29.735+0000 7ffb0f640080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:44:34.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:44:34.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:44:34.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:44:34.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:44:34.256 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:44:34.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:44:34.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:34.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:44:34.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:44:34.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:44:34.780 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:44:34.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:34.801+0000 7f8409830080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:34.835 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:34.834+0000 7f8409830080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:34.836 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:34.835+0000 7f8409830080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:35.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:44:35.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:44:35.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:44:35.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:44:35.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:44:35.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:35.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:44:35.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:35.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:44:35.175 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:35.350+0000 7f8409830080 -1 Falling back to public interface 2021-01-15T17:44:35.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:44:36.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:36.151+0000 7f8409830080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:44:36.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:44:36.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:36.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:44:36.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:36.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:44:36.569 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:44:36.978 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/547987,v1:127.0.0.1:6811/547987] [v2:127.0.0.1:6812/547987,v1:127.0.0.1:6813/547987] exists,up 18df9754-7e30-4a50-a9cd-ea609300aa22 2021-01-15T17:44:37.043 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 b753fdd3-e4fe-4a5f-a949-c8ffd544b402 2021-01-15T17:44:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:44:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:44:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:625: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:44:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:44:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:44:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:44:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:44:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:44:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:44:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:44:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:44:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:44:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:44:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:44:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:44:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:37.048 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:37.048 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:37.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:44:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:44:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:44:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:44:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:44:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:44:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:44:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:44:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:44:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:44:37.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:44:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b753fdd3-e4fe-4a5f-a949-c8ffd544b402 2021-01-15T17:44:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b753fdd3-e4fe-4a5f-a949-c8ffd544b402' 2021-01-15T17:44:37.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:44:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCE1AFg4YDsOhAAXg35D098psqx2BLBfM+VZg== 2021-01-15T17:44:37.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCE1AFg4YDsOhAAXg35D098psqx2BLBfM+VZg=="}' 2021-01-15T17:44:37.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b753fdd3-e4fe-4a5f-a949-c8ffd544b402 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:44:37.620 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:44:37.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:44:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCE1AFg4YDsOhAAXg35D098psqx2BLBfM+VZg== --osd-uuid b753fdd3-e4fe-4a5f-a949-c8ffd544b402 2021-01-15T17:44:37.658 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.657+0000 7f7a9e887080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:37.659 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.659+0000 7f7a9e887080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:37.660 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.659+0000 7f7a9e887080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:37.661 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.660+0000 7f7a9e887080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:44:37.661 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.660+0000 7f7a9e887080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:44:37.662 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.660+0000 7f7a9e887080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:44:37.700 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:37.700+0000 7f7a9e887080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:44:42.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:44:42.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:44:42.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:44:42.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:44:42.230 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:44:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:44:42.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:44:42.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:42.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:44:42.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:44:42.760 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:44:42.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:42.777+0000 7f2269bc9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:42.799 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:42.798+0000 7f2269bc9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:42.801 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:42.801+0000 7f2269bc9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:43.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:44:43.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:44:43.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:44:43.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:44:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:44:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:44:43.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:43.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:44:43.149 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:43.321 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:43.320+0000 7f2269bc9080 -1 Falling back to public interface 2021-01-15T17:44:43.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:44:44.200 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:44.200+0000 7f2269bc9080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:44:44.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:44:44.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:44.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:44:44.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:44.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:44:44.576 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:44:44.976 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/549251,v1:127.0.0.1:6819/549251] [v2:127.0.0.1:6820/549251,v1:127.0.0.1:6821/549251] exists,up b753fdd3-e4fe-4a5f-a949-c8ffd544b402 2021-01-15T17:44:44.976 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 11b1ecad-fe0d-4606-ae13-ff361b8d9406 2021-01-15T17:44:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:44:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:44:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:44:44.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:626: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:44:44.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:44:44.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:44:44.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:44:44.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:44:44.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:44:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:44:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:44:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:44:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:44:44.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:44:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:44:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:44:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:44:44.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:44:44.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:44:44.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:44.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:44.981 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:44.981 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:44.981 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:44.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:44.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:44.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:44:44.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:44:44.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:44:44.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:44:44.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:44:44.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:44:44.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:44:44.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:44.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:44:44.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:44:44.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:44:44.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:44:44.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=11b1ecad-fe0d-4606-ae13-ff361b8d9406 2021-01-15T17:44:44.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 11b1ecad-fe0d-4606-ae13-ff361b8d9406' 2021-01-15T17:44:44.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:44:44.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCM1AFgkdAXOxAA0BVRllM3j6fmdREijED1lw== 2021-01-15T17:44:44.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCM1AFgkdAXOxAA0BVRllM3j6fmdREijED1lw=="}' 2021-01-15T17:44:44.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 11b1ecad-fe0d-4606-ae13-ff361b8d9406 -i td/osd-backfill-stats/3/new.json 2021-01-15T17:44:45.591 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:44:45.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:44:45.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCM1AFgkdAXOxAA0BVRllM3j6fmdREijED1lw== --osd-uuid 11b1ecad-fe0d-4606-ae13-ff361b8d9406 2021-01-15T17:44:45.629 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.628+0000 7f88152e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:45.630 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.630+0000 7f88152e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.631+0000 7f88152e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.632+0000 7f88152e0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:44:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.632+0000 7f88152e0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:44:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.632+0000 7f88152e0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:44:45.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:45.680+0000 7f88152e0080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:44:50.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:44:50.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:44:50.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:44:50.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:44:50.299 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:44:50.827 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:44:50.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:44:50.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:50.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:44:50.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:44:50.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:44:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:50.849+0000 7f6136aba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:50.855+0000 7f6136aba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:50.856 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:50.856+0000 7f6136aba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:51.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:44:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:44:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:44:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:44:51.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:44:51.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:51.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:44:51.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:51.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:44:51.228 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:51.375 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:51.375+0000 7f6136aba080 -1 Falling back to public interface 2021-01-15T17:44:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:44:52.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:52.172+0000 7f6136aba080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:44:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:44:52.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:52.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:44:52.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:44:52.647 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:44:53.060 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/550515,v1:127.0.0.1:6827/550515] [v2:127.0.0.1:6828/550515,v1:127.0.0.1:6829/550515] exists,up 11b1ecad-fe0d-4606-ae13-ff361b8d9406 2021-01-15T17:44:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:44:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:44:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:44:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:627: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:44:53.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:44:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:44:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:44:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:44:53.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:44:53.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:44:53.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:44:53.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:44:53.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:44:53.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:44:53.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:44:53.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:44:53.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:44:53.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:44:53.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:44:53.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:44:53.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:44:53.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:44:53.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:44:53.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:44:53.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:53.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:44:53.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:44:53.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:44:53.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:44:53.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:44:53.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:44:53.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:44:53.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:44:53.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:44:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:44:53.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:44:53.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:44:53.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5a3c854d-9d13-42d5-992a-4718c56fc0e0 2021-01-15T17:44:53.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 5a3c854d-9d13-42d5-992a-4718c56fc0e0' 2021-01-15T17:44:53.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:44:53.071 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 5a3c854d-9d13-42d5-992a-4718c56fc0e0 2021-01-15T17:44:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCV1AFgcKiGBBAAQB/6iBk2LrdoWGv1MfkbVw== 2021-01-15T17:44:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCV1AFgcKiGBBAAQB/6iBk2LrdoWGv1MfkbVw=="}' 2021-01-15T17:44:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5a3c854d-9d13-42d5-992a-4718c56fc0e0 -i td/osd-backfill-stats/4/new.json 2021-01-15T17:44:53.641 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:44:53.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:44:53.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCV1AFgcKiGBBAAQB/6iBk2LrdoWGv1MfkbVw== --osd-uuid 5a3c854d-9d13-42d5-992a-4718c56fc0e0 2021-01-15T17:44:53.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.676+0000 7f77c7d89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.678+0000 7f77c7d89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:53.680 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.680+0000 7f77c7d89080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:53.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.680+0000 7f77c7d89080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:44:53.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.680+0000 7f77c7d89080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:44:53.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.680+0000 7f77c7d89080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:44:53.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:53.763+0000 7f77c7d89080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:44:58.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:44:58.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:44:58.550 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:44:58.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:44:58.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:44:59.076 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:44:59.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:44:59.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:44:59.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:44:59.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:44:59.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:44:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:59.100+0000 7f284b239080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:59.122 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:59.121+0000 7f284b239080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:59.123 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:59.123+0000 7f284b239080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:44:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:44:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:44:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:44:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:44:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:44:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:44:59.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:44:59.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:44:59.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:44:59.482 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:44:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:44:59.639+0000 7f284b239080 -1 Falling back to public interface 2021-01-15T17:44:59.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:45:00.425 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:00.423+0000 7f284b239080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:45:00.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:45:00.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:45:00.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:45:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:45:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:45:00.880 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:45:01.310 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/551779,v1:127.0.0.1:6835/551779] [v2:127.0.0.1:6836/551779,v1:127.0.0.1:6837/551779] exists,up 5a3c854d-9d13-42d5-992a-4718c56fc0e0 2021-01-15T17:45:01.310 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 8fb314bb-055b-4fea-9911-1c87c35a7e0d 2021-01-15T17:45:01.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:45:01.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:45:01.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:45:01.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:628: TEST_backfill_ec_down_all_out: run_osd td/osd-backfill-stats 5 2021-01-15T17:45:01.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:45:01.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:45:01.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:45:01.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:45:01.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:45:01.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:45:01.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:45:01.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:45:01.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:45:01.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:45:01.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:45:01.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:45:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:45:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:45:01.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:45:01.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:45:01.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:45:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:45:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:45:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:45:01.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:45:01.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:45:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:45:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:45:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:45:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:45:01.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:45:01.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:45:01.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:45:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:45:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:45:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:45:01.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:45:01.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:45:01.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8fb314bb-055b-4fea-9911-1c87c35a7e0d 2021-01-15T17:45:01.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 8fb314bb-055b-4fea-9911-1c87c35a7e0d' 2021-01-15T17:45:01.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:45:01.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCd1AFgDkteExAA+p32zKbVdMpDyAVXdhyaPA== 2021-01-15T17:45:01.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCd1AFgDkteExAA+p32zKbVdMpDyAVXdhyaPA=="}' 2021-01-15T17:45:01.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8fb314bb-055b-4fea-9911-1c87c35a7e0d -i td/osd-backfill-stats/5/new.json 2021-01-15T17:45:01.859 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:45:01.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:45:01.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCd1AFgDkteExAA+p32zKbVdMpDyAVXdhyaPA== --osd-uuid 8fb314bb-055b-4fea-9911-1c87c35a7e0d 2021-01-15T17:45:01.896 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.895+0000 7f2176dde080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:01.901 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.897+0000 7f2176dde080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:01.901 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.898+0000 7f2176dde080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:01.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.898+0000 7f2176dde080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:45:01.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.898+0000 7f2176dde080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:45:01.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.898+0000 7f2176dde080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:45:01.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:01.947+0000 7f2176dde080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:45:06.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:45:06.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:45:06.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:45:06.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:45:06.500 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:45:07.025 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:45:07.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:45:07.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:45:07.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:45:07.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:45:07.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:45:07.048 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:07.048+0000 7ff9f153d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:07.073 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:07.072+0000 7ff9f153d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:07.074+0000 7ff9f153d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:45:07.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:45:07.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:45:07.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:45:07.423 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:45:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:07.588+0000 7ff9f153d080 -1 Falling back to public interface 2021-01-15T17:45:07.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:45:08.374 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:45:08.374+0000 7ff9f153d080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:45:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:45:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:45:08.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:45:08.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:45:08.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:45:08.825 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:45:09.229 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/553043,v1:127.0.0.1:6843/553043] [v2:127.0.0.1:6844/553043,v1:127.0.0.1:6845/553043] exists,up 8fb314bb-055b-4fea-9911-1c87c35a7e0d 2021-01-15T17:45:09.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:45:09.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:45:09.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:45:09.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:630: TEST_backfill_ec_down_all_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T17:45:09.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:631: TEST_backfill_ec_down_all_out: create_pool test 1 1 erasure myprofile 2021-01-15T17:45:09.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T17:45:10.614 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:45:10.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:45:11.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:632: TEST_backfill_ec_down_all_out: ceph osd pool set test min_size 2 2021-01-15T17:45:12.336 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T17:45:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:634: TEST_backfill_ec_down_all_out: wait_for_clean 2021-01-15T17:45:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:45:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:45:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:45:12.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:45:12.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:45:12.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:45:12.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:45:12.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:45:12.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:45:12.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:45:12.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:45:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:45:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:45:12.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:45:12.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:45:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:45:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:45:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:45:12.849 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:45:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:45:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:45:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:45:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:12.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:45:13.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T17:45:13.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T17:45:13.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T17:45:13.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:13.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:45:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T17:45:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T17:45:13.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-01-15T17:45:13.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:13.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:45:13.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T17:45:13.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T17:45:13.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448' 2021-01-15T17:45:13.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:13.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:45:13.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:45:13.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:45:13.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927' 2021-01-15T17:45:13.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:13.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:45:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T17:45:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T17:45:13.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405' 2021-01-15T17:45:13.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:45:13.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:45:13.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018883 2021-01-15T17:45:13.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018883 2021-01-15T17:45:13.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405 5-128849018883' 2021-01-15T17:45:13.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:13.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T17:45:13.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:13.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:45:13.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T17:45:13.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:13.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T17:45:13.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T17:45:13.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:45:13.713 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T17:45:14.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2021-01-15T17:45:14.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:14.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T17:45:14.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:14.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:45:14.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T17:45:14.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:14.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T17:45:14.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T17:45:14.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:45:14.101 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T17:45:14.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T17:45:14.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:14.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T17:45:14.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:14.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:45:14.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T17:45:14.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:14.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T17:45:14.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T17:45:14.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:45:14.497 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T17:45:14.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509448 2021-01-15T17:45:14.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:45:15.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:45:15.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:45:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T17:45:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:16.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:45:16.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:45:16.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:45:16.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:45:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:45:16.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:45:16.299 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:45:16.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2021-01-15T17:45:16.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:16.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T17:45:16.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:16.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:45:16.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T17:45:16.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:16.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T17:45:16.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T17:45:16.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:45:16.694 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T17:45:17.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T17:45:17.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:45:17.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018883 2021-01-15T17:45:17.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:45:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:45:17.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018883 2021-01-15T17:45:17.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:45:17.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018883 2021-01-15T17:45:17.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-01-15T17:45:17.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:45:17.088 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018883 2021-01-15T17:45:17.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018884 -lt 128849018883 2021-01-15T17:45:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:45:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:45:17.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:45:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:45:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:45:18.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:45:18.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:45:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:45:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:45:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:45:18.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:45:18.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:45:18.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:45:18.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:45:18.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:45:18.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:45:18.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:45:18.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:45:18.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: seq 1 200 2021-01-15T17:45:18.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:18.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj1 /dev/null 2021-01-15T17:45:19.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj2 /dev/null 2021-01-15T17:45:19.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj3 /dev/null 2021-01-15T17:45:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj4 /dev/null 2021-01-15T17:45:19.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj5 /dev/null 2021-01-15T17:45:19.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj6 /dev/null 2021-01-15T17:45:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:19.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj7 /dev/null 2021-01-15T17:45:20.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj8 /dev/null 2021-01-15T17:45:20.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj9 /dev/null 2021-01-15T17:45:20.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj10 /dev/null 2021-01-15T17:45:20.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj11 /dev/null 2021-01-15T17:45:20.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj12 /dev/null 2021-01-15T17:45:20.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:20.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj13 /dev/null 2021-01-15T17:45:21.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj14 /dev/null 2021-01-15T17:45:21.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj15 /dev/null 2021-01-15T17:45:21.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj16 /dev/null 2021-01-15T17:45:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj17 /dev/null 2021-01-15T17:45:21.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj18 /dev/null 2021-01-15T17:45:21.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:21.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj19 /dev/null 2021-01-15T17:45:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj20 /dev/null 2021-01-15T17:45:22.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj21 /dev/null 2021-01-15T17:45:22.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj22 /dev/null 2021-01-15T17:45:22.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj23 /dev/null 2021-01-15T17:45:22.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj24 /dev/null 2021-01-15T17:45:22.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:22.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj25 /dev/null 2021-01-15T17:45:23.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj26 /dev/null 2021-01-15T17:45:23.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj27 /dev/null 2021-01-15T17:45:23.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj28 /dev/null 2021-01-15T17:45:23.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj29 /dev/null 2021-01-15T17:45:23.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj30 /dev/null 2021-01-15T17:45:23.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:23.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj31 /dev/null 2021-01-15T17:45:24.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:24.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj32 /dev/null 2021-01-15T17:45:24.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:24.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj33 /dev/null 2021-01-15T17:45:24.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj34 /dev/null 2021-01-15T17:45:24.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:24.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj35 /dev/null 2021-01-15T17:45:24.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:24.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj36 /dev/null 2021-01-15T17:45:25.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:25.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj37 /dev/null 2021-01-15T17:45:25.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:25.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj38 /dev/null 2021-01-15T17:45:25.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:25.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj39 /dev/null 2021-01-15T17:45:25.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:25.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj40 /dev/null 2021-01-15T17:45:25.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:25.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj41 /dev/null 2021-01-15T17:45:26.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj42 /dev/null 2021-01-15T17:45:26.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj43 /dev/null 2021-01-15T17:45:26.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj44 /dev/null 2021-01-15T17:45:26.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj45 /dev/null 2021-01-15T17:45:26.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj46 /dev/null 2021-01-15T17:45:26.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:26.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj47 /dev/null 2021-01-15T17:45:27.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj48 /dev/null 2021-01-15T17:45:27.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj49 /dev/null 2021-01-15T17:45:27.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj50 /dev/null 2021-01-15T17:45:27.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj51 /dev/null 2021-01-15T17:45:27.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj52 /dev/null 2021-01-15T17:45:27.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:27.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj53 /dev/null 2021-01-15T17:45:28.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj54 /dev/null 2021-01-15T17:45:28.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj55 /dev/null 2021-01-15T17:45:28.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj56 /dev/null 2021-01-15T17:45:28.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj57 /dev/null 2021-01-15T17:45:28.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj58 /dev/null 2021-01-15T17:45:28.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:28.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj59 /dev/null 2021-01-15T17:45:29.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:29.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj60 /dev/null 2021-01-15T17:45:29.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:29.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj61 /dev/null 2021-01-15T17:45:29.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:29.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj62 /dev/null 2021-01-15T17:45:29.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:29.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj63 /dev/null 2021-01-15T17:45:29.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:29.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj64 /dev/null 2021-01-15T17:45:30.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj65 /dev/null 2021-01-15T17:45:30.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj66 /dev/null 2021-01-15T17:45:30.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj67 /dev/null 2021-01-15T17:45:30.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj68 /dev/null 2021-01-15T17:45:30.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj69 /dev/null 2021-01-15T17:45:30.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj70 /dev/null 2021-01-15T17:45:30.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:30.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj71 /dev/null 2021-01-15T17:45:31.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:31.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj72 /dev/null 2021-01-15T17:45:31.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:31.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj73 /dev/null 2021-01-15T17:45:31.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:31.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj74 /dev/null 2021-01-15T17:45:31.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:31.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj75 /dev/null 2021-01-15T17:45:31.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:31.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj76 /dev/null 2021-01-15T17:45:32.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj77 /dev/null 2021-01-15T17:45:32.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj78 /dev/null 2021-01-15T17:45:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj79 /dev/null 2021-01-15T17:45:32.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj80 /dev/null 2021-01-15T17:45:32.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj81 /dev/null 2021-01-15T17:45:32.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:32.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj82 /dev/null 2021-01-15T17:45:33.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj83 /dev/null 2021-01-15T17:45:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj84 /dev/null 2021-01-15T17:45:33.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj85 /dev/null 2021-01-15T17:45:33.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj86 /dev/null 2021-01-15T17:45:33.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj87 /dev/null 2021-01-15T17:45:33.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:33.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj88 /dev/null 2021-01-15T17:45:34.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj89 /dev/null 2021-01-15T17:45:34.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj90 /dev/null 2021-01-15T17:45:34.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj91 /dev/null 2021-01-15T17:45:34.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj92 /dev/null 2021-01-15T17:45:34.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj93 /dev/null 2021-01-15T17:45:34.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:34.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj94 /dev/null 2021-01-15T17:45:35.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:35.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj95 /dev/null 2021-01-15T17:45:35.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:35.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj96 /dev/null 2021-01-15T17:45:35.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:35.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj97 /dev/null 2021-01-15T17:45:35.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:35.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj98 /dev/null 2021-01-15T17:45:35.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:35.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj99 /dev/null 2021-01-15T17:45:36.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:36.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj100 /dev/null 2021-01-15T17:45:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:36.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj101 /dev/null 2021-01-15T17:45:36.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:36.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj102 /dev/null 2021-01-15T17:45:36.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:36.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj103 /dev/null 2021-01-15T17:45:36.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:36.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj104 /dev/null 2021-01-15T17:45:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj105 /dev/null 2021-01-15T17:45:37.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj106 /dev/null 2021-01-15T17:45:37.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj107 /dev/null 2021-01-15T17:45:37.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj108 /dev/null 2021-01-15T17:45:37.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj109 /dev/null 2021-01-15T17:45:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj110 /dev/null 2021-01-15T17:45:38.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj111 /dev/null 2021-01-15T17:45:38.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj112 /dev/null 2021-01-15T17:45:38.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj113 /dev/null 2021-01-15T17:45:38.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj114 /dev/null 2021-01-15T17:45:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj115 /dev/null 2021-01-15T17:45:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj116 /dev/null 2021-01-15T17:45:39.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj117 /dev/null 2021-01-15T17:45:39.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj118 /dev/null 2021-01-15T17:45:39.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj119 /dev/null 2021-01-15T17:45:39.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj120 /dev/null 2021-01-15T17:45:39.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj121 /dev/null 2021-01-15T17:45:39.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:39.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj122 /dev/null 2021-01-15T17:45:40.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:40.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj123 /dev/null 2021-01-15T17:45:40.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:40.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj124 /dev/null 2021-01-15T17:45:40.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:40.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj125 /dev/null 2021-01-15T17:45:40.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:40.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj126 /dev/null 2021-01-15T17:45:40.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:40.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj127 /dev/null 2021-01-15T17:45:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj128 /dev/null 2021-01-15T17:45:41.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj129 /dev/null 2021-01-15T17:45:41.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj130 /dev/null 2021-01-15T17:45:41.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj131 /dev/null 2021-01-15T17:45:41.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj132 /dev/null 2021-01-15T17:45:41.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:41.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj133 /dev/null 2021-01-15T17:45:42.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj134 /dev/null 2021-01-15T17:45:42.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj135 /dev/null 2021-01-15T17:45:42.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj136 /dev/null 2021-01-15T17:45:42.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj137 /dev/null 2021-01-15T17:45:42.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj138 /dev/null 2021-01-15T17:45:42.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:42.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj139 /dev/null 2021-01-15T17:45:43.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:43.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj140 /dev/null 2021-01-15T17:45:43.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:43.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj141 /dev/null 2021-01-15T17:45:43.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:43.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj142 /dev/null 2021-01-15T17:45:43.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:43.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj143 /dev/null 2021-01-15T17:45:43.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:43.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj144 /dev/null 2021-01-15T17:45:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj145 /dev/null 2021-01-15T17:45:44.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj146 /dev/null 2021-01-15T17:45:44.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj147 /dev/null 2021-01-15T17:45:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj148 /dev/null 2021-01-15T17:45:44.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj149 /dev/null 2021-01-15T17:45:44.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:44.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj150 /dev/null 2021-01-15T17:45:45.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:45.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj151 /dev/null 2021-01-15T17:45:45.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:45.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj152 /dev/null 2021-01-15T17:45:45.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:45.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj153 /dev/null 2021-01-15T17:45:45.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:45.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj154 /dev/null 2021-01-15T17:45:45.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:45.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj155 /dev/null 2021-01-15T17:45:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj156 /dev/null 2021-01-15T17:45:46.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj157 /dev/null 2021-01-15T17:45:46.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj158 /dev/null 2021-01-15T17:45:46.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj159 /dev/null 2021-01-15T17:45:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj160 /dev/null 2021-01-15T17:45:46.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:46.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj161 /dev/null 2021-01-15T17:45:47.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj162 /dev/null 2021-01-15T17:45:47.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj163 /dev/null 2021-01-15T17:45:47.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj164 /dev/null 2021-01-15T17:45:47.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj165 /dev/null 2021-01-15T17:45:47.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj166 /dev/null 2021-01-15T17:45:47.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:47.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj167 /dev/null 2021-01-15T17:45:48.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:48.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj168 /dev/null 2021-01-15T17:45:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:48.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj169 /dev/null 2021-01-15T17:45:48.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:48.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj170 /dev/null 2021-01-15T17:45:48.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj171 /dev/null 2021-01-15T17:45:48.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:48.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj172 /dev/null 2021-01-15T17:45:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj173 /dev/null 2021-01-15T17:45:49.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj174 /dev/null 2021-01-15T17:45:49.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj175 /dev/null 2021-01-15T17:45:49.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj176 /dev/null 2021-01-15T17:45:49.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj177 /dev/null 2021-01-15T17:45:49.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:49.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj178 /dev/null 2021-01-15T17:45:50.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj179 /dev/null 2021-01-15T17:45:50.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj180 /dev/null 2021-01-15T17:45:50.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj181 /dev/null 2021-01-15T17:45:50.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj182 /dev/null 2021-01-15T17:45:50.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj183 /dev/null 2021-01-15T17:45:50.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:50.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj184 /dev/null 2021-01-15T17:45:51.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:51.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj185 /dev/null 2021-01-15T17:45:51.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:51.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj186 /dev/null 2021-01-15T17:45:51.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:51.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj187 /dev/null 2021-01-15T17:45:51.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:51.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj188 /dev/null 2021-01-15T17:45:51.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:51.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj189 /dev/null 2021-01-15T17:45:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj190 /dev/null 2021-01-15T17:45:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj191 /dev/null 2021-01-15T17:45:52.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj192 /dev/null 2021-01-15T17:45:52.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj193 /dev/null 2021-01-15T17:45:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj194 /dev/null 2021-01-15T17:45:52.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:52.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj195 /dev/null 2021-01-15T17:45:53.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:53.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj196 /dev/null 2021-01-15T17:45:53.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:53.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj197 /dev/null 2021-01-15T17:45:53.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:53.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj198 /dev/null 2021-01-15T17:45:53.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:53.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj199 /dev/null 2021-01-15T17:45:53.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:636: TEST_backfill_ec_down_all_out: for i in $(seq 1 $objects) 2021-01-15T17:45:53.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:638: TEST_backfill_ec_down_all_out: rados -p test put obj200 /dev/null 2021-01-15T17:45:54.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:641: TEST_backfill_ec_down_all_out: get_pg test obj1 2021-01-15T17:45:54.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:45:54.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:45:54.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:45:54.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:45:54.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:641: TEST_backfill_ec_down_all_out: local PG=1.0 2021-01-15T17:45:54.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:643: TEST_backfill_ec_down_all_out: get_primary test obj1 2021-01-15T17:45:54.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:45:54.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:45:54.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:45:54.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:45:54.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:643: TEST_backfill_ec_down_all_out: local primary=1 2021-01-15T17:45:54.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:644: TEST_backfill_ec_down_all_out: get_not_primary test obj1 2021-01-15T17:45:54.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:45:54.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:45:54.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:45:54.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:45:54.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:45:54.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:45:54.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:45:55.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:45:55.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:45:55.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:45:55.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:644: TEST_backfill_ec_down_all_out: local otherosd=0 2021-01-15T17:45:55.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:645: TEST_backfill_ec_down_all_out: get_osds test obj1 2021-01-15T17:45:55.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2021-01-15T17:45:55.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2021-01-15T17:45:55.635 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2021-01-15T17:45:55.636 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T17:45:56.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2021-01-15T17:45:56.060 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T17:45:56.061 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T17:45:56.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 2021-01-15T17:45:56.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:645: TEST_backfill_ec_down_all_out: local 'allosds=1 0 2' 2021-01-15T17:45:56.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:647: TEST_backfill_ec_down_all_out: ceph osd set nobackfill 2021-01-15T17:45:56.751 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:45:56.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:648: TEST_backfill_ec_down_all_out: cat td/osd-backfill-stats/osd.0.pid 2021-01-15T17:45:56.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:648: TEST_backfill_ec_down_all_out: kill 546715 2021-01-15T17:45:56.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:649: TEST_backfill_ec_down_all_out: ceph osd down osd.0 2021-01-15T17:45:57.186 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T17:45:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2021-01-15T17:45:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.1 2021-01-15T17:45:57.839 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:45:57.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2021-01-15T17:45:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.0 2021-01-15T17:45:58.570 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:45:58.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:650: TEST_backfill_ec_down_all_out: for o in $allosds 2021-01-15T17:45:58.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:652: TEST_backfill_ec_down_all_out: ceph osd out osd.2 2021-01-15T17:45:59.171 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T17:45:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:655: TEST_backfill_ec_down_all_out: sleep 2 2021-01-15T17:46:01.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:656: TEST_backfill_ec_down_all_out: get_primary test obj1 2021-01-15T17:46:01.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:46:01.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:46:01.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:46:01.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:46:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:656: TEST_backfill_ec_down_all_out: primary=1 2021-01-15T17:46:01.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:657: TEST_backfill_ec_down_all_out: ceph osd unset nobackfill 2021-01-15T17:46:02.292 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:46:02.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:658: TEST_backfill_ec_down_all_out: ceph tell osd.1 get_latest_osdmap 2021-01-15T17:46:02.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:659: TEST_backfill_ec_down_all_out: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:46:02.573 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:46:02.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:660: TEST_backfill_ec_down_all_out: sleep 2 2021-01-15T17:46:04.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:661: TEST_backfill_ec_down_all_out: flush_pg_stats 2021-01-15T17:46:04.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:46:04.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:46:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:46:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:46:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:46:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:46:04.989 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:46:04.989 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:46:04.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:46:04.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:04.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:46:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T17:46:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:46:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:46:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:46:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:05.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:46:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2021-01-15T17:46:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2021-01-15T17:46:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982' 2021-01-15T17:46:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:05.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:46:05.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509460 2021-01-15T17:46:05.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509460 2021-01-15T17:46:05.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509460' 2021-01-15T17:46:05.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:05.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:46:05.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345940 2021-01-15T17:46:05.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345940 2021-01-15T17:46:05.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345940' 2021-01-15T17:46:05.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:05.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:46:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182417 2021-01-15T17:46:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182417 2021-01-15T17:46:05.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345940 4-107374182417' 2021-01-15T17:46:05.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:46:05.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:46:05.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018896 2021-01-15T17:46:05.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018896 2021-01-15T17:46:05.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509460 3-85899345940 4-107374182417 5-128849018896' 2021-01-15T17:46:05.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:46:05.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2021-01-15T17:46:05.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:46:05.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:46:05.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2021-01-15T17:46:05.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:46:05.782 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672982 2021-01-15T17:46:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2021-01-15T17:46:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2021-01-15T17:46:05.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:46:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672982 2021-01-15T17:46:06.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:46:07.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:46:07.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:46:07.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672982 -lt 42949672982 2021-01-15T17:46:07.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:46:07.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509460 2021-01-15T17:46:07.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:46:07.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:46:07.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509460 2021-01-15T17:46:07.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:46:07.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509460 2021-01-15T17:46:07.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2021-01-15T17:46:07.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:46:07.585 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509460 2021-01-15T17:46:07.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509461 -lt 64424509460 2021-01-15T17:46:07.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:46:07.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345940 2021-01-15T17:46:07.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:46:07.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:46:07.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345940 2021-01-15T17:46:07.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:46:07.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345940 2021-01-15T17:46:07.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2021-01-15T17:46:07.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:46:07.984 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345940 2021-01-15T17:46:08.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345940 -lt 85899345940 2021-01-15T17:46:08.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:46:08.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182417 2021-01-15T17:46:08.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:46:08.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:46:08.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182417 2021-01-15T17:46:08.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:46:08.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182417 2021-01-15T17:46:08.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2021-01-15T17:46:08.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:46:08.440 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182417 2021-01-15T17:46:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182418 -lt 107374182417 2021-01-15T17:46:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:46:08.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018896 2021-01-15T17:46:08.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:46:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:46:08.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018896 2021-01-15T17:46:08.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:46:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018896 2021-01-15T17:46:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018896' 2021-01-15T17:46:08.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:46:08.838 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018896 2021-01-15T17:46:09.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018896 -lt 128849018896 2021-01-15T17:46:09.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:666: TEST_backfill_ec_down_all_out: true 2021-01-15T17:46:09.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:669: TEST_backfill_ec_down_all_out: ceph --format json pg dump pgs 2021-01-15T17:46:09.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:670: TEST_backfill_ec_down_all_out: jq '.pg_stats | [.[] | .state | select(. == "incomplete")] | length' 2021-01-15T17:46:09.623 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:09.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:669: TEST_backfill_ec_down_all_out: test 0 -ne 0 2021-01-15T17:46:09.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:674: TEST_backfill_ec_down_all_out: break 2021-01-15T17:46:09.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:676: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:10.005 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:46:10.005 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 184 368 0 0 0 0 100 100 active+undersized+degraded+remapped+backfilling 2021-01-15T17:46:00.088241+0000 38'200 50:311 [4,3,5] 4 [1,NONE,2] 1 0'0 2021-01-15T17:45:10.390725+0000 0'0 2021-01-15T17:45:10.390725+0000 0 2021-01-15T17:46:10.006 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:46:10.006 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:46:10.007 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:10.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: seq 1 60 2021-01-15T17:46:10.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:10.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:10.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:10.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:10.392 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 1 = 60 ']' 2021-01-15T17:46:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:11.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:11.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:11.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:11.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:11.811 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:11.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 2 = 60 ']' 2021-01-15T17:46:11.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:12.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:12.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:12.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:12.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:13.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 3 = 60 ']' 2021-01-15T17:46:13.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:14.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:14.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:14.597 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:14.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 4 = 60 ']' 2021-01-15T17:46:14.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:15.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:15.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:15.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:15.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:15.988 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:15.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 5 = 60 ']' 2021-01-15T17:46:15.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:16.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:16.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:17.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:17.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:17.378 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:17.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 6 = 60 ']' 2021-01-15T17:46:17.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:18.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:18.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:18.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:18.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:18.765 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:18.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 7 = 60 ']' 2021-01-15T17:46:18.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:19.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:19.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:19.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:19.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:20.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 8 = 60 ']' 2021-01-15T17:46:20.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:21.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:21.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:21.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:21.553 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:21.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 9 = 60 ']' 2021-01-15T17:46:21.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:22.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:22.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:22.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:22.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:22.947 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:22.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 10 = 60 ']' 2021-01-15T17:46:22.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:23.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:23.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:23.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:23.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:24.339 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:24.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 11 = 60 ']' 2021-01-15T17:46:24.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:25.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:25.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:25.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:25.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:25.733 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:25.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 12 = 60 ']' 2021-01-15T17:46:25.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:26.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:26.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:26.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:26.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:27.131 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 13 = 60 ']' 2021-01-15T17:46:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:28.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:28.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:28.524 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 14 = 60 ']' 2021-01-15T17:46:28.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:29.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:29.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:29.917 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:29.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 15 = 60 ']' 2021-01-15T17:46:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:30.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:30.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:30.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:30.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:31.314 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:31.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 16 = 60 ']' 2021-01-15T17:46:31.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:32.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:32.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:32.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:32.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:32.712 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:32.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 17 = 60 ']' 2021-01-15T17:46:32.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:33.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:33.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:33.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:33.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:34.105 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 18 = 60 ']' 2021-01-15T17:46:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:35.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:35.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:35.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:35.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:35.496 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:35.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 19 = 60 ']' 2021-01-15T17:46:35.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:36.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:36.882 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:36.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 20 = 60 ']' 2021-01-15T17:46:36.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:37.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:38.269 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:38.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 21 = 60 ']' 2021-01-15T17:46:38.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:39.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:39.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:39.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:39.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:39.669 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:39.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 22 = 60 ']' 2021-01-15T17:46:39.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:40.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:41.056 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:41.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 23 = 60 ']' 2021-01-15T17:46:41.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:42.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:42.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:42.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:42.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:42.439 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:42.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 24 = 60 ']' 2021-01-15T17:46:42.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:43.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:43.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:43.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:43.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:43.832 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 25 = 60 ']' 2021-01-15T17:46:43.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:44.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:44.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:44.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:44.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:45.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 26 = 60 ']' 2021-01-15T17:46:45.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:46.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:46.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:46.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:46.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:46.614 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:46.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 27 = 60 ']' 2021-01-15T17:46:46.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:47.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:47.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:47.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:47.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:48.008 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:48.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 28 = 60 ']' 2021-01-15T17:46:48.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:49.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:49.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:49.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:49.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:49.399 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:49.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 29 = 60 ']' 2021-01-15T17:46:49.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:50.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:50.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:50.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:50.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:50.792 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:50.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 30 = 60 ']' 2021-01-15T17:46:50.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:51.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:51.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:52.216 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:52.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 31 = 60 ']' 2021-01-15T17:46:52.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:53.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:53.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:53.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:53.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:53.614 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:53.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 32 = 60 ']' 2021-01-15T17:46:53.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:54.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:54.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:54.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:54.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:55.006 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 33 = 60 ']' 2021-01-15T17:46:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:56.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:56.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:56.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:56.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:56.394 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:56.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 34 = 60 ']' 2021-01-15T17:46:56.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:57.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:57.793 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:57.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 35 = 60 ']' 2021-01-15T17:46:57.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:46:58.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:46:58.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:46:58.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:46:58.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:46:59.189 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:46:59.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 36 = 60 ']' 2021-01-15T17:46:59.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:47:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:47:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:47:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:47:00.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:47:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:47:00.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 37 = 60 ']' 2021-01-15T17:47:00.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:47:01.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:47:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:47:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:47:01.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:47:01.970 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:47:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:683: TEST_backfill_ec_down_all_out: '[' 38 = 60 ']' 2021-01-15T17:47:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:688: TEST_backfill_ec_down_all_out: sleep 1 2021-01-15T17:47:02.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:677: TEST_backfill_ec_down_all_out: for i in $(seq 1 60) 2021-01-15T17:47:02.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:47:02.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep '^1.0' 2021-01-15T17:47:02.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:679: TEST_backfill_ec_down_all_out: grep -qv backfilling 2021-01-15T17:47:03.362 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:47:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:681: TEST_backfill_ec_down_all_out: break 2021-01-15T17:47:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:691: TEST_backfill_ec_down_all_out: ceph pg dump pgs 2021-01-15T17:47:03.758 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T17:47:03.874 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 0 0 0 10 10 active+clean 2021-01-15T17:46:59.510907+0000 38'200 52:1055 [4,3,5] 4 [4,3,5] 4 0'0 2021-01-15T17:45:10.390725+0000 0'0 2021-01-15T17:45:10.390725+0000 0 2021-01-15T17:47:03.878 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T17:47:03.878 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T17:47:03.885 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T17:47:03.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:693: TEST_backfill_ec_down_all_out: expr 200 '*' 2 2021-01-15T17:47:03.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:693: TEST_backfill_ec_down_all_out: local misplaced=400 2021-01-15T17:47:03.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:694: TEST_backfill_ec_down_all_out: check td/osd-backfill-stats 1.0 1 erasure 200 0 400 0 2021-01-15T17:47:03.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:47:03.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:47:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:47:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2021-01-15T17:47:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2021-01-15T17:47:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:47:03.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2021-01-15T17:47:03.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:47:03.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:47:03.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:47:03.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:47:03.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:47:03.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T17:47:03.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:47:03.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:47:03.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:47:03.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2021-01-15T17:47:03.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2021-01-15T17:47:03.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:47:03.891 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.891 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:47:03.892 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:47:03.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,3,5]/[1,NONE,2]' 2021-01-15T17:47:03.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,3,5]/[1,NONE,2]' ']' 2021-01-15T17:47:03.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:47:03.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:47:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,3,5]/[1,NONE,2]p' 2021-01-15T17:47:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:47:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:47:03.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:47:03.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2021-01-15T17:47:03.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2021-01-15T17:47:03.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2021-01-15T17:47:03.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:47:03.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:47:03.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:47:03.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:47:03.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:47:03.896 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.896 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:47:03.896 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:47:03.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2021-01-15T17:47:03.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2021-01-15T17:47:03.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2021-01-15T17:47:03.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:47:03.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:47:03.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:47:03.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:47:03.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:47:03.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:47:03.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,3,5]/[1,NONE,2]p' 2021-01-15T17:47:03.899 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:47:03.899 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:47:03.899 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:47:03.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2021-01-15T17:47:03.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2021-01-15T17:47:03.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2021-01-15T17:47:03.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T17:47:03.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:47:03.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T17:47:03.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:47:03.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:47:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:47:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:47:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:47:03.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=2 2021-01-15T17:47:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 2 ']' 2021-01-15T17:47:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 2 0 2021-01-15T17:47:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2021-01-15T17:47:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:47:03.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:47:03.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:47:03.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:47:03.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:696: TEST_backfill_ec_down_all_out: delete_pool test 2021-01-15T17:47:03.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:47:03.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:47:04.379 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:47:04.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:697: TEST_backfill_ec_down_all_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:47:04.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:47:04.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:47:04.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:47:04.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:47:04.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:47:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:47:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:47:04.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:47:04.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:47:04.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:47:04.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:47:04.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:47:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:47:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:47:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:47:04.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:47:04.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:47:04.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:47:04.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:47:04.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:47:04.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:47:04.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:47:04.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:47:04.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:47:04.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:47:04.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:47:04.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:47:04.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:47:04.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:47:04.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:47:04.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:04.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:04.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:47:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:47:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:47:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:47:04.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:47:04.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:47:04.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:47:04.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:47:04.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:47:04.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:47:04.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:47:04.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:47:04.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:47:04.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:47:04.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:47:04.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:47:04.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:47:04.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:47:04.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:47:04.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:47:04.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:47:04.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:47:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:47:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:47:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:47:04.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:47:04.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:47:04.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:47:04.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:47:04.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:47:04.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:04.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:04.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:47:04.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:47:04.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:47:04.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:47:04.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:47:04.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:04.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:04.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:47:04.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:47:04.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:47:04.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:47:04.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:47:04.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_down_out td/osd-backfill-stats 2021-01-15T17:47:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:708: TEST_backfill_ec_down_out: local dir=td/osd-backfill-stats 2021-01-15T17:47:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:710: TEST_backfill_ec_down_out: run_mon td/osd-backfill-stats a 2021-01-15T17:47:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:47:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:47:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:47:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:47:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:47:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:47:04.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:47:04.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:04.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:04.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:04.839 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:04.839 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:04.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:04.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:47:05.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:47:05.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:47:05.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:47:05.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:47:05.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:47:05.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:47:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:47:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:47:05.011 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:47:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:47:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:05.013 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:47:05.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:47:05.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:47:05.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:47:05.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:47:05.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:47:05.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:47:05.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:47:05.110 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:47:05.111 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:47:05.111 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:47:05.111 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:47:05.112 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:05.112 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:05.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:47:05.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:47:05.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:47:05.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:711: TEST_backfill_ec_down_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:47:05.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:47:05.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:47:05.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:47:05.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:47:05.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:47:05.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:47:05.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:47:05.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:05.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:05.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:05.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:05.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:05.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:05.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:47:05.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:47:05.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:712: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:47:05.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:05.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:05.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:47:05.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:05.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:47:05.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:05.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:05.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:05.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:05.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:47:05.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:47:05.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:05.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:05.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:05.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:05.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:05.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:05.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:05.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:05.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:05.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:05.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:05.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:47:05.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:05.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5ded6053-b674-4fd7-88f7-1e523923872a 2021-01-15T17:47:05.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 5ded6053-b674-4fd7-88f7-1e523923872a' 2021-01-15T17:47:05.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:05.773 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 5ded6053-b674-4fd7-88f7-1e523923872a 2021-01-15T17:47:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAZ1QFgunWeLhAAtljfenXXpJVImpUzz8X6gA== 2021-01-15T17:47:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAZ1QFgunWeLhAAtljfenXXpJVImpUzz8X6gA=="}' 2021-01-15T17:47:05.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5ded6053-b674-4fd7-88f7-1e523923872a -i td/osd-backfill-stats/0/new.json 2021-01-15T17:47:06.124 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:06.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:47:06.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZ1QFgunWeLhAAtljfenXXpJVImpUzz8X6gA== --osd-uuid 5ded6053-b674-4fd7-88f7-1e523923872a 2021-01-15T17:47:06.160 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.159+0000 7f1378b70080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:06.162 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.161+0000 7f1378b70080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:06.163 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.162+0000 7f1378b70080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.163+0000 7f1378b70080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:47:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.163+0000 7f1378b70080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:47:06.164 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.163+0000 7f1378b70080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:47:06.224 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:06.223+0000 7f1378b70080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:47:10.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:47:10.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:10.782 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:47:10.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:47:10.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:47:12.136 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:47:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:12.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:12.162 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:12.162+0000 7f4b2966a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:12.164 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:12.164+0000 7f4b2966a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:12.167 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:12.167+0000 7f4b2966a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:12.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:47:12.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:12.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:47:12.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:12.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:12.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:12.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:12.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:12.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:47:12.537 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:12.684 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:12.684+0000 7f4b2966a080 -1 Falling back to public interface 2021-01-15T17:47:12.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:13.464 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:13.464+0000 7f4b2966a080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:47:13.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:13.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:13.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:13.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:13.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:47:13.945 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/561150,v1:127.0.0.1:6803/561150] [v2:127.0.0.1:6804/561150,v1:127.0.0.1:6805/561150] exists,up 5ded6053-b674-4fd7-88f7-1e523923872a 2021-01-15T17:47:14.369 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff 2021-01-15T17:47:14.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:14.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:14.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:14.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:713: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:47:14.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:14.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:14.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:47:14.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:14.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:47:14.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:14.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:14.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:14.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:14.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:47:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:47:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:14.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:14.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:14.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:14.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:14.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:14.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:14.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:14.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:14.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:14.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:14.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:14.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:14.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:14.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:14.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:14.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:14.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:14.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:14.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:47:14.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:14.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff 2021-01-15T17:47:14.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff' 2021-01-15T17:47:14.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:14.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAi1QFgph+bFhAAjkg7yZPP6YkWCbP6vByIgg== 2021-01-15T17:47:14.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAi1QFgph+bFhAAjkg7yZPP6YkWCbP6vByIgg=="}' 2021-01-15T17:47:14.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff -i td/osd-backfill-stats/1/new.json 2021-01-15T17:47:14.865 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:14.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:47:14.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAi1QFgph+bFhAAjkg7yZPP6YkWCbP6vByIgg== --osd-uuid fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff 2021-01-15T17:47:14.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.901+0000 7f5592bae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:14.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.903+0000 7f5592bae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:14.905 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.905+0000 7f5592bae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:14.906 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.905+0000 7f5592bae080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:47:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.905+0000 7f5592bae080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:47:14.907 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.905+0000 7f5592bae080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:47:14.950 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:14.950+0000 7f5592bae080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:47:19.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:47:19.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:19.399 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:47:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:47:19.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:19.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:47:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:19.947 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:47:19.968 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:19.968+0000 7f3b7e10f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:19.970+0000 7f3b7e10f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:19.971+0000 7f3b7e10f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:20.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:47:20.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:20.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:47:20.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:20.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:20.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:20.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:20.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:20.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:47:20.339 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:20.487 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:20.486+0000 7f3b7e10f080 -1 Falling back to public interface 2021-01-15T17:47:20.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:21.285 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:21.285+0000 7f3b7e10f080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:47:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:21.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:47:21.735 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:22.147 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/562414,v1:127.0.0.1:6811/562414] [v2:127.0.0.1:6812/562414,v1:127.0.0.1:6813/562414] exists,up fb08a2ed-298e-45d9-b61d-9b60c0f6c0ff 2021-01-15T17:47:22.147 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 0867417e-880a-466a-8910-e7a995f1195b 2021-01-15T17:47:22.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:714: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:47:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:22.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:47:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:47:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:22.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:47:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:47:22.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:22.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:22.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:22.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:22.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:22.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:22.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:22.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:22.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:22.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:22.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:22.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:22.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:22.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:22.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:22.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:22.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:22.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:47:22.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0867417e-880a-466a-8910-e7a995f1195b 2021-01-15T17:47:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 0867417e-880a-466a-8910-e7a995f1195b' 2021-01-15T17:47:22.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:22.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAq1QFgoBClCRAAjlhJBRLi70BPyk8k6RzIHg== 2021-01-15T17:47:22.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAq1QFgoBClCRAAjlhJBRLi70BPyk8k6RzIHg=="}' 2021-01-15T17:47:22.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0867417e-880a-466a-8910-e7a995f1195b -i td/osd-backfill-stats/2/new.json 2021-01-15T17:47:22.685 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:47:22.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:47:22.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAq1QFgoBClCRAAjlhJBRLi70BPyk8k6RzIHg== --osd-uuid 0867417e-880a-466a-8910-e7a995f1195b 2021-01-15T17:47:22.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.722+0000 7f6454ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:22.724 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.723+0000 7f6454ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:22.724 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.724+0000 7f6454ba8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:22.725 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.725+0000 7f6454ba8080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:47:22.726 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.725+0000 7f6454ba8080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:47:22.726 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.725+0000 7f6454ba8080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:47:22.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:22.774+0000 7f6454ba8080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:47:27.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:47:27.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:27.225 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:47:27.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:47:27.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:27.748 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:47:27.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:47:27.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:27.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:27.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:27.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:27.771 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:27.770+0000 7fc14117e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:27.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:27.772+0000 7fc14117e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:27.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:27.773+0000 7fc14117e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:47:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:47:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:47:28.142 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:28.293 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:28.293+0000 7fc14117e080 -1 Falling back to public interface 2021-01-15T17:47:28.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:29.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:29.082+0000 7fc14117e080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:47:29.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:29.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:29.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:29.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:29.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:47:29.539 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:29.968 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/563678,v1:127.0.0.1:6819/563678] [v2:127.0.0.1:6820/563678,v1:127.0.0.1:6821/563678] exists,up 0867417e-880a-466a-8910-e7a995f1195b 2021-01-15T17:47:29.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:29.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:29.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:29.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:715: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:47:29.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:29.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:29.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:47:29.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:29.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:47:29.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:29.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:29.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:29.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:29.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:47:29.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:47:29.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:29.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:29.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:29.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:29.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:29.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:29.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:29.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:47:29.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:29.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=45f0c9b6-7d74-420a-ba44-af956ae17c3b 2021-01-15T17:47:29.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 45f0c9b6-7d74-420a-ba44-af956ae17c3b' 2021-01-15T17:47:29.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:29.978 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 45f0c9b6-7d74-420a-ba44-af956ae17c3b 2021-01-15T17:47:29.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAx1QFgVrOuOhAAWenkfU3DS1vp4buRX2T1dg== 2021-01-15T17:47:29.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAx1QFgVrOuOhAAWenkfU3DS1vp4buRX2T1dg=="}' 2021-01-15T17:47:29.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 45f0c9b6-7d74-420a-ba44-af956ae17c3b -i td/osd-backfill-stats/3/new.json 2021-01-15T17:47:30.518 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:47:30.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:47:30.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAx1QFgVrOuOhAAWenkfU3DS1vp4buRX2T1dg== --osd-uuid 45f0c9b6-7d74-420a-ba44-af956ae17c3b 2021-01-15T17:47:30.555 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.554+0000 7f8b91444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:30.556 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.556+0000 7f8b91444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:30.557 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.557+0000 7f8b91444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.558+0000 7f8b91444080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:47:30.559 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.558+0000 7f8b91444080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:47:30.559 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.558+0000 7f8b91444080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:47:30.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:30.605+0000 7f8b91444080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:47:35.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:47:35.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:35.117 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:47:35.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:47:35.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:35.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:47:35.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:35.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:35.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:35.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:35.659 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:47:35.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:35.681+0000 7fd080747080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:35.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:35.698+0000 7fd080747080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:35.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:35.699+0000 7fd080747080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:47:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:47:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:36.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:47:36.055 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:36.215 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:36.215+0000 7fd080747080 -1 Falling back to public interface 2021-01-15T17:47:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:37.008 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:37.007+0000 7fd080747080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:47:37.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:37.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:37.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:37.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:37.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:47:37.452 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:37.877 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/564950,v1:127.0.0.1:6827/564950] [v2:127.0.0.1:6828/564950,v1:127.0.0.1:6829/564950] exists,up 45f0c9b6-7d74-420a-ba44-af956ae17c3b 2021-01-15T17:47:37.878 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 f4f60d4f-c010-4f60-ab4a-e661fd304ceb 2021-01-15T17:47:37.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:37.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:37.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:716: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:47:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:37.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:47:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:47:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:47:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:47:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:37.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:37.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:37.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:37.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:37.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:37.883 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:37.883 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:37.883 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:37.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:37.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:37.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:37.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:37.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:37.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:37.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:37.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:37.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:37.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:37.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:47:37.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f4f60d4f-c010-4f60-ab4a-e661fd304ceb 2021-01-15T17:47:37.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 f4f60d4f-c010-4f60-ab4a-e661fd304ceb' 2021-01-15T17:47:37.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA51QFg0rE1NRAA2C8TObO6+Eo/uhRH4XL5Dw== 2021-01-15T17:47:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA51QFg0rE1NRAA2C8TObO6+Eo/uhRH4XL5Dw=="}' 2021-01-15T17:47:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f4f60d4f-c010-4f60-ab4a-e661fd304ceb -i td/osd-backfill-stats/4/new.json 2021-01-15T17:47:38.560 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA51QFg0rE1NRAA2C8TObO6+Eo/uhRH4XL5Dw== --osd-uuid f4f60d4f-c010-4f60-ab4a-e661fd304ceb 2021-01-15T17:47:38.598 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.597+0000 7fcaf39ec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:38.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.599+0000 7fcaf39ec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.600+0000 7fcaf39ec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.600+0000 7fcaf39ec080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:47:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.601+0000 7fcaf39ec080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:47:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.601+0000 7fcaf39ec080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:47:38.632 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:38.631+0000 7fcaf39ec080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:47:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:47:43.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:43.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:47:43.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:43.344 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:47:43.872 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:47:43.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:47:43.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:43.895+0000 7f20b7869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:43.897 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:43.897+0000 7f20b7869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:43.898 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:43.898+0000 7f20b7869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:44.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:47:44.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:44.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:47:44.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:47:44.265 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:44.414 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:44.414+0000 7f20b7869080 -1 Falling back to public interface 2021-01-15T17:47:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:45.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:45.258+0000 7f20b7869080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:47:45.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:45.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:45.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:45.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:45.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:47:45.676 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:46.086 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/566214,v1:127.0.0.1:6835/566214] [v2:127.0.0.1:6836/566214,v1:127.0.0.1:6837/566214] exists,up f4f60d4f-c010-4f60-ab4a-e661fd304ceb 2021-01-15T17:47:46.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:46.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:46.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:46.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:717: TEST_backfill_ec_down_out: run_osd td/osd-backfill-stats 5 2021-01-15T17:47:46.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:47:46.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:47:46.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:47:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:47:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:47:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:47:46.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:47:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:47:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:47:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:47:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:47:46.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:47:46.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:47:46.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:47:46.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:47:46.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:47:46.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:47:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:47:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:47:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:47:46.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:47:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:47:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:47:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:47:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:47:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:47:46.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:47:46.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:47:46.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=678bc3d0-7597-4a01-9575-2d2ddf9f2694 2021-01-15T17:47:46.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 678bc3d0-7597-4a01-9575-2d2ddf9f2694' 2021-01-15T17:47:46.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:47:46.095 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 678bc3d0-7597-4a01-9575-2d2ddf9f2694 2021-01-15T17:47:46.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBC1QFgU7MVBhAAleSPSywKkSHlXSS7f0jtsA== 2021-01-15T17:47:46.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBC1QFgU7MVBhAAleSPSywKkSHlXSS7f0jtsA=="}' 2021-01-15T17:47:46.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 678bc3d0-7597-4a01-9575-2d2ddf9f2694 -i td/osd-backfill-stats/5/new.json 2021-01-15T17:47:46.660 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:47:46.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:47:46.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBC1QFgU7MVBhAAleSPSywKkSHlXSS7f0jtsA== --osd-uuid 678bc3d0-7597-4a01-9575-2d2ddf9f2694 2021-01-15T17:47:46.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.696+0000 7f55daf54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:46.716 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.716+0000 7f55daf54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:46.718 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.718+0000 7f55daf54080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:46.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.719+0000 7f55daf54080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:47:46.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.719+0000 7f55daf54080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:47:46.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.719+0000 7f55daf54080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:47:46.783 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:46.783+0000 7f55daf54080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:47:51.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:47:51.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:47:51.521 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:47:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:47:51.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:47:52.051 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:47:52.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:47:52.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:47:52.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:47:52.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:47:52.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:47:52.075 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:52.075+0000 7fa0b0d48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:52.077 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:52.077+0000 7fa0b0d48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:52.079 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:52.079+0000 7fa0b0d48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:47:52.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:47:52.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:47:52.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:47:52.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:47:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:47:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:47:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:47:52.471 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:47:52.596 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:52.596+0000 7fa0b0d48080 -1 Falling back to public interface 2021-01-15T17:47:52.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:47:53.421 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:47:53.420+0000 7fa0b0d48080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:47:53.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:47:53.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:47:53.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:47:53.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:47:53.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:47:53.876 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:47:54.309 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/567478,v1:127.0.0.1:6843/567478] [v2:127.0.0.1:6844/567478,v1:127.0.0.1:6845/567478] exists,up 678bc3d0-7597-4a01-9575-2d2ddf9f2694 2021-01-15T17:47:54.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:47:54.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:47:54.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:47:54.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:719: TEST_backfill_ec_down_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T17:47:54.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:720: TEST_backfill_ec_down_out: create_pool test 1 1 erasure myprofile 2021-01-15T17:47:54.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T17:47:55.537 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:47:55.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:47:56.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:721: TEST_backfill_ec_down_out: ceph osd pool set test min_size 2 2021-01-15T17:47:57.493 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T17:47:57.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:723: TEST_backfill_ec_down_out: wait_for_clean 2021-01-15T17:47:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:47:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:47:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:47:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:47:57.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:47:57.504 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:47:57.505 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:47:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:47:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:47:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:47:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:47:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:47:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:47:57.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:47:57.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:47:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:47:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:47:58.015 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:47:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:47:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:47:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:47:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:47:58.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:47:58.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T17:47:58.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T17:47:58.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T17:47:58.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:47:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T17:47:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T17:47:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-01-15T17:47:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:47:58.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2021-01-15T17:47:58.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2021-01-15T17:47:58.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2021-01-15T17:47:58.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:47:58.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:47:58.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:47:58.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345927' 2021-01-15T17:47:58.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:47:58.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T17:47:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T17:47:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345927 4-107374182405' 2021-01-15T17:47:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:47:58.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:47:58.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T17:47:58.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T17:47:58.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345927 4-107374182405 5-128849018884' 2021-01-15T17:47:58.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:47:58.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T17:47:58.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:47:58.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:47:58.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T17:47:58.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:47:58.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T17:47:58.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T17:47:58.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:47:58.845 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T17:47:59.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836491 2021-01-15T17:47:59.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:48:00.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:48:00.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:48:00.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2021-01-15T17:48:00.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:00.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T17:48:00.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:00.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:48:00.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T17:48:00.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T17:48:00.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T17:48:00.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:48:00.646 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T17:48:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T17:48:01.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:01.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2021-01-15T17:48:01.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:48:01.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-01-15T17:48:01.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2021-01-15T17:48:01.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-01-15T17:48:01.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:48:01.034 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2021-01-15T17:48:01.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509449 -lt 64424509449 2021-01-15T17:48:01.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:01.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:48:01.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:01.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:48:01.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:48:01.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:01.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:48:01.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:48:01.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:48:01.436 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:48:01.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345927 2021-01-15T17:48:01.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:01.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T17:48:01.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:01.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:48:01.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T17:48:01.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:01.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T17:48:01.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T17:48:01.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:48:01.831 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T17:48:02.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T17:48:02.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:02.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T17:48:02.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:02.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:48:02.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T17:48:02.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:02.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T17:48:02.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T17:48:02.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:48:02.227 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T17:48:02.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2021-01-15T17:48:02.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:48:02.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:02.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:03.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:48:03.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:48:03.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:48:03.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:48:03.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:48:03.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:48:03.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:48:03.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:48:03.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:48:03.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:48:03.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:03.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:04.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:48:04.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:48:04.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:48:04.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: seq 1 200 2021-01-15T17:48:04.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj1 /dev/null 2021-01-15T17:48:04.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj2 /dev/null 2021-01-15T17:48:04.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj3 /dev/null 2021-01-15T17:48:04.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj4 /dev/null 2021-01-15T17:48:04.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj5 /dev/null 2021-01-15T17:48:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj6 /dev/null 2021-01-15T17:48:05.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:05.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj7 /dev/null 2021-01-15T17:48:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:05.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj8 /dev/null 2021-01-15T17:48:05.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:05.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj9 /dev/null 2021-01-15T17:48:05.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:05.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj10 /dev/null 2021-01-15T17:48:05.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:05.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj11 /dev/null 2021-01-15T17:48:06.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj12 /dev/null 2021-01-15T17:48:06.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj13 /dev/null 2021-01-15T17:48:06.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj14 /dev/null 2021-01-15T17:48:06.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj15 /dev/null 2021-01-15T17:48:06.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj16 /dev/null 2021-01-15T17:48:06.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:06.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj17 /dev/null 2021-01-15T17:48:07.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:07.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj18 /dev/null 2021-01-15T17:48:07.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:07.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj19 /dev/null 2021-01-15T17:48:07.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:07.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj20 /dev/null 2021-01-15T17:48:07.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:07.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj21 /dev/null 2021-01-15T17:48:07.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:07.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj22 /dev/null 2021-01-15T17:48:08.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:08.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj23 /dev/null 2021-01-15T17:48:08.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:08.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj24 /dev/null 2021-01-15T17:48:08.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:08.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj25 /dev/null 2021-01-15T17:48:08.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:08.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj26 /dev/null 2021-01-15T17:48:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj27 /dev/null 2021-01-15T17:48:09.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:09.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj28 /dev/null 2021-01-15T17:48:09.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:09.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj29 /dev/null 2021-01-15T17:48:09.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:09.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj30 /dev/null 2021-01-15T17:48:09.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:09.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj31 /dev/null 2021-01-15T17:48:09.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:09.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj32 /dev/null 2021-01-15T17:48:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj33 /dev/null 2021-01-15T17:48:10.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj34 /dev/null 2021-01-15T17:48:10.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj35 /dev/null 2021-01-15T17:48:10.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj36 /dev/null 2021-01-15T17:48:10.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj37 /dev/null 2021-01-15T17:48:10.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:10.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj38 /dev/null 2021-01-15T17:48:11.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:11.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj39 /dev/null 2021-01-15T17:48:11.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:11.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj40 /dev/null 2021-01-15T17:48:11.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:11.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj41 /dev/null 2021-01-15T17:48:11.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:11.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj42 /dev/null 2021-01-15T17:48:11.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:11.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj43 /dev/null 2021-01-15T17:48:12.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:12.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj44 /dev/null 2021-01-15T17:48:12.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:12.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj45 /dev/null 2021-01-15T17:48:12.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:12.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj46 /dev/null 2021-01-15T17:48:12.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:12.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj47 /dev/null 2021-01-15T17:48:12.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:12.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj48 /dev/null 2021-01-15T17:48:13.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj49 /dev/null 2021-01-15T17:48:13.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj50 /dev/null 2021-01-15T17:48:13.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj51 /dev/null 2021-01-15T17:48:13.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj52 /dev/null 2021-01-15T17:48:13.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj53 /dev/null 2021-01-15T17:48:13.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:13.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj54 /dev/null 2021-01-15T17:48:14.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:14.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj55 /dev/null 2021-01-15T17:48:14.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:14.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj56 /dev/null 2021-01-15T17:48:14.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:14.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj57 /dev/null 2021-01-15T17:48:14.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:14.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj58 /dev/null 2021-01-15T17:48:14.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:14.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj59 /dev/null 2021-01-15T17:48:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:15.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj60 /dev/null 2021-01-15T17:48:15.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:15.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj61 /dev/null 2021-01-15T17:48:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj62 /dev/null 2021-01-15T17:48:15.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:15.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj63 /dev/null 2021-01-15T17:48:15.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:15.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj64 /dev/null 2021-01-15T17:48:16.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj65 /dev/null 2021-01-15T17:48:16.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj66 /dev/null 2021-01-15T17:48:16.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj67 /dev/null 2021-01-15T17:48:16.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj68 /dev/null 2021-01-15T17:48:16.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj69 /dev/null 2021-01-15T17:48:16.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:16.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj70 /dev/null 2021-01-15T17:48:17.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:17.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj71 /dev/null 2021-01-15T17:48:17.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:17.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj72 /dev/null 2021-01-15T17:48:17.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:17.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj73 /dev/null 2021-01-15T17:48:17.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:17.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj74 /dev/null 2021-01-15T17:48:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj75 /dev/null 2021-01-15T17:48:18.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj76 /dev/null 2021-01-15T17:48:18.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj77 /dev/null 2021-01-15T17:48:18.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj78 /dev/null 2021-01-15T17:48:18.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj79 /dev/null 2021-01-15T17:48:18.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj80 /dev/null 2021-01-15T17:48:18.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:18.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj81 /dev/null 2021-01-15T17:48:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:19.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj82 /dev/null 2021-01-15T17:48:19.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:19.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj83 /dev/null 2021-01-15T17:48:19.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:19.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj84 /dev/null 2021-01-15T17:48:19.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:19.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj85 /dev/null 2021-01-15T17:48:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:20.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj86 /dev/null 2021-01-15T17:48:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj87 /dev/null 2021-01-15T17:48:20.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:20.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj88 /dev/null 2021-01-15T17:48:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj89 /dev/null 2021-01-15T17:48:20.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:20.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj90 /dev/null 2021-01-15T17:48:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:21.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj91 /dev/null 2021-01-15T17:48:21.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:21.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj92 /dev/null 2021-01-15T17:48:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj93 /dev/null 2021-01-15T17:48:21.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:21.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj94 /dev/null 2021-01-15T17:48:21.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:21.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj95 /dev/null 2021-01-15T17:48:22.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj96 /dev/null 2021-01-15T17:48:22.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj97 /dev/null 2021-01-15T17:48:22.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj98 /dev/null 2021-01-15T17:48:22.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj99 /dev/null 2021-01-15T17:48:22.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj100 /dev/null 2021-01-15T17:48:22.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:22.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj101 /dev/null 2021-01-15T17:48:23.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:23.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj102 /dev/null 2021-01-15T17:48:23.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:23.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj103 /dev/null 2021-01-15T17:48:23.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:23.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj104 /dev/null 2021-01-15T17:48:23.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:23.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj105 /dev/null 2021-01-15T17:48:23.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:23.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj106 /dev/null 2021-01-15T17:48:24.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:24.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj107 /dev/null 2021-01-15T17:48:24.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:24.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj108 /dev/null 2021-01-15T17:48:24.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:24.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj109 /dev/null 2021-01-15T17:48:24.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:24.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj110 /dev/null 2021-01-15T17:48:24.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:24.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj111 /dev/null 2021-01-15T17:48:25.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:25.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj112 /dev/null 2021-01-15T17:48:25.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:25.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj113 /dev/null 2021-01-15T17:48:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:25.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj114 /dev/null 2021-01-15T17:48:25.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:25.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj115 /dev/null 2021-01-15T17:48:25.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:25.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj116 /dev/null 2021-01-15T17:48:26.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:26.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj117 /dev/null 2021-01-15T17:48:26.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:26.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj118 /dev/null 2021-01-15T17:48:26.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:26.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj119 /dev/null 2021-01-15T17:48:26.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:26.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj120 /dev/null 2021-01-15T17:48:26.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:26.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj121 /dev/null 2021-01-15T17:48:27.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj122 /dev/null 2021-01-15T17:48:27.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj123 /dev/null 2021-01-15T17:48:27.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj124 /dev/null 2021-01-15T17:48:27.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj125 /dev/null 2021-01-15T17:48:27.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj126 /dev/null 2021-01-15T17:48:27.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:27.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj127 /dev/null 2021-01-15T17:48:28.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj128 /dev/null 2021-01-15T17:48:28.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:28.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj129 /dev/null 2021-01-15T17:48:28.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:28.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj130 /dev/null 2021-01-15T17:48:28.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:28.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj131 /dev/null 2021-01-15T17:48:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj132 /dev/null 2021-01-15T17:48:29.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:29.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj133 /dev/null 2021-01-15T17:48:29.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:29.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj134 /dev/null 2021-01-15T17:48:29.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:29.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj135 /dev/null 2021-01-15T17:48:29.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:29.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj136 /dev/null 2021-01-15T17:48:29.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:29.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj137 /dev/null 2021-01-15T17:48:30.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj138 /dev/null 2021-01-15T17:48:30.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj139 /dev/null 2021-01-15T17:48:30.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj140 /dev/null 2021-01-15T17:48:30.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj141 /dev/null 2021-01-15T17:48:30.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj142 /dev/null 2021-01-15T17:48:30.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:30.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj143 /dev/null 2021-01-15T17:48:31.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:31.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj144 /dev/null 2021-01-15T17:48:31.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj145 /dev/null 2021-01-15T17:48:31.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:31.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj146 /dev/null 2021-01-15T17:48:31.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:31.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj147 /dev/null 2021-01-15T17:48:31.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:31.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj148 /dev/null 2021-01-15T17:48:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj149 /dev/null 2021-01-15T17:48:32.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj150 /dev/null 2021-01-15T17:48:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj151 /dev/null 2021-01-15T17:48:32.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj152 /dev/null 2021-01-15T17:48:32.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj153 /dev/null 2021-01-15T17:48:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:32.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj154 /dev/null 2021-01-15T17:48:33.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:33.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj155 /dev/null 2021-01-15T17:48:33.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:33.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj156 /dev/null 2021-01-15T17:48:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:33.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj157 /dev/null 2021-01-15T17:48:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj158 /dev/null 2021-01-15T17:48:33.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:33.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj159 /dev/null 2021-01-15T17:48:34.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:34.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj160 /dev/null 2021-01-15T17:48:34.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:34.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj161 /dev/null 2021-01-15T17:48:34.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:34.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj162 /dev/null 2021-01-15T17:48:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj163 /dev/null 2021-01-15T17:48:34.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:34.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj164 /dev/null 2021-01-15T17:48:35.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj165 /dev/null 2021-01-15T17:48:35.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj166 /dev/null 2021-01-15T17:48:35.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj167 /dev/null 2021-01-15T17:48:35.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj168 /dev/null 2021-01-15T17:48:35.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj169 /dev/null 2021-01-15T17:48:35.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:35.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj170 /dev/null 2021-01-15T17:48:36.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:36.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj171 /dev/null 2021-01-15T17:48:36.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:36.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj172 /dev/null 2021-01-15T17:48:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj173 /dev/null 2021-01-15T17:48:36.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj174 /dev/null 2021-01-15T17:48:36.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:36.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj175 /dev/null 2021-01-15T17:48:37.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:37.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj176 /dev/null 2021-01-15T17:48:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj177 /dev/null 2021-01-15T17:48:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:37.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj178 /dev/null 2021-01-15T17:48:37.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:37.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj179 /dev/null 2021-01-15T17:48:37.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:37.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj180 /dev/null 2021-01-15T17:48:38.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:38.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj181 /dev/null 2021-01-15T17:48:38.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:38.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj182 /dev/null 2021-01-15T17:48:38.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:38.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj183 /dev/null 2021-01-15T17:48:38.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj184 /dev/null 2021-01-15T17:48:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:38.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj185 /dev/null 2021-01-15T17:48:39.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:39.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj186 /dev/null 2021-01-15T17:48:39.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:39.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj187 /dev/null 2021-01-15T17:48:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:39.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj188 /dev/null 2021-01-15T17:48:39.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:39.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj189 /dev/null 2021-01-15T17:48:40.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj190 /dev/null 2021-01-15T17:48:40.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj191 /dev/null 2021-01-15T17:48:40.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj192 /dev/null 2021-01-15T17:48:40.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj193 /dev/null 2021-01-15T17:48:40.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj194 /dev/null 2021-01-15T17:48:40.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:40.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj195 /dev/null 2021-01-15T17:48:41.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:41.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj196 /dev/null 2021-01-15T17:48:41.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:41.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj197 /dev/null 2021-01-15T17:48:41.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:41.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj198 /dev/null 2021-01-15T17:48:41.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:41.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj199 /dev/null 2021-01-15T17:48:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:725: TEST_backfill_ec_down_out: for i in $(seq 1 $objects) 2021-01-15T17:48:41.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:727: TEST_backfill_ec_down_out: rados -p test put obj200 /dev/null 2021-01-15T17:48:42.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:730: TEST_backfill_ec_down_out: get_pg test obj1 2021-01-15T17:48:42.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:48:42.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:48:42.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:48:42.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:48:42.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:730: TEST_backfill_ec_down_out: local PG=1.0 2021-01-15T17:48:42.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:732: TEST_backfill_ec_down_out: get_primary test obj1 2021-01-15T17:48:42.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:48:42.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:48:42.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:48:42.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:48:42.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:732: TEST_backfill_ec_down_out: local primary=1 2021-01-15T17:48:42.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:733: TEST_backfill_ec_down_out: get_not_primary test obj1 2021-01-15T17:48:42.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:48:42.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:48:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:48:42.937 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:48:42.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:48:42.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:48:42.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:48:43.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:48:43.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:48:43.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:48:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:733: TEST_backfill_ec_down_out: local otherosd=0 2021-01-15T17:48:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:735: TEST_backfill_ec_down_out: ceph osd set nobackfill 2021-01-15T17:48:44.456 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:48:44.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:736: TEST_backfill_ec_down_out: cat td/osd-backfill-stats/osd.0.pid 2021-01-15T17:48:44.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:736: TEST_backfill_ec_down_out: kill 561150 2021-01-15T17:48:44.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:737: TEST_backfill_ec_down_out: ceph osd down osd.0 2021-01-15T17:48:44.959 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T17:48:44.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:738: TEST_backfill_ec_down_out: ceph osd out osd.0 2021-01-15T17:48:45.688 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:48:45.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:740: TEST_backfill_ec_down_out: sleep 2 2021-01-15T17:48:47.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:741: TEST_backfill_ec_down_out: get_primary test obj1 2021-01-15T17:48:47.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:48:47.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:48:47.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:48:47.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:48:48.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:741: TEST_backfill_ec_down_out: primary=1 2021-01-15T17:48:48.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:742: TEST_backfill_ec_down_out: ceph osd unset nobackfill 2021-01-15T17:48:48.806 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:48:48.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:743: TEST_backfill_ec_down_out: ceph tell osd.1 get_latest_osdmap 2021-01-15T17:48:48.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:744: TEST_backfill_ec_down_out: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:48:49.084 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:48:49.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:745: TEST_backfill_ec_down_out: sleep 2 2021-01-15T17:48:51.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:747: TEST_backfill_ec_down_out: wait_for_clean 2021-01-15T17:48:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:48:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:48:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:48:51.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:48:51.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:48:51.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:48:51.097 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:48:51.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:48:51.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:48:51.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:48:51.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:48:51.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:48:51.595 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:48:51.595 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:48:51.595 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:48:51.595 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:48:51.595 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:48:51.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:48:51.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:51.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:48:51.740 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T17:48:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T17:48:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T17:48:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T17:48:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:51.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:48:51.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672982 2021-01-15T17:48:51.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672982 2021-01-15T17:48:51.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982' 2021-01-15T17:48:51.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:51.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:48:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509462 2021-01-15T17:48:52.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509462 2021-01-15T17:48:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509462' 2021-01-15T17:48:52.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:52.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:48:52.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345939 2021-01-15T17:48:52.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345939 2021-01-15T17:48:52.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509462 3-85899345939' 2021-01-15T17:48:52.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:52.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:48:52.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182418 2021-01-15T17:48:52.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182418 2021-01-15T17:48:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509462 3-85899345939 4-107374182418' 2021-01-15T17:48:52.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:48:52.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:48:52.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018897 2021-01-15T17:48:52.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018897 2021-01-15T17:48:52.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672982 2-64424509462 3-85899345939 4-107374182418 5-128849018897' 2021-01-15T17:48:52.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:52.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672982 2021-01-15T17:48:52.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:52.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:48:52.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672982 2021-01-15T17:48:52.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:52.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672982 2021-01-15T17:48:52.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2021-01-15T17:48:52.434 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672982 2021-01-15T17:48:52.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:48:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672982 2021-01-15T17:48:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:48:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:48:53.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:48:54.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672983 -lt 42949672982 2021-01-15T17:48:54.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:54.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509462 2021-01-15T17:48:54.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:54.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:48:54.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509462 2021-01-15T17:48:54.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:54.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509462 2021-01-15T17:48:54.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2021-01-15T17:48:54.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:48:54.237 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509462 2021-01-15T17:48:54.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509462 -lt 64424509462 2021-01-15T17:48:54.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:54.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345939 2021-01-15T17:48:54.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:54.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:48:54.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345939 2021-01-15T17:48:54.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:54.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345939 2021-01-15T17:48:54.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2021-01-15T17:48:54.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:48:54.633 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345939 2021-01-15T17:48:55.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345940 -lt 85899345939 2021-01-15T17:48:55.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:55.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182418 2021-01-15T17:48:55.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:55.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:48:55.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182418 2021-01-15T17:48:55.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:55.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182418 2021-01-15T17:48:55.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182418' 2021-01-15T17:48:55.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:48:55.029 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182418 2021-01-15T17:48:55.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182418 -lt 107374182418 2021-01-15T17:48:55.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:48:55.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018897 2021-01-15T17:48:55.426 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:48:55.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:48:55.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018897 2021-01-15T17:48:55.427 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:48:55.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018897 2021-01-15T17:48:55.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2021-01-15T17:48:55.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:48:55.428 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018897 2021-01-15T17:48:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018898 -lt 128849018897 2021-01-15T17:48:55.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:48:55.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:55.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:48:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:48:56.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:48:56.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:48:56.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:48:56.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:48:56.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:48:56.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:48:56.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:48:56.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:48:56.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:56.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:57.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:48:57.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:48:57.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:48:57.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:48:57.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:48:57.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:48:57.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:48:57.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:48:57.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:48:57.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:48:57.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:48:57.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:48:57.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:48:57.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:48:57.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:48:57.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:48:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:58.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:48:58.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:48:58.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:48:58.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:48:58.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:48:58.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:48:58.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:48:58.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:48:58.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:48:58.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2021-01-15T17:48:58.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2021-01-15T17:48:58.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:48:58.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:48:58.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:48:58.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:48:58.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:48:58.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:48:58.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:48:58.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:48:58.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:48:58.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:48:58.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:48:59.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:48:59.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:48:59.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:48:59.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:48:59.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:48:59.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:48:59.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:48:59.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:48:59.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:48:59.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:48:59.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:48:59.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:48:59.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:00.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:00.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:00.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:00.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:00.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:00.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:00.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:00.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:00.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:00.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:00.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:00.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:00.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:01.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:01.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:01.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:01.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:01.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:49:01.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:49:01.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:01.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:01.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:01.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:01.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:01.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:01.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:01.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:01.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:01.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:01.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:02.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:02.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:02.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:02.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:02.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:02.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:02.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:02.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:02.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:02.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:02.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:02.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:03.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:49:03.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:49:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:03.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:03.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:03.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:03.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:03.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:03.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:03.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:03.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:03.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:03.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:03.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:03.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:04.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:04.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:04.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:04.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:04.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:49:04.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:49:04.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:04.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:04.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:04.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:04.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:04.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:04.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:04.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:04.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:04.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:05.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:05.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:05.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:05.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:05.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:05.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:05.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:05.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:05.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:05.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:05.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:05.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:05.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:06.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:06.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:06.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:06.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:06.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:06.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:06.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:06.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:06.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:06.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:06.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:06.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:06.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:06.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:07.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:07.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:07.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:07.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:07.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:07.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:07.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:07.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:07.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:07.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:08.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:08.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:08.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:08.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:08.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:08.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:08.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:08.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:08.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:08.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:08.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:08.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:08.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:08.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:09.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:09.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:09.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:09.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:09.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:09.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:09.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:09.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:09.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:09.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:09.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:09.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:09.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:10.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:10.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:10.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:10.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:10.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:10.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:10.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:10.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:10.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:11.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:11.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:11.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:11.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:11.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:11.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:11.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:11.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:11.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:11.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:11.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:11.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:11.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:11.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:11.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:11.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:11.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:12.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:12.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:12.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:12.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:12.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:12.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:12.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:12.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:12.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:12.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:12.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:12.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:12.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:12.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:12.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:12.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:12.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:12.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:13.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:13.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:13.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:13.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:13.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:13.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:13.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:13.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:13.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:13.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:13.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:14.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:14.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:14.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:14.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:14.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:14.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:14.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:14.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:14.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:14.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:14.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:14.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:14.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:14.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:14.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:14.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:15.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:15.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:15.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:15.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:15.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:15.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:15.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:15.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:15.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:15.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:15.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:15.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:15.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:15.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:16.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:16.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:16.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:16.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:16.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:16.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:16.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:16.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:17.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:17.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:17.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:17.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:17.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:17.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:17.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:17.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:17.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:17.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:17.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:17.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:17.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:17.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:18.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:18.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:18.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:18.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:18.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:18.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:18.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:18.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:18.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:18.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:18.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:18.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:18.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:18.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:18.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:18.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:18.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:18.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:19.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:19.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:19.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:19.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:19.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:19.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:19.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:19.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:19.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:19.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:20.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:20.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:20.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:20.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:20.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:20.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:20.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:20.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:20.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:20.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:20.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:20.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:20.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:20.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:20.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:20.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:20.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:21.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:21.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:22.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:22.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:22.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:22.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:22.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:22.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:22.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:22.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:22.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:22.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:22.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:22.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:22.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:22.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:22.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:22.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:22.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:22.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:23.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:23.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:23.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:23.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:23.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:23.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:23.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:23.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:23.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:23.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:23.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:23.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:23.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:23.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:24.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:24.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:24.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:24.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:24.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:24.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:24.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:24.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:24.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:24.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:24.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:24.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:24.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:24.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:25.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:25.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:25.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:25.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:25.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:25.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:25.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:25.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:25.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:25.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:25.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:25.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:25.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:25.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:25.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:25.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:25.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:25.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:25.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:25.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:25.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:26.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:26.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:26.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:26.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:26.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:26.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:26.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:26.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:26.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:26.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:26.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:26.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:26.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:26.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:26.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:27.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:27.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:27.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:27.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:27.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:27.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:27.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:27.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:27.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:28.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:28.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:28.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:28.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:28.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:28.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:28.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:28.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:28.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:29.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:29.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:29.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:29.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:29.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:29.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:29.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:29.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:29.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:29.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:49:29.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:49:29.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:29.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:29.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:29.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:29.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:29.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:29.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:29.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:29.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:29.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:29.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:30.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:30.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:30.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:30.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:30.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:30.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:30.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:30.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:30.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:30.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:31.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:49:31.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:49:31.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:31.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:31.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:31.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:31.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:31.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:31.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:31.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:31.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:31.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:31.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:31.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:31.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:32.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:32.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:32.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:32.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:49:32.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:49:32.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:32.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:32.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:32.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:32.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:32.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:32.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:32.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:32.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:32.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:32.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:33.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:49:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:33.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:49:33.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:49:33.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:49:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:49:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:49:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:49:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:49:33.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:49:33.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:49:34.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:49:34.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:49:34.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:49:34.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:49:34.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:49:34.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:49:34.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:49:34.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:49:34.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:49:34.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:49:34.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:49:34.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:49:34.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:49:34.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:49:34.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:49:34.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:49:34.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:49:35.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:49:35.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:49:35.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:49:35.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:749: TEST_backfill_ec_down_out: expr 200 '*' 2 2021-01-15T17:49:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:749: TEST_backfill_ec_down_out: local misplaced=400 2021-01-15T17:49:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:750: TEST_backfill_ec_down_out: check td/osd-backfill-stats 1.0 1 erasure 200 0 0 0 2021-01-15T17:49:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:49:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:49:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2021-01-15T17:49:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2021-01-15T17:49:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:49:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2021-01-15T17:49:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:49:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:49:35.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:49:35.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:49:35.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:49:35.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T17:49:35.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:49:35.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:49:35.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:49:35.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2021-01-15T17:49:35.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2021-01-15T17:49:35.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:49:35.127 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.128 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:49:35.128 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:49:35.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,3,2]/[1,NONE,2]' 2021-01-15T17:49:35.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,3,2]/[1,NONE,2]' ']' 2021-01-15T17:49:35.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:49:35.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:49:35.144 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.144 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-01-15T17:49:35.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:49:35.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:49:35.146 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:49:35.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2021-01-15T17:49:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2021-01-15T17:49:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2021-01-15T17:49:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:49:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:49:35.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:49:35.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:49:35.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:49:35.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:49:35.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:49:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2021-01-15T17:49:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2021-01-15T17:49:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2021-01-15T17:49:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:49:35.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:49:35.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:49:35.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:49:35.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:49:35.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:49:35.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-01-15T17:49:35.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:49:35.166 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:49:35.166 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:49:35.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2021-01-15T17:49:35.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2021-01-15T17:49:35.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2021-01-15T17:49:35.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:49:35.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:49:35.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:49:35.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:49:35.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:49:35.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:49:35.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:49:35.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:49:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2021-01-15T17:49:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2021-01-15T17:49:35.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2021-01-15T17:49:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:49:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:49:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:49:35.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:49:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:49:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:752: TEST_backfill_ec_down_out: delete_pool test 2021-01-15T17:49:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:49:35.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:49:35.694 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:49:35.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:753: TEST_backfill_ec_down_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:49:35.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:49:35.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:49:35.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:49:35.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:49:35.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:49:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:49:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:49:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:49:35.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:49:35.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:49:35.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:49:35.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:49:35.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:49:35.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:49:35.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:49:35.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:49:35.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:49:35.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:49:35.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:49:35.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:49:35.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:49:35.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:49:35.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:49:35.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:49:35.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:49:35.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:49:35.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:49:35.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:49:35.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:49:35.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:35.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:35.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:49:35.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:49:35.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:49:35.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:49:35.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:49:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:49:35.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:49:35.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:49:35.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:49:35.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:49:35.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:49:35.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:49:35.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:49:35.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:49:35.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:49:35.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:49:35.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:49:35.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:49:35.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:49:35.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:49:35.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:49:35.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:49:35.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:49:35.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:49:35.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:49:35.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:49:35.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:49:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:49:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:35.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:49:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:49:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:49:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:49:35.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:49:35.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:35.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:35.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:49:35.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:49:35.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:49:35.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:49:35.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:49:35.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_ec_prim_out td/osd-backfill-stats 2021-01-15T17:49:35.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:569: TEST_backfill_ec_prim_out: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:571: TEST_backfill_ec_prim_out: run_mon td/osd-backfill-stats a 2021-01-15T17:49:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:49:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:49:35.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:49:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:49:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:49:35.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:49:36.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:49:36.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:49:36.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:49:36.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:49:36.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:36.147 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:36.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:36.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:49:36.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:49:36.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:49:36.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:49:36.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:49:36.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:49:36.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:49:36.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:49:36.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:49:36.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:49:36.318 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:49:36.318 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:36.319 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:36.319 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:49:36.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:49:36.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:49:36.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:49:36.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:49:36.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:49:36.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:49:36.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:49:36.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:49:36.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:49:36.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:49:36.421 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:49:36.421 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:36.421 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:36.422 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:49:36.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:49:36.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:49:36.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:572: TEST_backfill_ec_prim_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:49:36.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:49:36.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:49:36.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:49:36.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:49:36.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:49:36.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:49:37.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:49:37.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:49:37.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:49:37.016 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:49:37.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:37.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:37.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:37.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:49:37.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:49:37.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:573: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:49:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:49:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:49:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:49:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:49:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:49:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:49:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:49:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:49:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:49:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:49:37.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:49:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:49:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:49:37.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:49:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:49:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:49:37.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:49:37.048 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:49:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:37.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:49:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:49:37.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:49:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:49:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:49:37.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:49:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:49:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:49:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:49:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:49:37.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:49:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=35699563-129b-4c73-9af9-61b5fef3e12f 2021-01-15T17:49:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 35699563-129b-4c73-9af9-61b5fef3e12f' 2021-01-15T17:49:37.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:49:37.054 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 35699563-129b-4c73-9af9-61b5fef3e12f 2021-01-15T17:49:37.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCx1QFgDkFxAxAAIliNeh59GoXiz7whOU2b/Q== 2021-01-15T17:49:37.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCx1QFgDkFxAxAAIliNeh59GoXiz7whOU2b/Q=="}' 2021-01-15T17:49:37.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 35699563-129b-4c73-9af9-61b5fef3e12f -i td/osd-backfill-stats/0/new.json 2021-01-15T17:49:37.407 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:49:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:49:37.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCx1QFgDkFxAxAAIliNeh59GoXiz7whOU2b/Q== --osd-uuid 35699563-129b-4c73-9af9-61b5fef3e12f 2021-01-15T17:49:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.443+0000 7f0be6e14080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:37.445 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.445+0000 7f0be6e14080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:37.446 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.446+0000 7f0be6e14080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.447+0000 7f0be6e14080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:49:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.447+0000 7f0be6e14080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:49:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.447+0000 7f0be6e14080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:49:37.488 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:37.488+0000 7f0be6e14080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:49:42.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:49:42.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:49:42.065 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:49:42.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:49:42.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:49:42.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:49:42.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:42.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:49:42.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:49:42.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:49:42.595 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:49:42.616 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:42.616+0000 7feb94444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:42.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:42.636+0000 7feb94444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:42.638 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:42.638+0000 7feb94444080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:42.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:49:42.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:49:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:49:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:49:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:49:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:49:42.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:49:42.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:49:42.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:49:42.995 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:49:43.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:43.153+0000 7feb94444080 -1 Falling back to public interface 2021-01-15T17:49:43.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:49:43.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:43.930+0000 7feb94444080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:49:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:49:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:49:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:49:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:49:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:49:44.390 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:49:44.806 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/576514,v1:127.0.0.1:6803/576514] [v2:127.0.0.1:6804/576514,v1:127.0.0.1:6805/576514] exists,up 35699563-129b-4c73-9af9-61b5fef3e12f 2021-01-15T17:49:44.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:49:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:49:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:49:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:574: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:49:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:49:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:49:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:49:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:49:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:49:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:49:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:49:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:49:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:49:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:49:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:49:44.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:49:44.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:49:44.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:49:44.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:49:44.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:49:44.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:49:44.811 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:49:44.811 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:44.811 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:44.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:44.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:44.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:49:44.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:49:44.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:49:44.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:49:44.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:49:44.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:49:44.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:49:44.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:44.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:49:44.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:49:44.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:49:44.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:49:44.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=14ac27ed-f37b-48db-abdb-7299e4b81174 2021-01-15T17:49:44.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 14ac27ed-f37b-48db-abdb-7299e4b81174' 2021-01-15T17:49:44.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:49:44.815 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 14ac27ed-f37b-48db-abdb-7299e4b81174 2021-01-15T17:49:44.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC41QFglIoDMRAAl6r2x9v8Qvk1zwbdZJqYxw== 2021-01-15T17:49:44.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC41QFglIoDMRAAl6r2x9v8Qvk1zwbdZJqYxw=="}' 2021-01-15T17:49:44.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 14ac27ed-f37b-48db-abdb-7299e4b81174 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:49:45.349 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:49:45.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:49:45.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC41QFglIoDMRAAl6r2x9v8Qvk1zwbdZJqYxw== --osd-uuid 14ac27ed-f37b-48db-abdb-7299e4b81174 2021-01-15T17:49:45.387 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.387+0000 7f970e1d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:45.388 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.388+0000 7f970e1d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.389+0000 7f970e1d5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.390+0000 7f970e1d5080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:49:45.391 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.390+0000 7f970e1d5080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:49:45.391 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.390+0000 7f970e1d5080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:49:45.444 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:45.443+0000 7f970e1d5080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:49:49.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:49:49.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:49:49.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:49:49.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:49:49.982 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:49:50.504 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:49:50.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:49:50.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:50.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:49:50.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:49:50.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:49:50.527 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:50.527+0000 7f4fb3664080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:50.550+0000 7f4fb3664080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:50.551+0000 7f4fb3664080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:50.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:49:50.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:49:50.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:49:50.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:49:50.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:49:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:49:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:49:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:49:50.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:49:50.914 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:49:51.068 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:51.068+0000 7f4fb3664080 -1 Falling back to public interface 2021-01-15T17:49:51.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:49:51.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:51.943+0000 7f4fb3664080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:49:52.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:49:52.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:49:52.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:49:52.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:49:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:49:52.315 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:49:52.711 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/577778,v1:127.0.0.1:6811/577778] [v2:127.0.0.1:6812/577778,v1:127.0.0.1:6813/577778] exists,up 14ac27ed-f37b-48db-abdb-7299e4b81174 2021-01-15T17:49:52.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:49:52.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:49:52.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:49:52.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:575: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:49:52.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:49:52.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:49:52.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:49:52.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:49:52.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:49:52.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:49:52.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:49:52.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:49:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:49:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:49:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:49:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:49:52.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:49:52.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:49:52.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:49:52.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:49:52.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:49:52.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:49:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:49:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:49:52.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:49:52.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:49:52.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:49:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:49:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:49:52.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:49:52.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:49:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d091084d-b96d-41ca-9f4d-6e62a24b81db 2021-01-15T17:49:52.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 d091084d-b96d-41ca-9f4d-6e62a24b81db' 2021-01-15T17:49:52.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:49:52.721 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 d091084d-b96d-41ca-9f4d-6e62a24b81db 2021-01-15T17:49:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDA1QFgk/JxKxAA0JuEX+AW++A2crwWPmTU2g== 2021-01-15T17:49:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDA1QFgk/JxKxAA0JuEX+AW++A2crwWPmTU2g=="}' 2021-01-15T17:49:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d091084d-b96d-41ca-9f4d-6e62a24b81db -i td/osd-backfill-stats/2/new.json 2021-01-15T17:49:53.218 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:49:53.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:49:53.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDA1QFgk/JxKxAA0JuEX+AW++A2crwWPmTU2g== --osd-uuid d091084d-b96d-41ca-9f4d-6e62a24b81db 2021-01-15T17:49:53.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.254+0000 7f141689a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:53.256 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.256+0000 7f141689a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:53.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.257+0000 7f141689a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.259+0000 7f141689a080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:49:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.259+0000 7f141689a080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:49:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.259+0000 7f141689a080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:49:53.324 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:53.323+0000 7f141689a080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:49:58.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:49:58.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:49:58.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:49:58.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:49:58.059 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:49:58.587 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:49:58.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:49:58.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:49:58.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:49:58.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:49:58.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:49:58.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:58.609+0000 7f9bc568a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:58.611 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:58.611+0000 7f9bc568a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:58.612+0000 7f9bc568a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:49:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:49:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:49:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:49:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:49:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:49:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:49:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:49:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:49:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:49:58.983 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:49:59.135 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:59.135+0000 7f9bc568a080 -1 Falling back to public interface 2021-01-15T17:49:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:49:59.967 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:49:59.966+0000 7f9bc568a080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:50:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:50:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:50:00.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:50:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:50:00.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:50:00.388 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:50:00.830 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/579042,v1:127.0.0.1:6819/579042] [v2:127.0.0.1:6820/579042,v1:127.0.0.1:6821/579042] exists,up d091084d-b96d-41ca-9f4d-6e62a24b81db 2021-01-15T17:50:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:50:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:50:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:50:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:576: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:50:00.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:50:00.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:50:00.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:50:00.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:50:00.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:50:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:50:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:50:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:50:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:50:00.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:50:00.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:50:00.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:50:00.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:50:00.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:50:00.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:50:00.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:50:00.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:50:00.840 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:50:00.840 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:50:00.841 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:50:00.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:50:00.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:50:00.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:50:00.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:50:00.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:50:00.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:50:00.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:50:00.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:50:00.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:50:00.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:50:00.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:50:00.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:50:00.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:50:00.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:50:00.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1 2021-01-15T17:50:00.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1' 2021-01-15T17:50:00.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:50:00.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDI1QFgVvR0MhAAre3nx15lp2SAauff/2G+mA== 2021-01-15T17:50:00.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDI1QFgVvR0MhAAre3nx15lp2SAauff/2G+mA=="}' 2021-01-15T17:50:00.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1 -i td/osd-backfill-stats/3/new.json 2021-01-15T17:50:00.848 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1 2021-01-15T17:50:01.412 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:50:01.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:50:01.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDI1QFgVvR0MhAAre3nx15lp2SAauff/2G+mA== --osd-uuid c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1 2021-01-15T17:50:01.449 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.448+0000 7f61133d7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.482+0000 7f61133d7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:01.484 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.484+0000 7f61133d7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.485+0000 7f61133d7080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:50:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.485+0000 7f61133d7080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:50:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.485+0000 7f61133d7080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:50:01.558 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:01.558+0000 7f61133d7080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:50:06.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:50:06.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:50:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:50:06.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:50:06.100 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:50:06.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:50:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:50:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:50:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:50:06.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:50:06.634 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:50:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:06.656+0000 7fc7ee545080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:06.658 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:06.658+0000 7fc7ee545080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:06.659 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:06.659+0000 7fc7ee545080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:07.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:50:07.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:50:07.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:50:07.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:50:07.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:50:07.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:50:07.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:50:07.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:50:07.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:50:07.028 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:50:07.179 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:07.179+0000 7fc7ee545080 -1 Falling back to public interface 2021-01-15T17:50:07.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:50:07.983 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:07.983+0000 7fc7ee545080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:50:08.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:50:08.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:50:08.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:50:08.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:50:08.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:50:08.422 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:50:08.843 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/580306,v1:127.0.0.1:6827/580306] [v2:127.0.0.1:6828/580306,v1:127.0.0.1:6829/580306] exists,up c94de3b8-f2e1-45e8-bb69-96fd96f5ebd1 2021-01-15T17:50:08.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:50:08.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:50:08.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:50:08.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:577: TEST_backfill_ec_prim_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:50:08.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:50:08.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:50:08.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:50:08.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:50:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:50:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:50:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:50:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:50:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:50:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:50:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:50:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:50:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:50:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:50:08.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:50:08.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:50:08.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:50:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:50:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:50:08.849 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:50:08.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:50:08.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:50:08.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:50:08.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:50:08.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:50:08.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:50:08.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:50:08.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:50:08.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:50:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:50:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:50:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:50:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:50:08.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:50:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5bc1700b-6541-4df1-b5f5-93c64a2e599d 2021-01-15T17:50:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 5bc1700b-6541-4df1-b5f5-93c64a2e599d' 2021-01-15T17:50:08.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:50:08.853 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 5bc1700b-6541-4df1-b5f5-93c64a2e599d 2021-01-15T17:50:08.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDQ1QFg3Ag5MxAAaLl9BjgcdO2xb5jHAiivqQ== 2021-01-15T17:50:08.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDQ1QFg3Ag5MxAAaLl9BjgcdO2xb5jHAiivqQ=="}' 2021-01-15T17:50:08.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5bc1700b-6541-4df1-b5f5-93c64a2e599d -i td/osd-backfill-stats/4/new.json 2021-01-15T17:50:09.502 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:50:09.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:50:09.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQ1QFg3Ag5MxAAaLl9BjgcdO2xb5jHAiivqQ== --osd-uuid 5bc1700b-6541-4df1-b5f5-93c64a2e599d 2021-01-15T17:50:09.540 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.539+0000 7f2f794ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:09.541 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.541+0000 7f2f794ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:09.543 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.542+0000 7f2f794ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:09.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.543+0000 7f2f794ca080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:50:09.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.543+0000 7f2f794ca080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:50:09.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.543+0000 7f2f794ca080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:50:09.658 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:09.658+0000 7f2f794ca080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:50:14.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:50:14.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:50:14.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:50:14.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:50:14.400 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:50:14.930 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:50:14.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:50:14.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:50:14.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:50:14.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:50:14.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:50:14.953 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:14.953+0000 7f3645e2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:14.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:14.954+0000 7f3645e2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:14.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:14.956+0000 7f3645e2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:50:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:50:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:50:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:50:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:50:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:50:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:50:15.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:50:15.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:50:15.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:50:15.326 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:50:15.474 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:15.473+0000 7f3645e2b080 -1 Falling back to public interface 2021-01-15T17:50:15.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:50:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:50:16.275+0000 7f3645e2b080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:50:16.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:50:16.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:50:16.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:50:16.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:50:16.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:50:16.731 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:50:17.155 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/581570,v1:127.0.0.1:6835/581570] [v2:127.0.0.1:6836/581570,v1:127.0.0.1:6837/581570] exists,up 5bc1700b-6541-4df1-b5f5-93c64a2e599d 2021-01-15T17:50:17.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:50:17.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:50:17.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:50:17.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:579: TEST_backfill_ec_prim_out: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T17:50:17.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:580: TEST_backfill_ec_prim_out: create_pool test 1 1 erasure myprofile 2021-01-15T17:50:17.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T17:50:18.433 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:50:18.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:50:19.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:582: TEST_backfill_ec_prim_out: wait_for_clean 2021-01-15T17:50:19.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:50:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:50:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:50:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:50:19.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:50:19.471 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:50:19.471 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:50:19.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:50:19.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:50:19.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:50:19.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:50:19.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:50:19.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:50:19.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:50:19.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:50:19.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:50:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:50:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:50:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:50:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:50:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:50:19.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:50:19.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:50:20.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T17:50:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T17:50:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T17:50:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:50:20.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:50:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T17:50:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T17:50:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-01-15T17:50:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:50:20.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:50:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542151 2021-01-15T17:50:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542151 2021-01-15T17:50:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-60129542151' 2021-01-15T17:50:20.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:50:20.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:50:20.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378630 2021-01-15T17:50:20.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378630 2021-01-15T17:50:20.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-60129542151 3-81604378630' 2021-01-15T17:50:20.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:50:20.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:50:20.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215107 2021-01-15T17:50:20.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215107 2021-01-15T17:50:20.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-60129542151 3-81604378630 4-103079215107' 2021-01-15T17:50:20.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:50:20.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T17:50:20.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:50:20.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:50:20.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T17:50:20.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:50:20.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T17:50:20.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T17:50:20.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:50:20.698 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T17:50:21.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836490 2021-01-15T17:50:21.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:50:21.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T17:50:21.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:50:21.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:50:21.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T17:50:21.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:50:21.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T17:50:21.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T17:50:21.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:50:21.094 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T17:50:21.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T17:50:21.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:50:21.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542151 2021-01-15T17:50:21.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:50:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:50:21.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542151 2021-01-15T17:50:21.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:50:21.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542151 2021-01-15T17:50:21.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2021-01-15T17:50:21.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:50:21.524 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 60129542151 2021-01-15T17:50:21.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542151 -lt 60129542151 2021-01-15T17:50:21.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:50:21.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378630 2021-01-15T17:50:21.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:50:21.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:50:21.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378630 2021-01-15T17:50:21.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:50:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378630 2021-01-15T17:50:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2021-01-15T17:50:21.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:50:21.935 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378630 2021-01-15T17:50:22.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378630 -lt 81604378630 2021-01-15T17:50:22.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:50:22.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215107 2021-01-15T17:50:22.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:50:22.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:50:22.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215107 2021-01-15T17:50:22.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:50:22.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215107 2021-01-15T17:50:22.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215107' 2021-01-15T17:50:22.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:50:22.324 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215107 2021-01-15T17:50:22.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215107 -lt 103079215107 2021-01-15T17:50:22.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:50:22.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:50:22.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:50:23.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:50:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:50:23.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:50:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:50:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:50:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:50:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:50:23.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:50:23.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:50:23.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:50:23.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:50:23.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:50:24.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:50:24.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:50:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:50:24.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: seq 1 200 2021-01-15T17:50:24.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:24.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj1 /dev/null 2021-01-15T17:50:24.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:24.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj2 /dev/null 2021-01-15T17:50:24.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:24.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj3 /dev/null 2021-01-15T17:50:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:24.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj4 /dev/null 2021-01-15T17:50:24.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj5 /dev/null 2021-01-15T17:50:25.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj6 /dev/null 2021-01-15T17:50:25.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj7 /dev/null 2021-01-15T17:50:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj8 /dev/null 2021-01-15T17:50:25.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj9 /dev/null 2021-01-15T17:50:25.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj10 /dev/null 2021-01-15T17:50:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj11 /dev/null 2021-01-15T17:50:26.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj12 /dev/null 2021-01-15T17:50:26.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj13 /dev/null 2021-01-15T17:50:26.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj14 /dev/null 2021-01-15T17:50:26.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj15 /dev/null 2021-01-15T17:50:26.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj16 /dev/null 2021-01-15T17:50:26.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:26.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj17 /dev/null 2021-01-15T17:50:27.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj18 /dev/null 2021-01-15T17:50:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj19 /dev/null 2021-01-15T17:50:27.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj20 /dev/null 2021-01-15T17:50:27.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj21 /dev/null 2021-01-15T17:50:27.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj22 /dev/null 2021-01-15T17:50:27.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:27.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj23 /dev/null 2021-01-15T17:50:28.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:28.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj24 /dev/null 2021-01-15T17:50:28.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:28.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj25 /dev/null 2021-01-15T17:50:28.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:28.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj26 /dev/null 2021-01-15T17:50:28.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:28.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj27 /dev/null 2021-01-15T17:50:28.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:28.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj28 /dev/null 2021-01-15T17:50:29.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj29 /dev/null 2021-01-15T17:50:29.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj30 /dev/null 2021-01-15T17:50:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj31 /dev/null 2021-01-15T17:50:29.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj32 /dev/null 2021-01-15T17:50:29.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj33 /dev/null 2021-01-15T17:50:29.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:29.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj34 /dev/null 2021-01-15T17:50:30.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj35 /dev/null 2021-01-15T17:50:30.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj36 /dev/null 2021-01-15T17:50:30.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj37 /dev/null 2021-01-15T17:50:30.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj38 /dev/null 2021-01-15T17:50:30.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj39 /dev/null 2021-01-15T17:50:30.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:30.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj40 /dev/null 2021-01-15T17:50:31.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:31.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj41 /dev/null 2021-01-15T17:50:31.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:31.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj42 /dev/null 2021-01-15T17:50:31.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:31.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj43 /dev/null 2021-01-15T17:50:31.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:31.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj44 /dev/null 2021-01-15T17:50:31.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:31.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj45 /dev/null 2021-01-15T17:50:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:32.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj46 /dev/null 2021-01-15T17:50:32.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:32.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj47 /dev/null 2021-01-15T17:50:32.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:32.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj48 /dev/null 2021-01-15T17:50:32.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:32.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj49 /dev/null 2021-01-15T17:50:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:32.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj50 /dev/null 2021-01-15T17:50:33.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj51 /dev/null 2021-01-15T17:50:33.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj52 /dev/null 2021-01-15T17:50:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj53 /dev/null 2021-01-15T17:50:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj54 /dev/null 2021-01-15T17:50:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj55 /dev/null 2021-01-15T17:50:33.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:33.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj56 /dev/null 2021-01-15T17:50:34.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj57 /dev/null 2021-01-15T17:50:34.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj58 /dev/null 2021-01-15T17:50:34.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj59 /dev/null 2021-01-15T17:50:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj60 /dev/null 2021-01-15T17:50:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj61 /dev/null 2021-01-15T17:50:34.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:34.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj62 /dev/null 2021-01-15T17:50:35.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:35.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj63 /dev/null 2021-01-15T17:50:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj64 /dev/null 2021-01-15T17:50:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj65 /dev/null 2021-01-15T17:50:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj66 /dev/null 2021-01-15T17:50:35.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:35.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj67 /dev/null 2021-01-15T17:50:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj68 /dev/null 2021-01-15T17:50:36.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj69 /dev/null 2021-01-15T17:50:36.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj70 /dev/null 2021-01-15T17:50:36.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj71 /dev/null 2021-01-15T17:50:36.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj72 /dev/null 2021-01-15T17:50:36.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:36.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj73 /dev/null 2021-01-15T17:50:37.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:37.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj74 /dev/null 2021-01-15T17:50:37.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:37.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj75 /dev/null 2021-01-15T17:50:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj76 /dev/null 2021-01-15T17:50:37.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:37.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj77 /dev/null 2021-01-15T17:50:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:37.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj78 /dev/null 2021-01-15T17:50:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj79 /dev/null 2021-01-15T17:50:38.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:38.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj80 /dev/null 2021-01-15T17:50:38.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:38.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj81 /dev/null 2021-01-15T17:50:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:38.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj82 /dev/null 2021-01-15T17:50:38.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:38.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj83 /dev/null 2021-01-15T17:50:39.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj84 /dev/null 2021-01-15T17:50:39.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj85 /dev/null 2021-01-15T17:50:39.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj86 /dev/null 2021-01-15T17:50:39.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj87 /dev/null 2021-01-15T17:50:39.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj88 /dev/null 2021-01-15T17:50:39.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:39.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj89 /dev/null 2021-01-15T17:50:40.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:40.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj90 /dev/null 2021-01-15T17:50:40.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:40.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj91 /dev/null 2021-01-15T17:50:40.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:40.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj92 /dev/null 2021-01-15T17:50:40.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:40.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj93 /dev/null 2021-01-15T17:50:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj94 /dev/null 2021-01-15T17:50:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj95 /dev/null 2021-01-15T17:50:41.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj96 /dev/null 2021-01-15T17:50:41.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:41.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj97 /dev/null 2021-01-15T17:50:41.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:41.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj98 /dev/null 2021-01-15T17:50:41.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:41.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj99 /dev/null 2021-01-15T17:50:42.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj100 /dev/null 2021-01-15T17:50:42.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj101 /dev/null 2021-01-15T17:50:42.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj102 /dev/null 2021-01-15T17:50:42.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj103 /dev/null 2021-01-15T17:50:42.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj104 /dev/null 2021-01-15T17:50:42.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:42.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj105 /dev/null 2021-01-15T17:50:43.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:43.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj106 /dev/null 2021-01-15T17:50:43.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:43.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj107 /dev/null 2021-01-15T17:50:43.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:43.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj108 /dev/null 2021-01-15T17:50:43.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:43.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj109 /dev/null 2021-01-15T17:50:43.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:43.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj110 /dev/null 2021-01-15T17:50:44.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:44.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj111 /dev/null 2021-01-15T17:50:44.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:44.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj112 /dev/null 2021-01-15T17:50:44.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:44.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj113 /dev/null 2021-01-15T17:50:44.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:44.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj114 /dev/null 2021-01-15T17:50:44.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:44.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj115 /dev/null 2021-01-15T17:50:45.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:45.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj116 /dev/null 2021-01-15T17:50:45.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:45.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj117 /dev/null 2021-01-15T17:50:45.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:45.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj118 /dev/null 2021-01-15T17:50:45.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:45.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj119 /dev/null 2021-01-15T17:50:45.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:45.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj120 /dev/null 2021-01-15T17:50:46.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj121 /dev/null 2021-01-15T17:50:46.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj122 /dev/null 2021-01-15T17:50:46.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj123 /dev/null 2021-01-15T17:50:46.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj124 /dev/null 2021-01-15T17:50:46.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj125 /dev/null 2021-01-15T17:50:46.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:46.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj126 /dev/null 2021-01-15T17:50:47.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:47.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj127 /dev/null 2021-01-15T17:50:47.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:47.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj128 /dev/null 2021-01-15T17:50:47.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:47.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj129 /dev/null 2021-01-15T17:50:47.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:47.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj130 /dev/null 2021-01-15T17:50:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:47.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj131 /dev/null 2021-01-15T17:50:48.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:48.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj132 /dev/null 2021-01-15T17:50:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:48.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj133 /dev/null 2021-01-15T17:50:48.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:48.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj134 /dev/null 2021-01-15T17:50:48.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:48.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj135 /dev/null 2021-01-15T17:50:48.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:48.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj136 /dev/null 2021-01-15T17:50:49.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:49.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj137 /dev/null 2021-01-15T17:50:49.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:49.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj138 /dev/null 2021-01-15T17:50:49.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:49.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj139 /dev/null 2021-01-15T17:50:49.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:49.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj140 /dev/null 2021-01-15T17:50:49.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:49.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj141 /dev/null 2021-01-15T17:50:50.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj142 /dev/null 2021-01-15T17:50:50.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj143 /dev/null 2021-01-15T17:50:50.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj144 /dev/null 2021-01-15T17:50:50.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj145 /dev/null 2021-01-15T17:50:50.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj146 /dev/null 2021-01-15T17:50:50.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:50.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj147 /dev/null 2021-01-15T17:50:51.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:51.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj148 /dev/null 2021-01-15T17:50:51.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:51.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj149 /dev/null 2021-01-15T17:50:51.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:51.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj150 /dev/null 2021-01-15T17:50:51.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:51.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj151 /dev/null 2021-01-15T17:50:51.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:51.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj152 /dev/null 2021-01-15T17:50:52.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:52.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj153 /dev/null 2021-01-15T17:50:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj154 /dev/null 2021-01-15T17:50:52.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:52.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj155 /dev/null 2021-01-15T17:50:52.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:52.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj156 /dev/null 2021-01-15T17:50:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:52.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj157 /dev/null 2021-01-15T17:50:53.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:53.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj158 /dev/null 2021-01-15T17:50:53.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:53.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj159 /dev/null 2021-01-15T17:50:53.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:53.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj160 /dev/null 2021-01-15T17:50:53.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:53.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj161 /dev/null 2021-01-15T17:50:53.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:53.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj162 /dev/null 2021-01-15T17:50:54.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj163 /dev/null 2021-01-15T17:50:54.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj164 /dev/null 2021-01-15T17:50:54.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj165 /dev/null 2021-01-15T17:50:54.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj166 /dev/null 2021-01-15T17:50:54.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj167 /dev/null 2021-01-15T17:50:54.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:54.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj168 /dev/null 2021-01-15T17:50:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj169 /dev/null 2021-01-15T17:50:55.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:55.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj170 /dev/null 2021-01-15T17:50:55.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:55.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj171 /dev/null 2021-01-15T17:50:55.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:55.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj172 /dev/null 2021-01-15T17:50:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:55.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj173 /dev/null 2021-01-15T17:50:56.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:56.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj174 /dev/null 2021-01-15T17:50:56.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:56.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj175 /dev/null 2021-01-15T17:50:56.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:56.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj176 /dev/null 2021-01-15T17:50:56.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:56.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj177 /dev/null 2021-01-15T17:50:56.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:56.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj178 /dev/null 2021-01-15T17:50:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj179 /dev/null 2021-01-15T17:50:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj180 /dev/null 2021-01-15T17:50:57.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:57.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj181 /dev/null 2021-01-15T17:50:57.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:57.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj182 /dev/null 2021-01-15T17:50:57.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:57.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj183 /dev/null 2021-01-15T17:50:58.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj184 /dev/null 2021-01-15T17:50:58.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj185 /dev/null 2021-01-15T17:50:58.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj186 /dev/null 2021-01-15T17:50:58.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj187 /dev/null 2021-01-15T17:50:58.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj188 /dev/null 2021-01-15T17:50:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj189 /dev/null 2021-01-15T17:50:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:59.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj190 /dev/null 2021-01-15T17:50:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj191 /dev/null 2021-01-15T17:50:59.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:59.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj192 /dev/null 2021-01-15T17:50:59.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:59.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj193 /dev/null 2021-01-15T17:50:59.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:50:59.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj194 /dev/null 2021-01-15T17:51:00.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:00.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj195 /dev/null 2021-01-15T17:51:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj196 /dev/null 2021-01-15T17:51:00.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:00.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj197 /dev/null 2021-01-15T17:51:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj198 /dev/null 2021-01-15T17:51:00.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:00.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj199 /dev/null 2021-01-15T17:51:01.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:584: TEST_backfill_ec_prim_out: for i in $(seq 1 $objects) 2021-01-15T17:51:01.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:586: TEST_backfill_ec_prim_out: rados -p test put obj200 /dev/null 2021-01-15T17:51:01.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:589: TEST_backfill_ec_prim_out: get_pg test obj1 2021-01-15T17:51:01.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:51:01.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:51:01.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:51:01.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:51:01.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:589: TEST_backfill_ec_prim_out: local PG=1.0 2021-01-15T17:51:01.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:591: TEST_backfill_ec_prim_out: get_primary test obj1 2021-01-15T17:51:01.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:51:01.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:51:01.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:51:01.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:51:02.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:591: TEST_backfill_ec_prim_out: local primary=1 2021-01-15T17:51:02.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:593: TEST_backfill_ec_prim_out: ceph osd set nobackfill 2021-01-15T17:51:02.781 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:51:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:594: TEST_backfill_ec_prim_out: ceph osd out osd.1 2021-01-15T17:51:03.340 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:51:03.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:596: TEST_backfill_ec_prim_out: sleep 2 2021-01-15T17:51:05.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:597: TEST_backfill_ec_prim_out: get_primary test obj1 2021-01-15T17:51:05.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:51:05.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:51:05.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:51:05.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:51:05.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:597: TEST_backfill_ec_prim_out: primary=1 2021-01-15T17:51:05.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:598: TEST_backfill_ec_prim_out: ceph osd unset nobackfill 2021-01-15T17:51:06.578 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:51:06.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:599: TEST_backfill_ec_prim_out: ceph tell osd.1 get_latest_osdmap 2021-01-15T17:51:06.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:600: TEST_backfill_ec_prim_out: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:51:06.858 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:51:06.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:601: TEST_backfill_ec_prim_out: sleep 2 2021-01-15T17:51:08.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:603: TEST_backfill_ec_prim_out: wait_for_clean 2021-01-15T17:51:08.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:51:08.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:51:08.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:51:08.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:51:08.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:51:08.871 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:51:08.871 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:51:08.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:51:08.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:51:08.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:51:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:51:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:51:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:51:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:51:08.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:51:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:51:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:51:09.415 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:51:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:51:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T17:51:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:51:09.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:51:09.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:51:09.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2021-01-15T17:51:09.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2021-01-15T17:51:09.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2021-01-15T17:51:09.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:51:09.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:51:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672980 2021-01-15T17:51:09.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672980 2021-01-15T17:51:09.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980' 2021-01-15T17:51:09.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:51:09.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:51:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542163 2021-01-15T17:51:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542163 2021-01-15T17:51:09.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-60129542163' 2021-01-15T17:51:09.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:51:09.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:51:09.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378642 2021-01-15T17:51:09.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378642 2021-01-15T17:51:09.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-60129542163 3-81604378642' 2021-01-15T17:51:09.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:51:09.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:51:10.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215119 2021-01-15T17:51:10.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215119 2021-01-15T17:51:10.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672980 2-60129542163 3-81604378642 4-103079215119' 2021-01-15T17:51:10.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:51:10.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2021-01-15T17:51:10.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:51:10.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:51:10.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2021-01-15T17:51:10.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:51:10.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2021-01-15T17:51:10.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-01-15T17:51:10.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:51:10.107 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836502 2021-01-15T17:51:10.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836502 2021-01-15T17:51:10.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:51:11.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:51:11.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:51:11.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836502 2021-01-15T17:51:11.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:51:11.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672980 2021-01-15T17:51:11.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:51:11.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:51:11.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672980 2021-01-15T17:51:11.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:51:11.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672980 2021-01-15T17:51:11.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-01-15T17:51:11.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:51:11.904 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672980 2021-01-15T17:51:12.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672980 2021-01-15T17:51:12.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:51:12.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542163 2021-01-15T17:51:12.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:51:12.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:51:12.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542163 2021-01-15T17:51:12.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:51:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542163 2021-01-15T17:51:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542163' 2021-01-15T17:51:12.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:51:12.318 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 60129542163 2021-01-15T17:51:12.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542163 -lt 60129542163 2021-01-15T17:51:12.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:51:12.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378642 2021-01-15T17:51:12.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:51:12.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:51:12.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378642 2021-01-15T17:51:12.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:51:12.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378642 2021-01-15T17:51:12.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378642' 2021-01-15T17:51:12.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:51:12.736 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378642 2021-01-15T17:51:13.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378642 -lt 81604378642 2021-01-15T17:51:13.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:51:13.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215119 2021-01-15T17:51:13.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:51:13.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:51:13.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215119 2021-01-15T17:51:13.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:51:13.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215119 2021-01-15T17:51:13.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215119' 2021-01-15T17:51:13.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:51:13.132 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215119 2021-01-15T17:51:13.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215120 -lt 103079215119 2021-01-15T17:51:13.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:51:13.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:13.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:51:14.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:14.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:14.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:14.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:14.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:14.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:14.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:14.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:14.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:14.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:14.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:14.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:14.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:51:14.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:51:14.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:51:14.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:14.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:15.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:15.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:15.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:15.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:15.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:15.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:15.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:15.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:15.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:15.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:15.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:15.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:15.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:15.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:15.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:15.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:15.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:16.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:51:16.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:51:16.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:16.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:16.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:16.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:16.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:16.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:16.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:16.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:16.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:16.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:16.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:16.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:16.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:16.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:16.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:17.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:17.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:17.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:18.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:51:18.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:51:18.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:18.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:18.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:18.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:18.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:18.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:18.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:18.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:18.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:18.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:18.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:18.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:18.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:18.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:18.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:19.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:19.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:19.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:19.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:19.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:19.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:19.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:19.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:19.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:19.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:19.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:19.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:19.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:19.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:19.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:19.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:19.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:19.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:19.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:19.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:19.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:19.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:20.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:20.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:20.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:20.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:20.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:20.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:20.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:20.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:20.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:20.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:20.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:20.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:20.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:21.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:21.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:21.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:21.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:21.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:21.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:21.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:21.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:21.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:21.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:21.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:21.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:21.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:21.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:21.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:21.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:21.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:22.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:22.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:22.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:22.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:22.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:22.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:22.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:22.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:22.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:22.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:22.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:22.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:22.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:22.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:22.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:22.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:22.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:22.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:22.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:22.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:23.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:23.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:23.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:23.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:23.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:23.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:23.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:23.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:23.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:23.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:51:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:51:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:24.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:24.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:24.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:24.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:24.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:24.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:24.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:24.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:24.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:24.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:24.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:24.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:24.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:25.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:25.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:25.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:25.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:25.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:51:25.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:51:25.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:25.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:25.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:25.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:25.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:25.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:25.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:25.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:25.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:25.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:25.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:26.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:26.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:26.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:26.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:26.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:26.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:26.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:26.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:26.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:26.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:27.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:27.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:27.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:27.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:27.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:27.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:27.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:27.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:27.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:27.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:27.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:27.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:28.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:28.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:28.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:28.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:28.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:28.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:28.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:28.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:28.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:28.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:28.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:28.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:28.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:28.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:28.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:28.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:28.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:28.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:29.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:29.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:29.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:29.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:29.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:29.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:29.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:29.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:29.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:29.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:29.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:51:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:51:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:30.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:30.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:30.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:30.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:30.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:30.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:30.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:30.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:30.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:30.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:30.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:30.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:31.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:31.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:31.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:31.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:31.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:31.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:31.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:31.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:31.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:31.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:31.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:31.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:31.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:31.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:31.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:31.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:31.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:31.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:31.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:31.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:32.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:32.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:32.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:32.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:32.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:32.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:32.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:32.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:32.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:33.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:33.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:33.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:33.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:33.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:33.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:33.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:33.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:33.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:33.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:33.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:34.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:34.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:34.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:34.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:34.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:34.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:34.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:34.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:34.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:34.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:34.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:34.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:34.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:35.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:35.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:35.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:35.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:35.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:35.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:35.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:35.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:35.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:36.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:51:36.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:51:36.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:36.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:36.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:36.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:36.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:36.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:36.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:36.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:36.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:36.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:36.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:36.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:36.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:36.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:36.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:37.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:37.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:37.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:37.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:37.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:37.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:37.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:37.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:37.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:37.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:37.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:37.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:37.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:38.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:38.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:38.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:38.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:38.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:38.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:38.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:38.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:38.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:38.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:51:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:51:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:39.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:39.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:39.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:39.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:39.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:39.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:39.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:39.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:39.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:39.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:39.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:39.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:39.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:39.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:40.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:40.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:40.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:40.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:40.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:40.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:40.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:40.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:40.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:40.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:51:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:51:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:40.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:40.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:40.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:40.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:40.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:40.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:40.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:40.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:40.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:41.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:41.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:41.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:41.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:41.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:41.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:41.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:41.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:42.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:42.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:42.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:42.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:42.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:42.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:42.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:42.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:42.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:42.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:42.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:42.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:42.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:42.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:42.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:43.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:43.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:43.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:43.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:43.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:43.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:43.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:43.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:43.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:43.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:51:43.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:51:43.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:43.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:43.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:44.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:44.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:44.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:44.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:44.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:44.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:44.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:44.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:44.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:44.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:44.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:44.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:44.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:44.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:44.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:44.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:44.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:45.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:51:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:51:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:45.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:45.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:45.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:45.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:45.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:45.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:45.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:45.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:45.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:45.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:45.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:45.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:45.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:46.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:46.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:46.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:46.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:46.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:46.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:46.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:46.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:46.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:47.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:51:47.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:51:47.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:47.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:47.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:47.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:47.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:47.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:47.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:47.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:51:47.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:47.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:47.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:48.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:51:48.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:51:48.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:51:48.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:51:48.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:51:48.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:51:48.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:51:48.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:51:48.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:51:48.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:51:48.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:51:48.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:51:48.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:51:48.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:51:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:51:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:51:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:51:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:51:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:51:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:51:48.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:51:48.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:51:49.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:51:49.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:51:49.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:51:49.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:51:49.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:51:49.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:51:49.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:51:49.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:605: TEST_backfill_ec_prim_out: expr 200 '*' 3 2021-01-15T17:51:49.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:605: TEST_backfill_ec_prim_out: local misplaced=600 2021-01-15T17:51:49.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:606: TEST_backfill_ec_prim_out: check td/osd-backfill-stats 1.0 1 erasure 0 0 200 0 2021-01-15T17:51:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:51:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:51:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:51:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=erasure 2021-01-15T17:51:49.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2021-01-15T17:51:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:51:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2021-01-15T17:51:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:51:49.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:51:49.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:51:49.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:51:49.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:51:49.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2021-01-15T17:51:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:51:49.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:51:49.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:51:49.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' erasure = erasure ']' 2021-01-15T17:51:49.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:89: check: addp=p 2021-01-15T17:51:49.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:51:49.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:51:49.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:51:49.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[4,0,2]/[1,0,2]' 2021-01-15T17:51:49.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[4,0,2]/[1,0,2]' ']' 2021-01-15T17:51:49.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:51:49.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:51:49.604 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [4,0,2]/[1,0,2]p' 2021-01-15T17:51:49.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:51:49.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:51:49.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:51:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2021-01-15T17:51:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2021-01-15T17:51:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2021-01-15T17:51:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:51:49.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:51:49.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:51:49.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:51:49.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:51:49.613 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.613 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:51:49.613 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:51:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2021-01-15T17:51:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2021-01-15T17:51:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2021-01-15T17:51:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:51:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:51:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:51:49.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:51:49.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:51:49.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:51:49.624 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.624 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [4,0,2]/[1,0,2]p' 2021-01-15T17:51:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:51:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:51:49.625 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:51:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2021-01-15T17:51:49.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2021-01-15T17:51:49.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2021-01-15T17:51:49.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:51:49.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:51:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:51:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:51:49.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:51:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:51:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:51:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:51:49.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2021-01-15T17:51:49.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2021-01-15T17:51:49.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2021-01-15T17:51:49.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:51:49.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:51:49.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:51:49.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:51:49.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:51:49.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:608: TEST_backfill_ec_prim_out: delete_pool test 2021-01-15T17:51:49.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:51:49.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:51:50.225 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:51:50.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:609: TEST_backfill_ec_prim_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:51:50.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:51:50.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:51:50.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:51:50.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:51:50.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:51:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:51:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:51:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:51:50.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:51:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:51:50.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:51:50.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:51:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:51:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:51:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:51:50.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:51:50.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:51:50.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:51:50.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:51:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:51:50.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:51:50.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:51:50.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:51:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:51:50.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:51:50.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:51:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:51:50.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:51:50.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:51:50.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:51:50.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:50.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:50.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:51:50.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:51:50.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:51:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:51:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:51:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:51:50.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:51:50.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:51:50.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:51:50.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:51:50.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:51:50.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:51:50.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:51:50.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:51:50.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:51:50.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:51:50.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:51:50.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:51:50.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:51:50.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:51:50.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:51:50.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:51:50.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:51:50.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:51:50.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:51:50.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:51:50.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:51:50.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:51:50.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:51:50.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:51:50.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:50.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:51:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:51:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:51:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:51:50.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:51:50.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:50.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:50.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:51:50.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:51:50.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:51:50.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:51:50.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:51:50.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_out td/osd-backfill-stats 2021-01-15T17:51:50.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:230: TEST_backfill_out: local dir=td/osd-backfill-stats 2021-01-15T17:51:50.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:232: TEST_backfill_out: run_mon td/osd-backfill-stats a 2021-01-15T17:51:50.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:51:50.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:51:50.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:51:50.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:51:50.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:51:50.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:51:50.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:51:50.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:51:50.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:51:50.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:51:50.893 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:50.894 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:50.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:50.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:51:51.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:51:51.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:51:51.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:51:51.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:51:51.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:51:51.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:51:51.077 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:51:51.078 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:51:51.078 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:51:51.078 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:51:51.078 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:51.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:51:51.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:51:51.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:51:51.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:51:51.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:51:51.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:51:51.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:51:51.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:51:51.179 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:51:51.179 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:51:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:51:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:51:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:51.181 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:51:51.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:51:51.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:51:51.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:233: TEST_backfill_out: run_mgr td/osd-backfill-stats x 2021-01-15T17:51:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:51:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:51:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:51:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:51:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:51:51.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:51:51.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:51:51.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:51:51.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:51:51.780 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:51:51.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:51.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:51.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:51.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:51:51.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:51:51.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:234: TEST_backfill_out: run_osd td/osd-backfill-stats 0 2021-01-15T17:51:51.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:51:51.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:51:51.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:51:51.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:51:51.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:51:51.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:51:51.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:51:51.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:51:51.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:51:51.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:51:51.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:51:51.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:51:51.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:51:51.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:51:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:51:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:51:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:51:51.813 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:51:51.814 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:51.814 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:51.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:51:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:51:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:51:51.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:51:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:51:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:51:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:51:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:51:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:51:51.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:51:51.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:51:51.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:51:51.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=49d81300-419c-4b69-8b3b-e4f11015cc15 2021-01-15T17:51:51.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 49d81300-419c-4b69-8b3b-e4f11015cc15' 2021-01-15T17:51:51.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:51:51.818 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 49d81300-419c-4b69-8b3b-e4f11015cc15 2021-01-15T17:51:51.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA31gFgi6TnMBAAtSq+h6dFG6ZJb0FMEVA5Kw== 2021-01-15T17:51:51.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA31gFgi6TnMBAAtSq+h6dFG6ZJb0FMEVA5Kw=="}' 2021-01-15T17:51:51.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 49d81300-419c-4b69-8b3b-e4f11015cc15 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:51:52.160 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:51:52.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:51:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA31gFgi6TnMBAAtSq+h6dFG6ZJb0FMEVA5Kw== --osd-uuid 49d81300-419c-4b69-8b3b-e4f11015cc15 2021-01-15T17:51:52.198 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.197+0000 7f3e95468080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:52.199 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.199+0000 7f3e95468080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:52.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.200+0000 7f3e95468080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:52.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.201+0000 7f3e95468080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:51:52.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.201+0000 7f3e95468080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:51:52.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.201+0000 7f3e95468080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:51:52.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:52.241+0000 7f3e95468080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:51:56.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:51:56.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:51:56.808 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:51:56.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:51:56.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:51:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:51:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:51:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:51:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:51:57.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:51:57.332 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:51:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:57.353+0000 7f6856179080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:57.373 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:57.373+0000 7f6856179080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:57.374 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:57.374+0000 7f6856179080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:51:57.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:51:57.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:51:57.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:51:57.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:51:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:51:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:51:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:51:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:51:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:51:57.733 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:51:57.890 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:57.889+0000 7f6856179080 -1 Falling back to public interface 2021-01-15T17:51:58.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:51:58.674 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:51:58.673+0000 7f6856179080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:51:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:51:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:51:59.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:51:59.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:51:59.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:51:59.129 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:51:59.543 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/590244,v1:127.0.0.1:6803/590244] [v2:127.0.0.1:6804/590244,v1:127.0.0.1:6805/590244] exists,up 49d81300-419c-4b69-8b3b-e4f11015cc15 2021-01-15T17:51:59.543 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 88d80cb3-94c9-4141-8b23-0eec1b0bd870 2021-01-15T17:51:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:51:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:51:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:51:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:235: TEST_backfill_out: run_osd td/osd-backfill-stats 1 2021-01-15T17:51:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:51:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:51:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:51:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:51:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:51:59.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:51:59.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:51:59.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:51:59.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:51:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:51:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:51:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:51:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:51:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:51:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:51:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:51:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:51:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:51:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:51:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:51:59.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:51:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:51:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:51:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:51:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:51:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:51:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:51:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:51:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:51:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:51:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:51:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:51:59.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:51:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=88d80cb3-94c9-4141-8b23-0eec1b0bd870 2021-01-15T17:51:59.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 88d80cb3-94c9-4141-8b23-0eec1b0bd870' 2021-01-15T17:51:59.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:51:59.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA/1gFgpDxHIRAAhzepy8sWXDfPnVqIDA2yOg== 2021-01-15T17:51:59.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA/1gFgpDxHIRAAhzepy8sWXDfPnVqIDA2yOg=="}' 2021-01-15T17:51:59.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 88d80cb3-94c9-4141-8b23-0eec1b0bd870 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:52:00.116 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:52:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA/1gFgpDxHIRAAhzepy8sWXDfPnVqIDA2yOg== --osd-uuid 88d80cb3-94c9-4141-8b23-0eec1b0bd870 2021-01-15T17:52:00.153 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.152+0000 7f28e389f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:00.155 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.155+0000 7f28e389f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:00.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.156+0000 7f28e389f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.158+0000 7f28e389f080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:52:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.158+0000 7f28e389f080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:52:00.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.158+0000 7f28e389f080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:52:00.213 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:00.213+0000 7f28e389f080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:52:04.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:52:04.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:52:04.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:52:04.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:52:04.750 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:52:05.273 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:52:05.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:52:05.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:05.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:52:05.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:52:05.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:52:05.294 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:05.293+0000 7f0e2183b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:05.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:05.319+0000 7f0e2183b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:05.320 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:05.320+0000 7f0e2183b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:05.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:52:05.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:52:05.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:52:05.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:52:05.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:52:05.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:05.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:52:05.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:05.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:52:05.673 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:52:05.837 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:05.837+0000 7f0e2183b080 -1 Falling back to public interface 2021-01-15T17:52:06.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:52:06.645 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:06.645+0000 7f0e2183b080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:52:07.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:52:07.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:07.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:52:07.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:07.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:52:07.092 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:07.523 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/591508,v1:127.0.0.1:6811/591508] [v2:127.0.0.1:6812/591508,v1:127.0.0.1:6813/591508] exists,up 88d80cb3-94c9-4141-8b23-0eec1b0bd870 2021-01-15T17:52:07.523 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 6db12abb-2d93-4170-ad46-62350b0d1f04 2021-01-15T17:52:07.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:52:07.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:52:07.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:52:07.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:236: TEST_backfill_out: run_osd td/osd-backfill-stats 2 2021-01-15T17:52:07.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:52:07.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:52:07.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:52:07.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:52:07.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:52:07.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:52:07.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:52:07.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:52:07.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:52:07.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:52:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:52:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:52:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:52:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:52:07.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:52:07.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:52:07.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:52:07.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:52:07.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:52:07.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:52:07.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:07.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:07.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:52:07.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:52:07.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:52:07.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:52:07.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:52:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:52:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:52:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:52:07.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:52:07.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:52:07.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:52:07.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6db12abb-2d93-4170-ad46-62350b0d1f04 2021-01-15T17:52:07.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 6db12abb-2d93-4170-ad46-62350b0d1f04' 2021-01-15T17:52:07.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:52:07.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBH1gFgCYYVIBAAzU6hhuu9fqrXIa4lMzh5cQ== 2021-01-15T17:52:07.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBH1gFgCYYVIBAAzU6hhuu9fqrXIa4lMzh5cQ=="}' 2021-01-15T17:52:07.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6db12abb-2d93-4170-ad46-62350b0d1f04 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:52:08.131 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:52:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:52:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBH1gFgCYYVIBAAzU6hhuu9fqrXIa4lMzh5cQ== --osd-uuid 6db12abb-2d93-4170-ad46-62350b0d1f04 2021-01-15T17:52:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.168+0000 7f767b619080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:08.170 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.170+0000 7f767b619080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:08.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.171+0000 7f767b619080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:08.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.172+0000 7f767b619080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:52:08.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.172+0000 7f767b619080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:52:08.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.172+0000 7f767b619080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:52:08.219 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:08.218+0000 7f767b619080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:52:12.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:52:12.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:52:12.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:52:12.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:52:12.812 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:52:13.333 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:52:13.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:52:13.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:13.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:52:13.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:52:13.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:52:13.356 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:13.356+0000 7f9402361080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:13.375 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:13.375+0000 7f9402361080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:13.377 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:13.376+0000 7f9402361080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:13.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:52:13.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:52:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:52:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:52:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:52:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:13.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:52:13.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:13.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:52:13.733 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:52:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:13.894+0000 7f9402361080 -1 Falling back to public interface 2021-01-15T17:52:14.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:52:14.685 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:14.684+0000 7f9402361080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:52:15.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:52:15.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:15.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:52:15.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:52:15.126 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:15.536 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/592772,v1:127.0.0.1:6819/592772] [v2:127.0.0.1:6820/592772,v1:127.0.0.1:6821/592772] exists,up 6db12abb-2d93-4170-ad46-62350b0d1f04 2021-01-15T17:52:15.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:52:15.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:52:15.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:52:15.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:237: TEST_backfill_out: run_osd td/osd-backfill-stats 3 2021-01-15T17:52:15.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:52:15.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:52:15.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:52:15.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:52:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:52:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:52:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:52:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:52:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:52:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:52:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:52:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:52:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:52:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:52:15.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:52:15.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:52:15.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:52:15.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:52:15.541 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:52:15.542 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:52:15.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:15.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:15.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:52:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:52:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:52:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:52:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:52:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:52:15.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:52:15.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=26fac9a8-7877-40c8-8409-439c45a11aa8 2021-01-15T17:52:15.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 26fac9a8-7877-40c8-8409-439c45a11aa8' 2021-01-15T17:52:15.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:52:15.546 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 26fac9a8-7877-40c8-8409-439c45a11aa8 2021-01-15T17:52:15.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBP1gFguHAFIRAAxxGuOnTvgjE4SCxCgcuZww== 2021-01-15T17:52:15.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBP1gFguHAFIRAAxxGuOnTvgjE4SCxCgcuZww=="}' 2021-01-15T17:52:15.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 26fac9a8-7877-40c8-8409-439c45a11aa8 -i td/osd-backfill-stats/3/new.json 2021-01-15T17:52:16.118 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:52:16.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:52:16.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBP1gFguHAFIRAAxxGuOnTvgjE4SCxCgcuZww== --osd-uuid 26fac9a8-7877-40c8-8409-439c45a11aa8 2021-01-15T17:52:16.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.154+0000 7f6854af1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:16.156 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.156+0000 7f6854af1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:16.158 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.157+0000 7f6854af1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:16.158 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.158+0000 7f6854af1080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:52:16.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.158+0000 7f6854af1080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:52:16.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.158+0000 7f6854af1080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:52:16.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:16.207+0000 7f6854af1080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:52:20.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:52:20.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:52:20.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:52:20.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:52:20.796 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:52:21.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:52:21.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:21.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:52:21.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:52:21.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:52:21.316 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:52:21.338 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:21.338+0000 7fc87f7c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:21.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:21.339+0000 7fc87f7c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:21.341 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:21.341+0000 7fc87f7c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:21.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:52:21.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:52:21.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:52:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:52:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:52:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:52:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:21.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:52:21.724 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:52:21.856 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:21.855+0000 7fc87f7c3080 -1 Falling back to public interface 2021-01-15T17:52:22.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:52:22.652 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:22.652+0000 7fc87f7c3080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:52:23.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:52:23.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:23.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:52:23.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:23.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:52:23.119 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:23.537 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/594036,v1:127.0.0.1:6827/594036] [v2:127.0.0.1:6828/594036,v1:127.0.0.1:6829/594036] exists,up 26fac9a8-7877-40c8-8409-439c45a11aa8 2021-01-15T17:52:23.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:52:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:52:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:52:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:238: TEST_backfill_out: run_osd td/osd-backfill-stats 4 2021-01-15T17:52:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:52:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:52:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:52:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:52:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:52:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:52:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:52:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:52:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:52:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:52:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:52:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:52:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:52:23.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:52:23.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:52:23.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:52:23.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:52:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:52:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:52:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:52:23.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:23.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:23.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:52:23.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:52:23.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:52:23.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:52:23.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:52:23.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:52:23.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:52:23.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:23.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:52:23.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:52:23.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:52:23.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:52:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=34515a60-ad53-4202-ac72-27ddee2bb4f6 2021-01-15T17:52:23.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 34515a60-ad53-4202-ac72-27ddee2bb4f6' 2021-01-15T17:52:23.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:52:23.546 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 34515a60-ad53-4202-ac72-27ddee2bb4f6 2021-01-15T17:52:23.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBX1gFgpCAAIRAAG3dz2q1fLL4xTyMzCav73A== 2021-01-15T17:52:23.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBX1gFgpCAAIRAAG3dz2q1fLL4xTyMzCav73A=="}' 2021-01-15T17:52:23.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 34515a60-ad53-4202-ac72-27ddee2bb4f6 -i td/osd-backfill-stats/4/new.json 2021-01-15T17:52:24.071 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:52:24.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:52:24.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBX1gFgpCAAIRAAG3dz2q1fLL4xTyMzCav73A== --osd-uuid 34515a60-ad53-4202-ac72-27ddee2bb4f6 2021-01-15T17:52:24.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.108+0000 7ff0e3cb3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:24.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.109+0000 7ff0e3cb3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:24.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.110+0000 7ff0e3cb3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:24.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.111+0000 7ff0e3cb3080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:52:24.112 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.111+0000 7ff0e3cb3080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:52:24.112 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.111+0000 7ff0e3cb3080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:52:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:24.218+0000 7ff0e3cb3080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:52:28.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:52:28.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:52:28.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:52:28.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:52:28.995 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:52:29.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:52:29.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:29.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:52:29.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:52:29.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:52:29.517 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:52:29.539 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:29.539+0000 7f1cd5220080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:29.541 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:29.541+0000 7f1cd5220080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:29.541+0000 7f1cd5220080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:29.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:52:29.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:52:29.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:52:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:52:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:52:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:52:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:29.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:52:29.914 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:52:30.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:30.061+0000 7f1cd5220080 -1 Falling back to public interface 2021-01-15T17:52:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:52:30.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:30.845+0000 7f1cd5220080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:52:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:52:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:52:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:52:31.320 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:31.747 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/595300,v1:127.0.0.1:6835/595300] [v2:127.0.0.1:6836/595300,v1:127.0.0.1:6837/595300] exists,up 34515a60-ad53-4202-ac72-27ddee2bb4f6 2021-01-15T17:52:31.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:52:31.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:52:31.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:52:31.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:239: TEST_backfill_out: run_osd td/osd-backfill-stats 5 2021-01-15T17:52:31.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:52:31.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:52:31.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:52:31.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:52:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:52:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:52:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:52:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:52:31.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:52:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:52:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:52:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:52:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:52:31.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:52:31.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:52:31.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:52:31.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:52:31.751 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:52:31.751 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:52:31.752 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:52:31.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:31.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:52:31.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:52:31.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:52:31.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:52:31.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:31.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:52:31.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:52:31.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:52:31.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:52:31.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d7f50c99-8895-4675-bb24-f3020369c7ab 2021-01-15T17:52:31.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 d7f50c99-8895-4675-bb24-f3020369c7ab' 2021-01-15T17:52:31.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:52:31.756 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 d7f50c99-8895-4675-bb24-f3020369c7ab 2021-01-15T17:52:31.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBf1gFgPOp7LRAADRo6aZnvWEtZHhzZtcWR8A== 2021-01-15T17:52:31.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBf1gFgPOp7LRAADRo6aZnvWEtZHhzZtcWR8A=="}' 2021-01-15T17:52:31.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d7f50c99-8895-4675-bb24-f3020369c7ab -i td/osd-backfill-stats/5/new.json 2021-01-15T17:52:32.324 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:52:32.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:52:32.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBf1gFgPOp7LRAADRo6aZnvWEtZHhzZtcWR8A== --osd-uuid d7f50c99-8895-4675-bb24-f3020369c7ab 2021-01-15T17:52:32.361 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.360+0000 7f03d288b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:32.362 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.362+0000 7f03d288b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.364+0000 7f03d288b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:32.365 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.364+0000 7f03d288b080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:52:32.365 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.364+0000 7f03d288b080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:52:32.365 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.364+0000 7f03d288b080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:52:32.402 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:32.402+0000 7f03d288b080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:52:37.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:52:37.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:52:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:52:37.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:52:37.189 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:52:37.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:52:37.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:52:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:52:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:52:37.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:52:37.720 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:52:37.741 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:37.740+0000 7fb5de31d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:37.761 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:37.761+0000 7fb5de31d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:37.763 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:37.762+0000 7fb5de31d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:52:38.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:52:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:52:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:52:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:52:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:52:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:52:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:52:38.117 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:52:38.278 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:38.278+0000 7fb5de31d080 -1 Falling back to public interface 2021-01-15T17:52:38.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:52:39.114 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:52:39.113+0000 7fb5de31d080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:52:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:52:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:52:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:52:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:52:39.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:52:39.511 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:52:39.915 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/596572,v1:127.0.0.1:6843/596572] [v2:127.0.0.1:6844/596572,v1:127.0.0.1:6845/596572] exists,up d7f50c99-8895-4675-bb24-f3020369c7ab 2021-01-15T17:52:39.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:52:39.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:52:39.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:52:39.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:241: TEST_backfill_out: create_pool test 1 1 2021-01-15T17:52:39.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T17:52:40.593 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:52:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:52:41.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:242: TEST_backfill_out: ceph osd pool set test size 2 2021-01-15T17:52:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:52:42.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:243: TEST_backfill_out: sleep 5 2021-01-15T17:52:47.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:245: TEST_backfill_out: wait_for_clean 2021-01-15T17:52:47.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:52:47.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:52:47.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:52:47.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:52:47.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:52:47.536 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:52:47.536 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:52:47.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:52:47.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:52:47.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:52:47.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:52:47.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:52:47.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:52:47.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:52:47.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:52:48.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:52:48.225 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:52:48.225 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:52:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:52:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:52:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:52:48.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:52:48.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:52:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836492 2021-01-15T17:52:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836492 2021-01-15T17:52:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492' 2021-01-15T17:52:48.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:52:48.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T17:52:48.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T17:52:48.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971' 2021-01-15T17:52:48.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:52:48.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509450 2021-01-15T17:52:48.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509450 2021-01-15T17:52:48.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450' 2021-01-15T17:52:48.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:52:48.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:52:48.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:52:48.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927' 2021-01-15T17:52:48.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:52:48.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T17:52:48.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T17:52:48.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927 4-107374182406' 2021-01-15T17:52:48.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:52:48.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:52:49.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T17:52:49.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T17:52:49.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836492 1-42949672971 2-64424509450 3-85899345927 4-107374182406 5-128849018884' 2021-01-15T17:52:49.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:49.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836492 2021-01-15T17:52:49.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:49.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:52:49.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836492 2021-01-15T17:52:49.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:49.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836492 2021-01-15T17:52:49.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-01-15T17:52:49.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:52:49.051 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836492 2021-01-15T17:52:49.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836492 2021-01-15T17:52:49.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:52:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:52:50.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:52:50.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836492 2021-01-15T17:52:50.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:50.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T17:52:50.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:50.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:52:50.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T17:52:50.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:50.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T17:52:50.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T17:52:50.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:52:50.837 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T17:52:51.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T17:52:51.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:51.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509450 2021-01-15T17:52:51.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:51.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:52:51.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509450 2021-01-15T17:52:51.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:51.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509450 2021-01-15T17:52:51.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-01-15T17:52:51.237 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509450 2021-01-15T17:52:51.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:52:51.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509450 2021-01-15T17:52:51.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:51.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:52:51.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:51.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:52:51.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:52:51.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:51.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:52:51.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:52:51.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:52:51.630 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:52:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2021-01-15T17:52:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:52.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182406 2021-01-15T17:52:52.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:52.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:52:52.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182406 2021-01-15T17:52:52.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:52.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T17:52:52.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-01-15T17:52:52.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:52:52.040 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182406 2021-01-15T17:52:52.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2021-01-15T17:52:52.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:52:52.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T17:52:52.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:52:52.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:52:52.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T17:52:52.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:52:52.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T17:52:52.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T17:52:52.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:52:52.432 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T17:52:52.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2021-01-15T17:52:52.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:52:52.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:52:52.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:52:53.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:52:53.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:52:53.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:52:53.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:52:53.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:52:53.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:52:53.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:52:53.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:52:53.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:52:53.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:52:53.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:52:53.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:52:54.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:52:54.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:52:54.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:52:54.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: seq 1 200 2021-01-15T17:52:54.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj1 /dev/null 2021-01-15T17:52:54.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj2 /dev/null 2021-01-15T17:52:54.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj3 /dev/null 2021-01-15T17:52:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj4 /dev/null 2021-01-15T17:52:54.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj5 /dev/null 2021-01-15T17:52:54.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj6 /dev/null 2021-01-15T17:52:54.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj7 /dev/null 2021-01-15T17:52:54.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:54.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj8 /dev/null 2021-01-15T17:52:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj9 /dev/null 2021-01-15T17:52:55.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj10 /dev/null 2021-01-15T17:52:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj11 /dev/null 2021-01-15T17:52:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj12 /dev/null 2021-01-15T17:52:55.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj13 /dev/null 2021-01-15T17:52:55.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj14 /dev/null 2021-01-15T17:52:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj15 /dev/null 2021-01-15T17:52:55.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj16 /dev/null 2021-01-15T17:52:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj17 /dev/null 2021-01-15T17:52:55.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:55.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj18 /dev/null 2021-01-15T17:52:56.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj19 /dev/null 2021-01-15T17:52:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj20 /dev/null 2021-01-15T17:52:56.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj21 /dev/null 2021-01-15T17:52:56.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj22 /dev/null 2021-01-15T17:52:56.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj23 /dev/null 2021-01-15T17:52:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj24 /dev/null 2021-01-15T17:52:56.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj25 /dev/null 2021-01-15T17:52:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj26 /dev/null 2021-01-15T17:52:57.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj27 /dev/null 2021-01-15T17:52:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj28 /dev/null 2021-01-15T17:52:57.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj29 /dev/null 2021-01-15T17:52:57.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj30 /dev/null 2021-01-15T17:52:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj31 /dev/null 2021-01-15T17:52:57.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj32 /dev/null 2021-01-15T17:52:57.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj33 /dev/null 2021-01-15T17:52:57.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj34 /dev/null 2021-01-15T17:52:57.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj35 /dev/null 2021-01-15T17:52:57.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:57.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj36 /dev/null 2021-01-15T17:52:58.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj37 /dev/null 2021-01-15T17:52:58.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj38 /dev/null 2021-01-15T17:52:58.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj39 /dev/null 2021-01-15T17:52:58.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj40 /dev/null 2021-01-15T17:52:58.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj41 /dev/null 2021-01-15T17:52:58.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj42 /dev/null 2021-01-15T17:52:58.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj43 /dev/null 2021-01-15T17:52:58.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj44 /dev/null 2021-01-15T17:52:58.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj45 /dev/null 2021-01-15T17:52:58.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj46 /dev/null 2021-01-15T17:52:58.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:58.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj47 /dev/null 2021-01-15T17:52:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj48 /dev/null 2021-01-15T17:52:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj49 /dev/null 2021-01-15T17:52:59.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj50 /dev/null 2021-01-15T17:52:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj51 /dev/null 2021-01-15T17:52:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj52 /dev/null 2021-01-15T17:52:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj53 /dev/null 2021-01-15T17:52:59.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj54 /dev/null 2021-01-15T17:52:59.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj55 /dev/null 2021-01-15T17:52:59.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:52:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj56 /dev/null 2021-01-15T17:53:00.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj57 /dev/null 2021-01-15T17:53:00.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj58 /dev/null 2021-01-15T17:53:00.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj59 /dev/null 2021-01-15T17:53:00.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj60 /dev/null 2021-01-15T17:53:00.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj61 /dev/null 2021-01-15T17:53:00.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj62 /dev/null 2021-01-15T17:53:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj63 /dev/null 2021-01-15T17:53:00.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj64 /dev/null 2021-01-15T17:53:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj65 /dev/null 2021-01-15T17:53:00.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj66 /dev/null 2021-01-15T17:53:00.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:00.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj67 /dev/null 2021-01-15T17:53:01.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj68 /dev/null 2021-01-15T17:53:01.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj69 /dev/null 2021-01-15T17:53:01.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj70 /dev/null 2021-01-15T17:53:01.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj71 /dev/null 2021-01-15T17:53:01.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj72 /dev/null 2021-01-15T17:53:01.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj73 /dev/null 2021-01-15T17:53:01.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj74 /dev/null 2021-01-15T17:53:01.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj75 /dev/null 2021-01-15T17:53:01.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:01.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj76 /dev/null 2021-01-15T17:53:01.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj77 /dev/null 2021-01-15T17:53:02.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj78 /dev/null 2021-01-15T17:53:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj79 /dev/null 2021-01-15T17:53:02.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj80 /dev/null 2021-01-15T17:53:02.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj81 /dev/null 2021-01-15T17:53:02.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj82 /dev/null 2021-01-15T17:53:02.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj83 /dev/null 2021-01-15T17:53:02.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj84 /dev/null 2021-01-15T17:53:02.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj85 /dev/null 2021-01-15T17:53:02.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj86 /dev/null 2021-01-15T17:53:02.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:02.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj87 /dev/null 2021-01-15T17:53:03.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj88 /dev/null 2021-01-15T17:53:03.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj89 /dev/null 2021-01-15T17:53:03.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj90 /dev/null 2021-01-15T17:53:03.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj91 /dev/null 2021-01-15T17:53:03.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj92 /dev/null 2021-01-15T17:53:03.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj93 /dev/null 2021-01-15T17:53:03.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj94 /dev/null 2021-01-15T17:53:03.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj95 /dev/null 2021-01-15T17:53:03.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj96 /dev/null 2021-01-15T17:53:03.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:03.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj97 /dev/null 2021-01-15T17:53:04.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj98 /dev/null 2021-01-15T17:53:04.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj99 /dev/null 2021-01-15T17:53:04.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj100 /dev/null 2021-01-15T17:53:04.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj101 /dev/null 2021-01-15T17:53:04.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj102 /dev/null 2021-01-15T17:53:04.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj103 /dev/null 2021-01-15T17:53:04.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj104 /dev/null 2021-01-15T17:53:04.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj105 /dev/null 2021-01-15T17:53:04.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj106 /dev/null 2021-01-15T17:53:04.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:04.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj107 /dev/null 2021-01-15T17:53:05.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj108 /dev/null 2021-01-15T17:53:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj109 /dev/null 2021-01-15T17:53:05.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj110 /dev/null 2021-01-15T17:53:05.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj111 /dev/null 2021-01-15T17:53:05.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj112 /dev/null 2021-01-15T17:53:05.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj113 /dev/null 2021-01-15T17:53:05.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj114 /dev/null 2021-01-15T17:53:05.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj115 /dev/null 2021-01-15T17:53:05.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj116 /dev/null 2021-01-15T17:53:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj117 /dev/null 2021-01-15T17:53:06.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj118 /dev/null 2021-01-15T17:53:06.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj119 /dev/null 2021-01-15T17:53:06.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj120 /dev/null 2021-01-15T17:53:06.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj121 /dev/null 2021-01-15T17:53:06.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj122 /dev/null 2021-01-15T17:53:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj123 /dev/null 2021-01-15T17:53:06.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj124 /dev/null 2021-01-15T17:53:06.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj125 /dev/null 2021-01-15T17:53:06.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:06.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj126 /dev/null 2021-01-15T17:53:07.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj127 /dev/null 2021-01-15T17:53:07.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj128 /dev/null 2021-01-15T17:53:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj129 /dev/null 2021-01-15T17:53:07.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj130 /dev/null 2021-01-15T17:53:07.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj131 /dev/null 2021-01-15T17:53:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj132 /dev/null 2021-01-15T17:53:07.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj133 /dev/null 2021-01-15T17:53:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj134 /dev/null 2021-01-15T17:53:07.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj135 /dev/null 2021-01-15T17:53:07.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:07.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj136 /dev/null 2021-01-15T17:53:08.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj137 /dev/null 2021-01-15T17:53:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj138 /dev/null 2021-01-15T17:53:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj139 /dev/null 2021-01-15T17:53:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj140 /dev/null 2021-01-15T17:53:08.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj141 /dev/null 2021-01-15T17:53:08.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj142 /dev/null 2021-01-15T17:53:08.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj143 /dev/null 2021-01-15T17:53:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj144 /dev/null 2021-01-15T17:53:08.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj145 /dev/null 2021-01-15T17:53:08.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:08.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj146 /dev/null 2021-01-15T17:53:09.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj147 /dev/null 2021-01-15T17:53:09.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj148 /dev/null 2021-01-15T17:53:09.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj149 /dev/null 2021-01-15T17:53:09.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj150 /dev/null 2021-01-15T17:53:09.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj151 /dev/null 2021-01-15T17:53:09.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj152 /dev/null 2021-01-15T17:53:09.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj153 /dev/null 2021-01-15T17:53:09.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj154 /dev/null 2021-01-15T17:53:09.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj155 /dev/null 2021-01-15T17:53:09.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:09.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj156 /dev/null 2021-01-15T17:53:10.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj157 /dev/null 2021-01-15T17:53:10.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj158 /dev/null 2021-01-15T17:53:10.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj159 /dev/null 2021-01-15T17:53:10.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj160 /dev/null 2021-01-15T17:53:10.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj161 /dev/null 2021-01-15T17:53:10.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj162 /dev/null 2021-01-15T17:53:10.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj163 /dev/null 2021-01-15T17:53:10.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj164 /dev/null 2021-01-15T17:53:10.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj165 /dev/null 2021-01-15T17:53:10.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:10.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj166 /dev/null 2021-01-15T17:53:11.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj167 /dev/null 2021-01-15T17:53:11.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj168 /dev/null 2021-01-15T17:53:11.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj169 /dev/null 2021-01-15T17:53:11.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj170 /dev/null 2021-01-15T17:53:11.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj171 /dev/null 2021-01-15T17:53:11.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj172 /dev/null 2021-01-15T17:53:11.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj173 /dev/null 2021-01-15T17:53:11.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj174 /dev/null 2021-01-15T17:53:11.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj175 /dev/null 2021-01-15T17:53:11.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:11.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj176 /dev/null 2021-01-15T17:53:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj177 /dev/null 2021-01-15T17:53:12.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj178 /dev/null 2021-01-15T17:53:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj179 /dev/null 2021-01-15T17:53:12.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj180 /dev/null 2021-01-15T17:53:12.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj181 /dev/null 2021-01-15T17:53:12.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj182 /dev/null 2021-01-15T17:53:12.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj183 /dev/null 2021-01-15T17:53:12.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj184 /dev/null 2021-01-15T17:53:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:12.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj185 /dev/null 2021-01-15T17:53:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj186 /dev/null 2021-01-15T17:53:13.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj187 /dev/null 2021-01-15T17:53:13.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj188 /dev/null 2021-01-15T17:53:13.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj189 /dev/null 2021-01-15T17:53:13.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj190 /dev/null 2021-01-15T17:53:13.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj191 /dev/null 2021-01-15T17:53:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj192 /dev/null 2021-01-15T17:53:13.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj193 /dev/null 2021-01-15T17:53:13.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj194 /dev/null 2021-01-15T17:53:13.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:13.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj195 /dev/null 2021-01-15T17:53:14.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:14.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj196 /dev/null 2021-01-15T17:53:14.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:14.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj197 /dev/null 2021-01-15T17:53:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj198 /dev/null 2021-01-15T17:53:14.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:14.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj199 /dev/null 2021-01-15T17:53:14.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:247: TEST_backfill_out: for i in $(seq 1 $objects) 2021-01-15T17:53:14.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:249: TEST_backfill_out: rados -p test put obj200 /dev/null 2021-01-15T17:53:14.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:252: TEST_backfill_out: get_pg test obj1 2021-01-15T17:53:14.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:53:14.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:53:14.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:53:14.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:53:14.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:252: TEST_backfill_out: local PG=1.0 2021-01-15T17:53:14.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:254: TEST_backfill_out: get_primary test obj1 2021-01-15T17:53:14.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:53:14.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:53:14.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:53:14.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:53:15.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:254: TEST_backfill_out: local primary=1 2021-01-15T17:53:15.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:256: TEST_backfill_out: ceph osd set nobackfill 2021-01-15T17:53:16.049 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:53:16.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:257: TEST_backfill_out: get_not_primary test obj1 2021-01-15T17:53:16.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:53:16.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:53:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:53:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:53:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:53:16.062 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:53:16.062 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:53:16.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:53:16.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:53:16.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:53:16.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:257: TEST_backfill_out: ceph osd out osd.0 2021-01-15T17:53:17.417 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:53:17.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:258: TEST_backfill_out: sleep 2 2021-01-15T17:53:19.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:259: TEST_backfill_out: ceph osd unset nobackfill 2021-01-15T17:53:20.272 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:53:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:261: TEST_backfill_out: wait_for_clean 2021-01-15T17:53:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:53:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:53:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:53:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:53:20.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:53:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:53:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:53:20.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:53:20.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:53:20.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:53:20.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:53:20.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:53:20.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:53:20.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:53:20.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:53:20.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:53:20.768 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:53:20.768 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:53:20.768 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:53:20.768 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:53:20.769 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:53:20.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:53:20.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:20.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:53:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836501 2021-01-15T17:53:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836501 2021-01-15T17:53:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501' 2021-01-15T17:53:20.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:20.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:53:21.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T17:53:21.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T17:53:21.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2021-01-15T17:53:21.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:21.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:53:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509459 2021-01-15T17:53:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509459 2021-01-15T17:53:21.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509459' 2021-01-15T17:53:21.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:21.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:53:21.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2021-01-15T17:53:21.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2021-01-15T17:53:21.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509459 3-85899345936' 2021-01-15T17:53:21.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:21.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:53:21.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T17:53:21.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T17:53:21.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509459 3-85899345936 4-107374182414' 2021-01-15T17:53:21.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:53:21.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:53:21.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018893 2021-01-15T17:53:21.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018893 2021-01-15T17:53:21.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509459 3-85899345936 4-107374182414 5-128849018893' 2021-01-15T17:53:21.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:21.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836501 2021-01-15T17:53:21.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:21.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:53:21.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836501 2021-01-15T17:53:21.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:21.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836501 2021-01-15T17:53:21.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2021-01-15T17:53:21.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:53:21.654 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836501 2021-01-15T17:53:22.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836501 -lt 21474836501 2021-01-15T17:53:22.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:22.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T17:53:22.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:22.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:53:22.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T17:53:22.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:22.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T17:53:22.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T17:53:22.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:53:22.076 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T17:53:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672979 2021-01-15T17:53:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:22.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509459 2021-01-15T17:53:22.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:53:22.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509459 2021-01-15T17:53:22.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509459 2021-01-15T17:53:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-01-15T17:53:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:53:22.476 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509459 2021-01-15T17:53:22.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509459 -lt 64424509459 2021-01-15T17:53:22.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:22.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2021-01-15T17:53:22.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:22.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:53:22.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2021-01-15T17:53:22.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:22.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2021-01-15T17:53:22.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-01-15T17:53:22.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:53:22.869 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345936 2021-01-15T17:53:23.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345936 2021-01-15T17:53:23.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:23.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T17:53:23.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:23.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:53:23.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T17:53:23.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T17:53:23.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T17:53:23.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:53:23.270 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T17:53:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2021-01-15T17:53:23.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:53:23.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018893 2021-01-15T17:53:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:53:23.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:53:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018893 2021-01-15T17:53:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:53:23.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018893 2021-01-15T17:53:23.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-01-15T17:53:23.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:53:23.667 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018893 2021-01-15T17:53:24.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018893 -lt 128849018893 2021-01-15T17:53:24.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:53:24.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:24.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:24.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:53:24.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:24.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:24.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:24.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:24.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:24.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:24.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:24.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:24.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:24.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:24.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:25.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:25.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:53:25.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:53:25.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:53:25.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:25.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:25.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:25.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:25.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:25.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:25.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:25.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:25.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:25.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:25.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:25.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:25.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:25.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:26.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:26.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:26.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:26.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:26.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:26.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:26.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:26.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:26.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:53:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:53:27.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:27.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:27.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:27.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:27.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:27.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:27.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:27.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:27.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:28.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:28.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:28.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:28.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:28.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:28.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:28.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:28.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:28.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:28.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2021-01-15T17:53:28.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2021-01-15T17:53:28.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:28.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:28.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:28.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:28.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:28.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:28.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:28.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:28.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:28.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:28.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:29.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:29.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:29.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:29.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:29.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:29.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:29.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:29.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:29.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:30.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:53:30.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:53:30.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:30.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:30.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:30.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:30.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:30.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:30.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:30.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:30.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:30.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:30.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:30.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:30.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:30.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:30.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:31.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:31.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:31.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:31.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:31.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:31.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:31.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:31.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:31.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:31.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:53:31.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:53:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:31.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:31.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:31.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:31.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:31.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:31.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:31.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:31.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:32.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:32.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:32.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:32.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:32.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:32.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:32.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:32.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:32.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:32.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:32.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:32.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:32.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:53:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:53:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:33.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:33.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:33.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:33.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:33.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:33.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:33.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:33.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:33.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:34.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:34.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:34.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:53:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:53:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:34.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:34.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:34.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:34.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:34.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:34.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:34.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:34.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:34.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:35.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:35.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:35.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:35.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:35.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:35.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:35.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:35.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:35.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:36.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:36.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:36.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:36.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:36.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:36.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:36.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:36.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:36.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:36.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:36.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:36.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:36.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:36.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:36.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:36.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:36.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:37.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:37.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:37.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:37.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:37.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:37.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:37.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:37.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:37.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:37.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:37.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:38.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:38.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:38.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:38.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:38.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:38.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:39.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:39.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:39.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:39.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:39.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:39.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:39.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:39.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:39.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:39.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:39.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:39.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:39.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:39.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:39.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:39.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:39.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:40.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:40.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:40.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:40.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:40.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:40.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:40.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:53:40.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:53:40.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:40.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:40.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:40.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:40.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:40.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:40.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:40.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:40.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:40.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:40.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:41.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:41.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:41.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:41.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:41.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:41.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:41.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:41.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:41.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:41.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:41.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:41.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:42.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:42.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:42.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:42.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:42.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:42.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:42.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:42.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:42.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:42.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:42.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:42.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:42.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:42.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:42.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:42.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:42.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:43.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:43.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:43.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:43.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:43.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:43.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:43.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:53:43.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:53:43.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:43.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:43.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:44.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:44.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:44.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:44.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:44.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:44.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:44.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:44.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:44.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:44.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:44.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:44.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:44.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:44.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:44.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:44.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:44.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:44.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:44.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:44.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:44.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:53:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:53:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:45.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:45.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:45.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:45.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:45.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:45.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:45.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:45.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:45.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:45.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:45.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:45.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:45.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:45.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:46.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:46.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:46.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:46.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:46.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:46.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:46.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:46.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:46.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:46.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:46.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:47.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:47.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:47.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:47.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:47.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:47.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:47.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:47.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:47.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:47.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:47.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:47.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:47.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:47.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:47.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:47.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:47.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:47.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:47.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:47.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:47.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:48.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:53:48.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:53:48.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:48.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:48.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:48.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:48.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:48.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:48.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:48.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:48.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:48.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:48.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:48.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:48.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:48.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:48.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:49.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:49.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:49.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:49.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:49.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:49.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:49.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:49.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:49.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:49.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:53:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:53:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:50.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:50.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:50.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:50.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:50.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:50.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:50.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:50.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:50.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:50.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:50.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:50.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:51.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:51.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:51.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:51.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:51.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:51.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:51.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:53:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:53:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:51.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:51.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:51.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:51.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:51.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:51.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:51.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:51.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:51.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:52.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:52.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:52.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:52.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:52.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:52.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:52.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:52.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:52.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:52.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:53.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:53:53.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:53:53.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:53.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:53.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:53.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:53.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:53.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:53.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:53.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:53.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:53.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:53.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:53.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:53:53.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:53.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:54.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:53:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:53:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:53:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:53:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:53:54.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:53:54.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:53:54.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:53:54.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:53:54.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:53:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:53:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:53:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:53:54.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:53:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:53:54.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:53:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:53:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:53:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:53:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:53:54.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:53:54.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:53:55.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:53:55.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:53:55.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:53:55.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:53:55.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:53:55.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:53:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:53:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:263: TEST_backfill_out: check td/osd-backfill-stats 1.0 1 replicated 0 0 200 0 2021-01-15T17:53:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:53:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:53:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:53:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T17:53:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2021-01-15T17:53:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:53:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2021-01-15T17:53:55.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:53:55.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:53:55.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:53:55.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:53:55.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:53:55.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T17:53:55.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:53:55.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:53:55.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:53:55.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T17:53:55.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:53:55.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:53:55.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:53:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,2]/[1,0]' 2021-01-15T17:53:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,2]/[1,0]' ']' 2021-01-15T17:53:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:53:55.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:53:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,2]/[1,0] ' 2021-01-15T17:53:55.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:53:55.654 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:53:55.655 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:53:55.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2021-01-15T17:53:55.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2021-01-15T17:53:55.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2021-01-15T17:53:55.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:53:55.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:53:55.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:53:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:53:55.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:53:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:53:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:53:55.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2021-01-15T17:53:55.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2021-01-15T17:53:55.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2021-01-15T17:53:55.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:53:55.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:53:55.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:53:55.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:53:55.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:53:55.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:53:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,2]/[1,0] ' 2021-01-15T17:53:55.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:53:55.669 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:53:55.669 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:53:55.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2021-01-15T17:53:55.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2021-01-15T17:53:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2021-01-15T17:53:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:53:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:53:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:53:55.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:53:55.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:53:55.673 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:53:55.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:53:55.674 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:53:55.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2021-01-15T17:53:55.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2021-01-15T17:53:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2021-01-15T17:53:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:53:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:53:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:53:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:53:55.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:53:55.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:265: TEST_backfill_out: delete_pool test 2021-01-15T17:53:55.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:53:55.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:53:56.157 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:53:56.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:266: TEST_backfill_out: kill_daemons td/osd-backfill-stats 2021-01-15T17:53:56.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:53:56.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:53:56.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:53:56.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:53:56.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:53:56.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:53:56.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:53:56.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:53:56.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:53:56.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:53:56.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:53:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:53:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:53:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:53:56.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:53:56.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:53:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:53:56.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:53:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:53:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:53:56.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:53:56.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:53:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:53:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:53:56.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:53:56.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:53:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:53:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:53:56.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:53:56.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:53:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:53:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:53:56.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:53:56.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:53:56.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:53:56.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:53:56.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:53:56.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:53:56.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:53:56.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:53:56.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:53:56.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:53:56.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:53:56.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:53:56.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:53:56.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:53:56.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:53:56.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:53:56.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:53:56.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:53:56.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:53:56.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:53:56.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:53:56.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:53:56.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:53:56.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:53:56.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:53:56.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:53:56.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:53:56.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:53:56.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:53:56.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:53:56.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:53:56.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:53:56.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:53:56.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:53:56.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:53:56.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_out2 td/osd-backfill-stats 2021-01-15T17:53:56.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:329: TEST_backfill_out2: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:331: TEST_backfill_out2: run_mon td/osd-backfill-stats a 2021-01-15T17:53:56.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:53:56.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:53:56.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:53:56.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:53:56.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:53:56.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:53:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:53:56.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:53:56.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:53:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:53:56.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:53:56.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:53:56.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:53:56.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:53:56.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:53:56.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:53:56.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:53:56.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:53:56.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:53:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:53:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:53:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:53:56.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:53:56.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:53:56.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:53:56.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:53:56.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:53:56.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:53:56.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:53:56.895 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:53:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:53:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:53:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:53:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:56.897 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:53:56.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:53:56.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:53:56.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:332: TEST_backfill_out2: run_mgr td/osd-backfill-stats x 2021-01-15T17:53:56.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:53:56.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:53:56.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:53:56.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:53:56.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:53:56.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:53:57.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:53:57.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:53:57.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:53:57.502 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:53:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:57.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:53:57.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:53:57.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:53:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:333: TEST_backfill_out2: run_osd td/osd-backfill-stats 0 2021-01-15T17:53:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:53:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:53:57.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:53:57.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:53:57.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:53:57.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:53:57.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:53:57.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:53:57.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:53:57.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:53:57.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:53:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:53:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:53:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:53:57.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:53:57.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:53:57.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:53:57.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:53:57.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:53:57.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:53:57.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:53:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:53:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:53:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:53:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:53:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:53:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:53:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:53:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:53:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:53:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:53:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:53:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:53:57.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:53:57.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f09e2451-5258-4991-ac46-acc3ebbf56a7 2021-01-15T17:53:57.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 f09e2451-5258-4991-ac46-acc3ebbf56a7' 2021-01-15T17:53:57.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:53:57.537 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 f09e2451-5258-4991-ac46-acc3ebbf56a7 2021-01-15T17:53:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC11gFgQlNzIBAAcTHwBTO6y7QYNaYzRUNB7A== 2021-01-15T17:53:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC11gFgQlNzIBAAcTHwBTO6y7QYNaYzRUNB7A=="}' 2021-01-15T17:53:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f09e2451-5258-4991-ac46-acc3ebbf56a7 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:53:57.884 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:53:57.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:53:57.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC11gFgQlNzIBAAcTHwBTO6y7QYNaYzRUNB7A== --osd-uuid f09e2451-5258-4991-ac46-acc3ebbf56a7 2021-01-15T17:53:57.920 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.920+0000 7fd45d129080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:53:57.922 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.922+0000 7fd45d129080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:53:57.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.923+0000 7fd45d129080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:53:57.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.924+0000 7fd45d129080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:53:57.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.924+0000 7fd45d129080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:53:57.925 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.924+0000 7fd45d129080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:53:57.966 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:53:57.965+0000 7fd45d129080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:54:02.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:54:02.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:02.567 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:54:02.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:54:02.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:03.093 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:54:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:54:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:03.116 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:03.116+0000 7fe89c9a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:03.131 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:03.131+0000 7fe89c9a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:03.132+0000 7fe89c9a3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:03.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:54:03.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:03.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:54:03.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:03.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:03.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:03.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:03.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:03.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:54:03.488 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:03.661 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:03.661+0000 7fe89c9a3080 -1 Falling back to public interface 2021-01-15T17:54:03.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:04.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:04.458+0000 7fe89c9a3080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:54:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:04.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:04.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:54:04.881 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:05.326 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/605088,v1:127.0.0.1:6803/605088] [v2:127.0.0.1:6804/605088,v1:127.0.0.1:6805/605088] exists,up f09e2451-5258-4991-ac46-acc3ebbf56a7 2021-01-15T17:54:05.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:05.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:05.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:05.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:334: TEST_backfill_out2: run_osd td/osd-backfill-stats 1 2021-01-15T17:54:05.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:54:05.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:54:05.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:54:05.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:54:05.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:54:05.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:54:05.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:54:05.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:54:05.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:54:05.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:54:05.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:54:05.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:54:05.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:54:05.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:54:05.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:54:05.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:54:05.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:54:05.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:54:05.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:54:05.331 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:54:05.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:54:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:54:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:54:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:54:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:54:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:54:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:54:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:54:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:54:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:54:05.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:54:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5c435a74-c19a-4695-a55d-55c51329927d 2021-01-15T17:54:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 5c435a74-c19a-4695-a55d-55c51329927d' 2021-01-15T17:54:05.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:54:05.335 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 5c435a74-c19a-4695-a55d-55c51329927d 2021-01-15T17:54:05.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC91gFgbb9lFBAAqzBsoY0ULpbHaM8Mj9JVTA== 2021-01-15T17:54:05.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC91gFgbb9lFBAAqzBsoY0ULpbHaM8Mj9JVTA=="}' 2021-01-15T17:54:05.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5c435a74-c19a-4695-a55d-55c51329927d -i td/osd-backfill-stats/1/new.json 2021-01-15T17:54:05.862 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:05.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:54:05.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC91gFgbb9lFBAAqzBsoY0ULpbHaM8Mj9JVTA== --osd-uuid 5c435a74-c19a-4695-a55d-55c51329927d 2021-01-15T17:54:05.899 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.898+0000 7fa83ce6d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:05.900 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.900+0000 7fa83ce6d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:05.901 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.901+0000 7fa83ce6d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:05.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.902+0000 7fa83ce6d080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:54:05.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.902+0000 7fa83ce6d080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:54:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.902+0000 7fa83ce6d080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:54:05.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:05.955+0000 7fa83ce6d080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:54:10.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:54:10.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:10.516 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:54:10.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:54:10.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:11.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:54:11.041 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:54:11.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:11.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:11.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:11.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:11.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:11.064+0000 7f28d4f29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:11.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:11.086+0000 7f28d4f29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:11.089 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:11.089+0000 7f28d4f29080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:11.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:54:11.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:11.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:54:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:11.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:11.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:54:11.442 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:11.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:11.605+0000 7f28d4f29080 -1 Falling back to public interface 2021-01-15T17:54:11.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:12.398 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:12.397+0000 7f28d4f29080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:54:12.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:12.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:12.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:12.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:12.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:54:12.852 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:13.265 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/606352,v1:127.0.0.1:6811/606352] [v2:127.0.0.1:6812/606352,v1:127.0.0.1:6813/606352] exists,up 5c435a74-c19a-4695-a55d-55c51329927d 2021-01-15T17:54:13.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:13.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:13.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:13.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:335: TEST_backfill_out2: run_osd td/osd-backfill-stats 2 2021-01-15T17:54:13.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:54:13.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:54:13.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:54:13.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:54:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:54:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:54:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:54:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:54:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:54:13.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:54:13.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:54:13.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:54:13.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:54:13.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:54:13.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:54:13.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:54:13.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:54:13.270 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:54:13.271 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:54:13.271 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:54:13.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:13.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:54:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:54:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:54:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:54:13.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:54:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:54:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:54:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:54:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:54:13.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:54:13.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:54:13.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=adcf2a91-009f-4d41-ae80-ccbaee4e2f70 2021-01-15T17:54:13.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 adcf2a91-009f-4d41-ae80-ccbaee4e2f70' 2021-01-15T17:54:13.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:54:13.275 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 adcf2a91-009f-4d41-ae80-ccbaee4e2f70 2021-01-15T17:54:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDF1gFgLFfGEBAAKsuF8xTzKt5z3PhpIa02YQ== 2021-01-15T17:54:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDF1gFgLFfGEBAAKsuF8xTzKt5z3PhpIa02YQ=="}' 2021-01-15T17:54:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new adcf2a91-009f-4d41-ae80-ccbaee4e2f70 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:54:13.806 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:54:13.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:54:13.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDF1gFgLFfGEBAAKsuF8xTzKt5z3PhpIa02YQ== --osd-uuid adcf2a91-009f-4d41-ae80-ccbaee4e2f70 2021-01-15T17:54:13.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.843+0000 7f3142433080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:13.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.845+0000 7f3142433080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:13.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.846+0000 7f3142433080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:13.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.847+0000 7f3142433080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:54:13.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.847+0000 7f3142433080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:54:13.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.847+0000 7f3142433080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:54:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:13.895+0000 7f3142433080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:54:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:54:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:54:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:18.464 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:54:18.984 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:54:18.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:54:18.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:18.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:18.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:18.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:19.007+0000 7fb86a685080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:19.027 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:19.026+0000 7fb86a685080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:19.029 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:19.028+0000 7fb86a685080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:19.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:54:19.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:19.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:54:19.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:19.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:19.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:19.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:19.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:19.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:54:19.384 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:19.602 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:19.544+0000 7fb86a685080 -1 Falling back to public interface 2021-01-15T17:54:19.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:20.345+0000 7fb86a685080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:54:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:20.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:54:20.774 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:21.182 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/607616,v1:127.0.0.1:6819/607616] [v2:127.0.0.1:6820/607616,v1:127.0.0.1:6821/607616] exists,up adcf2a91-009f-4d41-ae80-ccbaee4e2f70 2021-01-15T17:54:21.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:21.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:336: TEST_backfill_out2: run_osd td/osd-backfill-stats 3 2021-01-15T17:54:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:54:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:54:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:54:21.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:54:21.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:54:21.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:54:21.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:54:21.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:54:21.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:54:21.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:54:21.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:54:21.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:54:21.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:54:21.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:54:21.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:54:21.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:54:21.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:54:21.187 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:54:21.187 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:54:21.187 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:54:21.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:21.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:21.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:54:21.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:54:21.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:54:21.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:54:21.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:54:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:54:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:54:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:54:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:54:21.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:54:21.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:54:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f5158a05-6d6c-4dbc-82d5-c4af8352ac3a 2021-01-15T17:54:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 f5158a05-6d6c-4dbc-82d5-c4af8352ac3a' 2021-01-15T17:54:21.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:54:21.192 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 f5158a05-6d6c-4dbc-82d5-c4af8352ac3a 2021-01-15T17:54:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDN1gFgL47KCxAA6VCI3Yn7de0OQw0dZgE/0A== 2021-01-15T17:54:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDN1gFgL47KCxAA6VCI3Yn7de0OQw0dZgE/0A=="}' 2021-01-15T17:54:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f5158a05-6d6c-4dbc-82d5-c4af8352ac3a -i td/osd-backfill-stats/3/new.json 2021-01-15T17:54:21.736 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:54:21.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:54:21.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDN1gFgL47KCxAA6VCI3Yn7de0OQw0dZgE/0A== --osd-uuid f5158a05-6d6c-4dbc-82d5-c4af8352ac3a 2021-01-15T17:54:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.773+0000 7f9cbd6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.775+0000 7f9cbd6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.777+0000 7f9cbd6de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.777+0000 7f9cbd6de080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:54:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.777+0000 7f9cbd6de080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:54:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.777+0000 7f9cbd6de080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:54:21.884 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:21.885+0000 7f9cbd6de080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:54:26.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:54:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:26.395 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:54:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:54:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:26.930 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:54:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:54:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:26.953 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:26.952+0000 7f1a28b55080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:26.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:26.954+0000 7f1a28b55080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:26.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:26.956+0000 7f1a28b55080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:27.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:54:27.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:27.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:54:27.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:27.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:27.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:27.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:27.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:27.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:54:27.359 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:27.476 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:27.476+0000 7f1a28b55080 -1 Falling back to public interface 2021-01-15T17:54:27.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:28.286 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:28.285+0000 7f1a28b55080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:54:28.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:28.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:54:28.751 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:29.183 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/608880,v1:127.0.0.1:6827/608880] [v2:127.0.0.1:6828/608880,v1:127.0.0.1:6829/608880] exists,up f5158a05-6d6c-4dbc-82d5-c4af8352ac3a 2021-01-15T17:54:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:29.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:337: TEST_backfill_out2: run_osd td/osd-backfill-stats 4 2021-01-15T17:54:29.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:54:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:54:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:54:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:54:29.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:54:29.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:54:29.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:54:29.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:54:29.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:54:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:54:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:54:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:54:29.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:54:29.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:54:29.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:54:29.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:54:29.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:54:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:54:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:54:29.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:54:29.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:54:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:54:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:54:29.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:54:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:54:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:54:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:54:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:29.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:54:29.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:54:29.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:54:29.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:54:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e720263e-12ac-4e12-84e8-246aa7d5d7cb 2021-01-15T17:54:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 e720263e-12ac-4e12-84e8-246aa7d5d7cb' 2021-01-15T17:54:29.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:54:29.195 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 e720263e-12ac-4e12-84e8-246aa7d5d7cb 2021-01-15T17:54:29.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDV1gFg+LjgCxAA+mqMtrdYdYyEzj6ZYUbXrw== 2021-01-15T17:54:29.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDV1gFg+LjgCxAA+mqMtrdYdYyEzj6ZYUbXrw=="}' 2021-01-15T17:54:29.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e720263e-12ac-4e12-84e8-246aa7d5d7cb -i td/osd-backfill-stats/4/new.json 2021-01-15T17:54:29.769 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:54:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:54:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDV1gFg+LjgCxAA+mqMtrdYdYyEzj6ZYUbXrw== --osd-uuid e720263e-12ac-4e12-84e8-246aa7d5d7cb 2021-01-15T17:54:29.806 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.805+0000 7f9218cbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:29.807 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.807+0000 7f9218cbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:29.809 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.808+0000 7f9218cbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:29.809 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.809+0000 7f9218cbe080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:54:29.810 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.809+0000 7f9218cbe080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:54:29.810 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.809+0000 7f9218cbe080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:54:29.869 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:29.869+0000 7f9218cbe080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:54:34.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:54:34.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:34.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:54:34.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:34.447 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:54:34.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:54:34.976 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:54:34.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:34.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:34.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:34.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:34.999+0000 7effe7ea2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:35.036 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:35.036+0000 7effe7ea2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:35.038 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:35.038+0000 7effe7ea2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:35.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:54:35.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:35.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:54:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:35.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:54:35.384 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:35.553 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:35.552+0000 7effe7ea2080 -1 Falling back to public interface 2021-01-15T17:54:35.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:36.353 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:36.353+0000 7effe7ea2080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:54:36.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:36.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:36.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:36.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:36.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:54:36.795 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:37.219 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/610152,v1:127.0.0.1:6835/610152] [v2:127.0.0.1:6836/610152,v1:127.0.0.1:6837/610152] exists,up e720263e-12ac-4e12-84e8-246aa7d5d7cb 2021-01-15T17:54:37.219 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 d4c8c366-79b1-49fd-9378-d05004b02db5 2021-01-15T17:54:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:338: TEST_backfill_out2: run_osd td/osd-backfill-stats 5 2021-01-15T17:54:37.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:54:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:54:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:54:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:54:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:54:37.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:54:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:54:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:54:37.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:54:37.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:54:37.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:54:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:54:37.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:54:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:54:37.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:54:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:54:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:54:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:54:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:54:37.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:54:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:54:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:54:37.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:54:37.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:54:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d4c8c366-79b1-49fd-9378-d05004b02db5 2021-01-15T17:54:37.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 d4c8c366-79b1-49fd-9378-d05004b02db5' 2021-01-15T17:54:37.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:54:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDd1gFgbSPxDRAAnIXWeq+GJxdmARbTUR+nkw== 2021-01-15T17:54:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDd1gFgbSPxDRAAnIXWeq+GJxdmARbTUR+nkw=="}' 2021-01-15T17:54:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d4c8c366-79b1-49fd-9378-d05004b02db5 -i td/osd-backfill-stats/5/new.json 2021-01-15T17:54:37.806 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:54:37.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:54:37.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDd1gFgbSPxDRAAnIXWeq+GJxdmARbTUR+nkw== --osd-uuid d4c8c366-79b1-49fd-9378-d05004b02db5 2021-01-15T17:54:37.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.842+0000 7f1dbf6fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:37.844 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.844+0000 7f1dbf6fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:37.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.845+0000 7f1dbf6fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:37.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.846+0000 7f1dbf6fb080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:54:37.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.846+0000 7f1dbf6fb080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:54:37.847 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.846+0000 7f1dbf6fb080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:54:37.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:37.937+0000 7f1dbf6fb080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:54:42.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:54:42.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:54:42.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:54:42.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:54:42.711 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:54:43.236 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:54:43.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:54:43.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:54:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:54:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:54:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:54:43.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:43.259+0000 7f34234b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:43.288 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:43.287+0000 7f34234b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:43.289 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:43.288+0000 7f34234b8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:54:43.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:54:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:54:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:54:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:54:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:54:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:54:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:54:43.638 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:54:43.804 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:43.804+0000 7f34234b8080 -1 Falling back to public interface 2021-01-15T17:54:44.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:54:44.598 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:54:44.598+0000 7f34234b8080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:54:45.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:54:45.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:54:45.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:54:45.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:54:45.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:54:45.049 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:54:45.468 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/611416,v1:127.0.0.1:6843/611416] [v2:127.0.0.1:6844/611416,v1:127.0.0.1:6845/611416] exists,up d4c8c366-79b1-49fd-9378-d05004b02db5 2021-01-15T17:54:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:54:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:54:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:54:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:340: TEST_backfill_out2: create_pool test 1 1 2021-01-15T17:54:45.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T17:54:46.069 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:54:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:54:47.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:341: TEST_backfill_out2: ceph osd pool set test size 2 2021-01-15T17:54:48.160 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:54:48.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:342: TEST_backfill_out2: sleep 5 2021-01-15T17:54:53.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:344: TEST_backfill_out2: wait_for_clean 2021-01-15T17:54:53.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:54:53.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:54:53.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:54:53.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:54:53.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:54:53.174 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:54:53.174 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:54:53.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:54:53.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:54:53.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:54:53.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:54:53.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:54:53.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:54:53.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:54:53.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:54:53.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:54:53.670 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:54:53.671 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:54:53.671 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:54:53.671 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:54:53.671 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:54:53.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:54:53.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:53.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:54:53.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836493 2021-01-15T17:54:53.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836493 2021-01-15T17:54:53.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493' 2021-01-15T17:54:53.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:53.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:54:53.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672972 2021-01-15T17:54:53.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672972 2021-01-15T17:54:53.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972' 2021-01-15T17:54:53.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:53.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:54:54.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509450 2021-01-15T17:54:54.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509450 2021-01-15T17:54:54.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450' 2021-01-15T17:54:54.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:54.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:54:54.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T17:54:54.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T17:54:54.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450 3-85899345927' 2021-01-15T17:54:54.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:54.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:54:54.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T17:54:54.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T17:54:54.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450 3-85899345927 4-107374182406' 2021-01-15T17:54:54.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:54:54.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:54:54.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T17:54:54.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T17:54:54.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672972 2-64424509450 3-85899345927 4-107374182406 5-128849018884' 2021-01-15T17:54:54.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:54.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2021-01-15T17:54:54.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:54.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:54:54.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-01-15T17:54:54.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:54.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836493 2021-01-15T17:54:54.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-01-15T17:54:54.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:54:54.497 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2021-01-15T17:54:54.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836493 2021-01-15T17:54:54.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:54:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:54:55.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:54:56.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836493 2021-01-15T17:54:56.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:56.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672972 2021-01-15T17:54:56.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:54:56.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672972 2021-01-15T17:54:56.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672972 2021-01-15T17:54:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2021-01-15T17:54:56.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:54:56.287 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672972 2021-01-15T17:54:56.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672972 -lt 42949672972 2021-01-15T17:54:56.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:56.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509450 2021-01-15T17:54:56.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:56.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:54:56.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509450 2021-01-15T17:54:56.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:56.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509450 2021-01-15T17:54:56.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2021-01-15T17:54:56.686 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509450 2021-01-15T17:54:56.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:54:57.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509450 -lt 64424509450 2021-01-15T17:54:57.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:57.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T17:54:57.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:57.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:54:57.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T17:54:57.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:57.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T17:54:57.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T17:54:57.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:54:57.083 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T17:54:57.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345927 2021-01-15T17:54:57.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:57.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182406 2021-01-15T17:54:57.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:57.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:54:57.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182406 2021-01-15T17:54:57.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:57.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T17:54:57.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2021-01-15T17:54:57.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:54:57.488 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182406 2021-01-15T17:54:57.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2021-01-15T17:54:57.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:54:57.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T17:54:57.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:54:57.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:54:57.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T17:54:57.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:54:57.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T17:54:57.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T17:54:57.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:54:57.890 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T17:54:58.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2021-01-15T17:54:58.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:54:58.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:54:58.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:54:58.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:54:58.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:54:58.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:54:58.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:54:58.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:54:58.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:54:58.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:54:58.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:54:59.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:54:59.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:54:59.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:54:59.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:54:59.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:54:59.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:54:59.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:54:59.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: seq 1 200 2021-01-15T17:54:59.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:54:59.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj1 /dev/null 2021-01-15T17:54:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:54:59.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj2 /dev/null 2021-01-15T17:54:59.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:54:59.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj3 /dev/null 2021-01-15T17:55:00.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj4 /dev/null 2021-01-15T17:55:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj5 /dev/null 2021-01-15T17:55:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj6 /dev/null 2021-01-15T17:55:00.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj7 /dev/null 2021-01-15T17:55:00.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj8 /dev/null 2021-01-15T17:55:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj9 /dev/null 2021-01-15T17:55:00.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj10 /dev/null 2021-01-15T17:55:00.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj11 /dev/null 2021-01-15T17:55:00.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj12 /dev/null 2021-01-15T17:55:00.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj13 /dev/null 2021-01-15T17:55:00.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:00.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj14 /dev/null 2021-01-15T17:55:01.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj15 /dev/null 2021-01-15T17:55:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj16 /dev/null 2021-01-15T17:55:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj17 /dev/null 2021-01-15T17:55:01.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj18 /dev/null 2021-01-15T17:55:01.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj19 /dev/null 2021-01-15T17:55:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj20 /dev/null 2021-01-15T17:55:01.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj21 /dev/null 2021-01-15T17:55:01.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj22 /dev/null 2021-01-15T17:55:01.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj23 /dev/null 2021-01-15T17:55:01.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:01.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj24 /dev/null 2021-01-15T17:55:02.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj25 /dev/null 2021-01-15T17:55:02.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj26 /dev/null 2021-01-15T17:55:02.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj27 /dev/null 2021-01-15T17:55:02.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj28 /dev/null 2021-01-15T17:55:02.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj29 /dev/null 2021-01-15T17:55:02.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj30 /dev/null 2021-01-15T17:55:02.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj31 /dev/null 2021-01-15T17:55:02.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj32 /dev/null 2021-01-15T17:55:02.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj33 /dev/null 2021-01-15T17:55:02.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:02.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj34 /dev/null 2021-01-15T17:55:03.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj35 /dev/null 2021-01-15T17:55:03.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj36 /dev/null 2021-01-15T17:55:03.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj37 /dev/null 2021-01-15T17:55:03.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj38 /dev/null 2021-01-15T17:55:03.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj39 /dev/null 2021-01-15T17:55:03.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj40 /dev/null 2021-01-15T17:55:03.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj41 /dev/null 2021-01-15T17:55:03.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj42 /dev/null 2021-01-15T17:55:03.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:03.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj43 /dev/null 2021-01-15T17:55:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj44 /dev/null 2021-01-15T17:55:04.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj45 /dev/null 2021-01-15T17:55:04.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj46 /dev/null 2021-01-15T17:55:04.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj47 /dev/null 2021-01-15T17:55:04.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj48 /dev/null 2021-01-15T17:55:04.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj49 /dev/null 2021-01-15T17:55:04.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj50 /dev/null 2021-01-15T17:55:04.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj51 /dev/null 2021-01-15T17:55:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj52 /dev/null 2021-01-15T17:55:04.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:04.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj53 /dev/null 2021-01-15T17:55:05.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj54 /dev/null 2021-01-15T17:55:05.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj55 /dev/null 2021-01-15T17:55:05.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj56 /dev/null 2021-01-15T17:55:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj57 /dev/null 2021-01-15T17:55:05.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj58 /dev/null 2021-01-15T17:55:05.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj59 /dev/null 2021-01-15T17:55:05.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj60 /dev/null 2021-01-15T17:55:05.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj61 /dev/null 2021-01-15T17:55:05.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj62 /dev/null 2021-01-15T17:55:05.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:05.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj63 /dev/null 2021-01-15T17:55:06.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj64 /dev/null 2021-01-15T17:55:06.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj65 /dev/null 2021-01-15T17:55:06.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj66 /dev/null 2021-01-15T17:55:06.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj67 /dev/null 2021-01-15T17:55:06.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj68 /dev/null 2021-01-15T17:55:06.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj69 /dev/null 2021-01-15T17:55:06.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj70 /dev/null 2021-01-15T17:55:06.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj71 /dev/null 2021-01-15T17:55:06.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj72 /dev/null 2021-01-15T17:55:06.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:06.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj73 /dev/null 2021-01-15T17:55:07.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj74 /dev/null 2021-01-15T17:55:07.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj75 /dev/null 2021-01-15T17:55:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj76 /dev/null 2021-01-15T17:55:07.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj77 /dev/null 2021-01-15T17:55:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj78 /dev/null 2021-01-15T17:55:07.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj79 /dev/null 2021-01-15T17:55:07.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj80 /dev/null 2021-01-15T17:55:07.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj81 /dev/null 2021-01-15T17:55:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj82 /dev/null 2021-01-15T17:55:07.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:07.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj83 /dev/null 2021-01-15T17:55:08.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj84 /dev/null 2021-01-15T17:55:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj85 /dev/null 2021-01-15T17:55:08.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj86 /dev/null 2021-01-15T17:55:08.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj87 /dev/null 2021-01-15T17:55:08.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj88 /dev/null 2021-01-15T17:55:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj89 /dev/null 2021-01-15T17:55:08.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj90 /dev/null 2021-01-15T17:55:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj91 /dev/null 2021-01-15T17:55:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj92 /dev/null 2021-01-15T17:55:08.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj93 /dev/null 2021-01-15T17:55:08.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:08.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj94 /dev/null 2021-01-15T17:55:09.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj95 /dev/null 2021-01-15T17:55:09.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj96 /dev/null 2021-01-15T17:55:09.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj97 /dev/null 2021-01-15T17:55:09.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj98 /dev/null 2021-01-15T17:55:09.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj99 /dev/null 2021-01-15T17:55:09.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj100 /dev/null 2021-01-15T17:55:09.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj101 /dev/null 2021-01-15T17:55:09.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj102 /dev/null 2021-01-15T17:55:09.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:09.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj103 /dev/null 2021-01-15T17:55:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj104 /dev/null 2021-01-15T17:55:10.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj105 /dev/null 2021-01-15T17:55:10.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj106 /dev/null 2021-01-15T17:55:10.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj107 /dev/null 2021-01-15T17:55:10.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj108 /dev/null 2021-01-15T17:55:10.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj109 /dev/null 2021-01-15T17:55:10.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj110 /dev/null 2021-01-15T17:55:10.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj111 /dev/null 2021-01-15T17:55:10.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj112 /dev/null 2021-01-15T17:55:10.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:10.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj113 /dev/null 2021-01-15T17:55:11.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj114 /dev/null 2021-01-15T17:55:11.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj115 /dev/null 2021-01-15T17:55:11.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj116 /dev/null 2021-01-15T17:55:11.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj117 /dev/null 2021-01-15T17:55:11.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj118 /dev/null 2021-01-15T17:55:11.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj119 /dev/null 2021-01-15T17:55:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj120 /dev/null 2021-01-15T17:55:11.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj121 /dev/null 2021-01-15T17:55:11.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj122 /dev/null 2021-01-15T17:55:11.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:11.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj123 /dev/null 2021-01-15T17:55:12.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj124 /dev/null 2021-01-15T17:55:12.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj125 /dev/null 2021-01-15T17:55:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj126 /dev/null 2021-01-15T17:55:12.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj127 /dev/null 2021-01-15T17:55:12.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj128 /dev/null 2021-01-15T17:55:12.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj129 /dev/null 2021-01-15T17:55:12.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj130 /dev/null 2021-01-15T17:55:12.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj131 /dev/null 2021-01-15T17:55:12.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj132 /dev/null 2021-01-15T17:55:12.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:12.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj133 /dev/null 2021-01-15T17:55:13.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj134 /dev/null 2021-01-15T17:55:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj135 /dev/null 2021-01-15T17:55:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj136 /dev/null 2021-01-15T17:55:13.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj137 /dev/null 2021-01-15T17:55:13.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj138 /dev/null 2021-01-15T17:55:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj139 /dev/null 2021-01-15T17:55:13.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj140 /dev/null 2021-01-15T17:55:13.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj141 /dev/null 2021-01-15T17:55:13.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj142 /dev/null 2021-01-15T17:55:13.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:13.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj143 /dev/null 2021-01-15T17:55:14.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj144 /dev/null 2021-01-15T17:55:14.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj145 /dev/null 2021-01-15T17:55:14.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj146 /dev/null 2021-01-15T17:55:14.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj147 /dev/null 2021-01-15T17:55:14.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj148 /dev/null 2021-01-15T17:55:14.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj149 /dev/null 2021-01-15T17:55:14.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj150 /dev/null 2021-01-15T17:55:14.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj151 /dev/null 2021-01-15T17:55:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj152 /dev/null 2021-01-15T17:55:14.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj153 /dev/null 2021-01-15T17:55:14.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:14.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj154 /dev/null 2021-01-15T17:55:15.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj155 /dev/null 2021-01-15T17:55:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj156 /dev/null 2021-01-15T17:55:15.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj157 /dev/null 2021-01-15T17:55:15.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj158 /dev/null 2021-01-15T17:55:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj159 /dev/null 2021-01-15T17:55:15.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj160 /dev/null 2021-01-15T17:55:15.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj161 /dev/null 2021-01-15T17:55:15.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj162 /dev/null 2021-01-15T17:55:15.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:15.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj163 /dev/null 2021-01-15T17:55:16.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj164 /dev/null 2021-01-15T17:55:16.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj165 /dev/null 2021-01-15T17:55:16.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj166 /dev/null 2021-01-15T17:55:16.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj167 /dev/null 2021-01-15T17:55:16.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj168 /dev/null 2021-01-15T17:55:16.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj169 /dev/null 2021-01-15T17:55:16.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj170 /dev/null 2021-01-15T17:55:16.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj171 /dev/null 2021-01-15T17:55:16.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj172 /dev/null 2021-01-15T17:55:16.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:16.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj173 /dev/null 2021-01-15T17:55:17.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj174 /dev/null 2021-01-15T17:55:17.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj175 /dev/null 2021-01-15T17:55:17.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj176 /dev/null 2021-01-15T17:55:17.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj177 /dev/null 2021-01-15T17:55:17.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj178 /dev/null 2021-01-15T17:55:17.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj179 /dev/null 2021-01-15T17:55:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj180 /dev/null 2021-01-15T17:55:17.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj181 /dev/null 2021-01-15T17:55:17.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj182 /dev/null 2021-01-15T17:55:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj183 /dev/null 2021-01-15T17:55:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj184 /dev/null 2021-01-15T17:55:18.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj185 /dev/null 2021-01-15T17:55:18.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj186 /dev/null 2021-01-15T17:55:18.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj187 /dev/null 2021-01-15T17:55:18.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj188 /dev/null 2021-01-15T17:55:18.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj189 /dev/null 2021-01-15T17:55:18.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj190 /dev/null 2021-01-15T17:55:18.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj191 /dev/null 2021-01-15T17:55:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj192 /dev/null 2021-01-15T17:55:18.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:18.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj193 /dev/null 2021-01-15T17:55:19.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj194 /dev/null 2021-01-15T17:55:19.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj195 /dev/null 2021-01-15T17:55:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj196 /dev/null 2021-01-15T17:55:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj197 /dev/null 2021-01-15T17:55:19.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj198 /dev/null 2021-01-15T17:55:19.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj199 /dev/null 2021-01-15T17:55:19.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:346: TEST_backfill_out2: for i in $(seq 1 $objects) 2021-01-15T17:55:19.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:348: TEST_backfill_out2: rados -p test put obj200 /dev/null 2021-01-15T17:55:19.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:351: TEST_backfill_out2: get_pg test obj1 2021-01-15T17:55:19.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:55:19.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:55:19.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:55:19.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:55:20.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:351: TEST_backfill_out2: local PG=1.0 2021-01-15T17:55:20.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:353: TEST_backfill_out2: get_primary test obj1 2021-01-15T17:55:20.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:55:20.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:55:20.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:55:20.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:55:20.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:353: TEST_backfill_out2: local primary=1 2021-01-15T17:55:20.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:354: TEST_backfill_out2: get_not_primary test obj1 2021-01-15T17:55:20.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:55:20.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:55:20.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:55:20.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:55:20.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:55:20.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:55:20.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:55:20.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:55:20.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:55:20.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:55:21.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:354: TEST_backfill_out2: local otherosd=0 2021-01-15T17:55:21.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:356: TEST_backfill_out2: ceph osd set nobackfill 2021-01-15T17:55:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:55:22.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:357: TEST_backfill_out2: ceph osd pool set test size 3 2021-01-15T17:55:22.914 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T17:55:22.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:358: TEST_backfill_out2: ceph osd out osd.0 2021-01-15T17:55:23.790 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:55:23.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:359: TEST_backfill_out2: ceph osd out osd.1 2021-01-15T17:55:24.471 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:55:24.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:361: TEST_backfill_out2: sleep 2 2021-01-15T17:55:26.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:362: TEST_backfill_out2: get_primary test obj1 2021-01-15T17:55:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:55:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:55:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:55:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:55:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:362: TEST_backfill_out2: primary=1 2021-01-15T17:55:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:363: TEST_backfill_out2: ceph osd unset nobackfill 2021-01-15T17:55:27.630 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:55:27.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:364: TEST_backfill_out2: ceph tell osd.1 get_latest_osdmap 2021-01-15T17:55:27.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:365: TEST_backfill_out2: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T17:55:27.910 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:55:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:366: TEST_backfill_out2: sleep 2 2021-01-15T17:55:29.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:368: TEST_backfill_out2: wait_for_clean 2021-01-15T17:55:29.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:55:29.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:55:29.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:55:29.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:55:29.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:55:29.923 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:55:29.923 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:55:29.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:55:29.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:55:29.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:55:30.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:55:30.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:55:30.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:55:30.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:55:30.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:55:30.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:55:30.423 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:55:30.423 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:55:30.423 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:55:30.423 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:55:30.424 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:55:30.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:55:30.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:30.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:55:30.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2021-01-15T17:55:30.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2021-01-15T17:55:30.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2021-01-15T17:55:30.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:30.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:55:30.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672981 2021-01-15T17:55:30.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672981 2021-01-15T17:55:30.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981' 2021-01-15T17:55:30.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:30.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:55:30.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509459 2021-01-15T17:55:30.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509459 2021-01-15T17:55:30.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459' 2021-01-15T17:55:30.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:30.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:55:30.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345937 2021-01-15T17:55:30.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345937 2021-01-15T17:55:30.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459 3-85899345937' 2021-01-15T17:55:30.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:30.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:55:31.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182415 2021-01-15T17:55:31.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182415 2021-01-15T17:55:31.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459 3-85899345937 4-107374182415' 2021-01-15T17:55:31.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:55:31.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:55:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018894 2021-01-15T17:55:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018894 2021-01-15T17:55:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509459 3-85899345937 4-107374182415 5-128849018894' 2021-01-15T17:55:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:31.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2021-01-15T17:55:31.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:31.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:55:31.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2021-01-15T17:55:31.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:31.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2021-01-15T17:55:31.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-01-15T17:55:31.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:55:31.252 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836502 2021-01-15T17:55:31.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836502 2021-01-15T17:55:31.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:31.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672981 2021-01-15T17:55:31.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:31.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:55:31.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672981 2021-01-15T17:55:31.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:31.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672981 2021-01-15T17:55:31.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2021-01-15T17:55:31.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:55:31.657 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672981 2021-01-15T17:55:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672981 -lt 42949672981 2021-01-15T17:55:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:32.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509459 2021-01-15T17:55:32.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:32.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:55:32.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509459 2021-01-15T17:55:32.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509459 2021-01-15T17:55:32.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-01-15T17:55:32.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:55:32.051 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509459 2021-01-15T17:55:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509459 -lt 64424509459 2021-01-15T17:55:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:32.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345937 2021-01-15T17:55:32.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:32.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:55:32.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345937 2021-01-15T17:55:32.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:32.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345937 2021-01-15T17:55:32.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2021-01-15T17:55:32.451 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345937 2021-01-15T17:55:32.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:55:32.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345937 -lt 85899345937 2021-01-15T17:55:32.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:32.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182415 2021-01-15T17:55:32.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:32.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:55:32.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182415 2021-01-15T17:55:32.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:32.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182415 2021-01-15T17:55:32.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2021-01-15T17:55:32.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:55:32.888 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182415 2021-01-15T17:55:33.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182415 -lt 107374182415 2021-01-15T17:55:33.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:55:33.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018894 2021-01-15T17:55:33.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:55:33.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:55:33.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018894 2021-01-15T17:55:33.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:55:33.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018894 2021-01-15T17:55:33.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2021-01-15T17:55:33.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:55:33.285 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018894 2021-01-15T17:55:33.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018894 2021-01-15T17:55:33.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:55:33.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:33.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:34.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:55:34.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:34.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:34.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:34.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:34.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:34.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:34.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:34.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:34.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:34.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:35.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:35.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:55:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:55:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:55:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:35.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:35.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:35.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:35.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:35.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:35.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:35.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:35.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:35.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:35.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:35.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:36.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:36.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:36.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:36.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:36.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:36.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:36.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:36.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:36.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:36.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:36.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:36.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:36.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:36.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:36.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:37.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:37.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:37.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:37.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:37.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:37.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:37.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:37.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:37.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:37.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:37.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:37.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:38.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:55:38.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:55:38.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:38.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:38.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:38.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:38.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:38.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:38.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:38.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:38.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:38.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:38.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:38.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:38.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:38.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:38.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:39.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:39.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:39.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:39.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:55:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:55:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:39.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:39.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:39.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:39.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:39.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:39.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:39.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:39.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:40.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:40.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:40.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:40.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:40.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:40.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:40.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:40.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:40.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:40.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:40.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:40.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:41.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:55:41.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:55:41.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:41.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:41.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:41.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:41.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:41.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:41.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:41.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:41.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:41.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:41.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:41.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:41.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:41.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:41.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:42.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:42.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:42.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:42.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:42.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:42.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:42.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:42.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:42.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:42.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:42.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:42.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:42.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:42.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:42.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:42.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:42.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:43.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:43.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:43.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:43.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:43.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:43.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:43.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:44.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:55:44.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:55:44.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:44.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:44.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:44.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:44.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:44.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:44.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:44.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:44.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:44.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:44.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:44.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:44.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:44.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:44.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:45.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:45.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:45.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:45.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:45.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:45.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:45.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:45.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:55:45.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:55:45.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:45.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:45.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:45.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:45.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:45.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:45.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:45.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:45.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:45.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:45.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:46.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:46.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:46.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:46.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:46.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:46.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:46.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:46.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:46.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:46.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:46.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:47.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:55:47.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:55:47.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:47.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:47.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:47.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:47.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:47.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:47.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:47.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:47.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:47.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:47.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:47.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:47.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:47.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:48.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:48.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:48.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:48.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:48.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:49.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:49.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:49.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:49.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:49.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:49.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:49.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:49.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:50.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:55:50.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:55:50.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:50.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:50.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:50.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:50.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:50.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:50.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:50.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:50.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:50.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:50.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:50.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:50.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:51.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:51.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:51.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:51.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:51.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:51.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:51.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:51.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:51.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:51.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:52.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:52.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:52.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:52.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:52.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:52.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:52.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:52.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:53.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:53.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:53.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:53.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:53.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:53.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:53.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:53.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:53.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:53.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:53.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:53.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:53.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:53.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:53.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:54.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:54.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:54.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:54.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:54.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:54.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:54.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:54.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:55.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:55.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:55.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:55.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:55.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:55.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:55.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:55.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:55.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:55.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:56.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:56.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:56.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:56.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:55:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:55:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:56.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:56.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:56.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:56.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:56.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:56.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:56.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:56.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:56.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:56.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:57.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:57.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:57.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:57.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:57.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:57.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:57.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:57.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:57.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:57.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:57.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:57.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:57.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:58.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:55:58.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:55:58.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:58.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:58.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:58.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:58.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:58.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:58.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:58.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:58.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:58.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:58.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:55:58.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:55:58.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:55:58.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:55:58.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:55:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:55:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:55:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:55:59.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:55:59.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:55:59.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:55:59.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:55:59.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:55:59.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:55:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:55:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:55:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:55:59.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:55:59.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:55:59.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:55:59.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:55:59.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:55:59.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:55:59.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:55:59.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:55:59.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:55:59.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:00.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:00.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:00.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:00.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:00.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:00.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:00.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:00.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:00.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:01.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:56:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:56:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:01.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:01.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:01.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:01.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:01.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:01.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:01.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:01.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:01.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:01.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:01.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:01.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:02.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:02.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:02.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:02.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:02.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:02.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:02.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:02.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:02.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:02.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:02.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:02.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:02.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:02.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:02.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:02.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:02.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:02.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:02.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:03.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:03.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:03.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:03.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:03.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:03.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:03.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:03.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:03.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:03.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:03.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:03.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:03.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:04.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:04.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:04.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:04.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:04.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:04.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:04.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:04.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:04.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:04.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:04.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:04.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:04.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:04.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:04.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:04.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:04.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:05.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:05.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:05.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:05.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:05.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:56:05.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:56:05.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:05.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:05.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:06.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:06.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:06.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:06.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:06.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:06.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:06.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:06.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:06.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:06.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:06.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:06.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:06.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:06.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:06.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:06.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:06.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:06.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:06.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:06.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:07.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:07.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:07.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:07.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:07.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:07.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:07.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:07.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:07.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:07.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:07.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:07.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:07.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:07.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:07.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:08.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:08.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:08.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:08.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:08.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:08.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:08.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:08.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:08.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:08.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:08.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:08.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:09.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:09.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:09.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:09.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:09.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:09.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:09.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:09.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:09.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:09.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:09.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:09.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:09.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:09.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:09.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:09.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:10.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:56:10.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:56:10.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:10.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:10.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:10.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:10.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:10.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:10.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:10.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:10.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:10.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:11.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:11.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:11.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:11.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:11.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:11.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:11.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:11.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:11.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:11.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:12.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:12.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:12.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:12.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:12.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:12.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:12.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:12.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:12.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:12.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:12.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:13.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:13.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:13.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:13.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:13.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:13.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:13.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:13.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:13.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:13.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:13.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:13.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:13.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:13.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:13.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:13.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:13.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:13.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:14.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:14.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:14.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:14.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:14.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:14.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:14.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:14.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:14.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:14.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:14.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:14.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:14.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:15.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:56:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:56:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:15.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:15.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:15.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:15.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:15.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:15.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:15.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:15.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:15.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:15.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:15.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:15.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:16.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:16.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:16.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:16.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:16.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:16.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:56:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:56:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:56:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:56:16.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:16.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:16.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:16.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:16.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:16.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:16.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:16.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:17.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:56:17.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:17.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:17.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:17.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:56:17.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:56:17.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:56:17.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:56:17.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:56:17.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:56:17.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:56:17.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:56:17.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:56:18.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T17:56:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T17:56:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2021-01-15T17:56:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:56:18.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T17:56:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:56:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:56:18.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:56:18.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:56:18.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:56:18.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:56:18.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:56:18.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:56:18.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:56:18.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:56:18.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:56:18.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:56:19.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:56:19.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:56:19.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:56:19.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:370: TEST_backfill_out2: expr 200 '*' 2 2021-01-15T17:56:19.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:370: TEST_backfill_out2: local misplaced=400 2021-01-15T17:56:19.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:372: TEST_backfill_out2: check td/osd-backfill-stats 1.0 1 replicated 200 0 400 0 2021-01-15T17:56:19.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:56:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:56:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T17:56:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T17:56:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=200 2021-01-15T17:56:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:56:19.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2021-01-15T17:56:19.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T17:56:19.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:56:19.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:56:19.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T17:56:19.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T17:56:19.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T17:56:19.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T17:56:19.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T17:56:19.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:56:19.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T17:56:19.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:56:19.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:56:19.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:56:19.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[2,4,3]/[1,0]' 2021-01-15T17:56:19.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[2,4,3]/[1,0]' ']' 2021-01-15T17:56:19.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:56:19.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:56:19.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [2,4,3]/[1,0] ' 2021-01-15T17:56:19.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:56:19.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:56:19.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:56:19.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=200 2021-01-15T17:56:19.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 200 ']' 2021-01-15T17:56:19.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 200 200 2021-01-15T17:56:19.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T17:56:19.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:56:19.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T17:56:19.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:56:19.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:56:19.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:56:19.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:56:19.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=1 2021-01-15T17:56:19.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 1 ']' 2021-01-15T17:56:19.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 1 0 2021-01-15T17:56:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T17:56:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:56:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:56:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:56:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:56:19.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:56:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [2,4,3]/[1,0] ' 2021-01-15T17:56:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:56:19.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:56:19.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:56:19.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2021-01-15T17:56:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2021-01-15T17:56:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2021-01-15T17:56:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T17:56:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:56:19.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T17:56:19.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:56:19.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:56:19.368 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T17:56:19.368 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:56:19.369 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:56:19.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=2 2021-01-15T17:56:19.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 2 ']' 2021-01-15T17:56:19.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 2 0 2021-01-15T17:56:19.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2021-01-15T17:56:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:56:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:56:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:56:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:56:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:374: TEST_backfill_out2: delete_pool test 2021-01-15T17:56:19.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:56:19.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:56:19.985 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:56:19.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:375: TEST_backfill_out2: kill_daemons td/osd-backfill-stats 2021-01-15T17:56:19.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:56:19.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:56:19.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:56:19.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:56:19.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:56:20.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:56:20.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:56:20.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:56:20.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:56:20.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:56:20.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:56:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:56:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:56:20.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:56:20.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:56:20.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:56:20.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:56:20.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:56:20.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:56:20.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:56:20.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:56:20.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:56:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:56:20.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:56:20.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:56:20.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:56:20.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:56:20.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:56:20.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:56:20.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:56:20.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:56:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:56:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:56:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:56:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:56:20.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:56:20.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:56:20.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:56:20.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:56:20.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:56:20.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:56:20.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:56:20.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:56:20.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:56:20.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:56:20.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:56:20.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:56:20.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:56:20.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:56:20.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:56:20.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:56:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:56:20.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:56:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:56:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:56:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:56:20.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:56:20.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:56:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:56:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:56:20.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:56:20.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:56:20.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:56:20.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:56:20.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:56:20.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:56:20.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:56:20.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_remapped td/osd-backfill-stats 2021-01-15T17:56:20.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:442: TEST_backfill_remapped: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:444: TEST_backfill_remapped: run_mon td/osd-backfill-stats a 2021-01-15T17:56:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:56:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:56:20.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:56:20.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:56:20.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:56:20.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:56:20.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:20.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:20.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:20.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.433 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:20.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:56:20.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:56:20.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:56:20.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:56:20.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:56:20.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:56:20.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:56:20.609 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:56:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:56:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:56:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:56:20.610 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.611 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.611 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:56:20.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:56:20.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:56:20.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:56:20.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:56:20.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:56:20.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:56:20.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:56:20.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:56:20.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:56:20.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:56:20.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:56:20.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:20.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:20.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:56:20.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:56:20.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:56:20.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:445: TEST_backfill_remapped: run_mgr td/osd-backfill-stats x 2021-01-15T17:56:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:56:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:56:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:56:20.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:56:20.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:56:20.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:56:21.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:56:21.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:21.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:21.702 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:21.702 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:21.702 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:21.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:21.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:56:21.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:56:21.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:446: TEST_backfill_remapped: run_osd td/osd-backfill-stats 0 2021-01-15T17:56:21.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:56:21.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:56:21.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:56:21.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:56:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:56:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:56:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:56:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:56:21.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:56:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:56:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:56:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:56:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:56:21.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:56:21.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:56:21.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:21.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:21.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:21.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:21.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:56:21.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:56:21.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:56:21.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:56:21.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:56:21.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:56:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:56:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:56:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:56:21.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:56:21.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:56:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=798337a5-902e-4504-a686-086f3a2da1c8 2021-01-15T17:56:21.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 798337a5-902e-4504-a686-086f3a2da1c8' 2021-01-15T17:56:21.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:56:21.735 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 798337a5-902e-4504-a686-086f3a2da1c8 2021-01-15T17:56:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBF1wFgucpNLBAARssjts/y7cNRMyjoZULcDg== 2021-01-15T17:56:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBF1wFgucpNLBAARssjts/y7cNRMyjoZULcDg=="}' 2021-01-15T17:56:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 798337a5-902e-4504-a686-086f3a2da1c8 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:56:22.083 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:56:22.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:56:22.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBF1wFgucpNLBAARssjts/y7cNRMyjoZULcDg== --osd-uuid 798337a5-902e-4504-a686-086f3a2da1c8 2021-01-15T17:56:22.121 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.120+0000 7f24cdaa5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:22.123 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.123+0000 7f24cdaa5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:22.124 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.123+0000 7f24cdaa5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:22.124 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.124+0000 7f24cdaa5080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:56:22.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.124+0000 7f24cdaa5080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:56:22.125 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.124+0000 7f24cdaa5080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:56:22.165 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:22.164+0000 7f24cdaa5080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:56:26.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:56:26.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:56:26.634 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:56:26.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:56:26.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:56:27.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:56:27.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:27.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:56:27.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:56:27.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:56:27.160 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:56:27.181 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:27.181+0000 7f3fcf4a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:27.183 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:27.183+0000 7f3fcf4a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:27.184 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:27.184+0000 7f3fcf4a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:27.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:56:27.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:56:27.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:56:27.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:56:27.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:56:27.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:27.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:56:27.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:27.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:56:27.555 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:56:27.702 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:27.702+0000 7f3fcf4a2080 -1 Falling back to public interface 2021-01-15T17:56:27.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:56:28.506 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:28.505+0000 7f3fcf4a2080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:56:28.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:56:28.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:28.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:56:28.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:28.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:56:28.950 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:56:29.382 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/620768,v1:127.0.0.1:6803/620768] [v2:127.0.0.1:6804/620768,v1:127.0.0.1:6805/620768] exists,up 798337a5-902e-4504-a686-086f3a2da1c8 2021-01-15T17:56:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:56:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:56:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:56:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:447: TEST_backfill_remapped: run_osd td/osd-backfill-stats 1 2021-01-15T17:56:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:56:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:56:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:56:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:56:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:56:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:56:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:56:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:56:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:56:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:56:29.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:56:29.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:56:29.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:56:29.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:56:29.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:56:29.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:29.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:29.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:29.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:29.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:29.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:29.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:29.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:56:29.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:56:29.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:56:29.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:56:29.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:56:29.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:56:29.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:56:29.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:29.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:56:29.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:56:29.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:56:29.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:56:29.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1747aba7-fc88-4120-831c-af4b49a85e7d 2021-01-15T17:56:29.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 1747aba7-fc88-4120-831c-af4b49a85e7d' 2021-01-15T17:56:29.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:56:29.392 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 1747aba7-fc88-4120-831c-af4b49a85e7d 2021-01-15T17:56:29.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBN1wFgYxO/FxAAiuLmTgLXn+UNglJ/2tTb7Q== 2021-01-15T17:56:29.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBN1wFgYxO/FxAAiuLmTgLXn+UNglJ/2tTb7Q=="}' 2021-01-15T17:56:29.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1747aba7-fc88-4120-831c-af4b49a85e7d -i td/osd-backfill-stats/1/new.json 2021-01-15T17:56:29.914 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:56:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:56:29.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBN1wFgYxO/FxAAiuLmTgLXn+UNglJ/2tTb7Q== --osd-uuid 1747aba7-fc88-4120-831c-af4b49a85e7d 2021-01-15T17:56:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.951+0000 7f09a203e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:29.953 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.953+0000 7f09a203e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:29.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.954+0000 7f09a203e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:29.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.954+0000 7f09a203e080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:56:29.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.954+0000 7f09a203e080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:56:29.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:29.954+0000 7f09a203e080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:56:30.003 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:30.003+0000 7f09a203e080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:56:34.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:56:34.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:56:34.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:56:34.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:56:34.547 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:56:35.074 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:56:35.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:56:35.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:56:35.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:35.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:56:35.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:56:35.097 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:35.096+0000 7f17a04eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:35.109+0000 7f17a04eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:35.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:35.110+0000 7f17a04eb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:35.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:56:35.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:56:35.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:56:35.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:56:35.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:56:35.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:35.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:56:35.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:35.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:56:35.472 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:56:35.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:35.627+0000 7f17a04eb080 -1 Falling back to public interface 2021-01-15T17:56:35.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:56:36.428 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:36.428+0000 7f17a04eb080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:56:36.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:56:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:56:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:36.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:56:36.895 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:56:37.315 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/622040,v1:127.0.0.1:6811/622040] [v2:127.0.0.1:6812/622040,v1:127.0.0.1:6813/622040] exists,up 1747aba7-fc88-4120-831c-af4b49a85e7d 2021-01-15T17:56:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:56:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:56:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:56:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:448: TEST_backfill_remapped: run_osd td/osd-backfill-stats 2 2021-01-15T17:56:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:56:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:56:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:56:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:56:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:56:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:56:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:56:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:56:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:56:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:56:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:56:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:56:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:56:37.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:56:37.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:56:37.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:37.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:37.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:56:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:56:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:56:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:56:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:56:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:56:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:56:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:56:37.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:56:37.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:56:37.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:56:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fce295e3-ac8a-4361-a218-081fc5954209 2021-01-15T17:56:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 fce295e3-ac8a-4361-a218-081fc5954209' 2021-01-15T17:56:37.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:56:37.324 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 fce295e3-ac8a-4361-a218-081fc5954209 2021-01-15T17:56:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBV1wFgPKi4ExAA7yf+YKJ/ZnTgmCqdPOmW5g== 2021-01-15T17:56:37.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBV1wFgPKi4ExAA7yf+YKJ/ZnTgmCqdPOmW5g=="}' 2021-01-15T17:56:37.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fce295e3-ac8a-4361-a218-081fc5954209 -i td/osd-backfill-stats/2/new.json 2021-01-15T17:56:37.886 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:56:37.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:56:37.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBV1wFgPKi4ExAA7yf+YKJ/ZnTgmCqdPOmW5g== --osd-uuid fce295e3-ac8a-4361-a218-081fc5954209 2021-01-15T17:56:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.922+0000 7f41aabbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.924+0000 7f41aabbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:37.926 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.926+0000 7f41aabbe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.926+0000 7f41aabbe080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:56:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.926+0000 7f41aabbe080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:56:37.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.926+0000 7f41aabbe080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:56:37.967 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:37.967+0000 7f41aabbe080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:56:42.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:56:42.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:56:42.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:56:42.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:56:42.714 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:56:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:56:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:56:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:56:43.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:56:43.238 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:56:43.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:43.259+0000 7fc1cbd3b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:43.275 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:43.274+0000 7fc1cbd3b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:43.276 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:43.276+0000 7fc1cbd3b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:56:43.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:56:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:56:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:56:43.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:56:43.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:43.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:56:43.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:43.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:56:43.639 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:56:43.799 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:43.799+0000 7fc1cbd3b080 -1 Falling back to public interface 2021-01-15T17:56:44.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:56:44.568 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:44.567+0000 7fc1cbd3b080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:56:45.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:56:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:56:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:45.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:56:45.032 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:56:45.453 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/623304,v1:127.0.0.1:6819/623304] [v2:127.0.0.1:6820/623304,v1:127.0.0.1:6821/623304] exists,up fce295e3-ac8a-4361-a218-081fc5954209 2021-01-15T17:56:45.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:56:45.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:56:45.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:56:45.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:449: TEST_backfill_remapped: run_osd td/osd-backfill-stats 3 2021-01-15T17:56:45.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:56:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:56:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:56:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:56:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:56:45.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:56:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:56:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:56:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:56:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:56:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:56:45.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:56:45.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:56:45.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:56:45.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:56:45.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:56:45.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:56:45.458 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:56:45.458 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:56:45.458 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:56:45.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:56:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:56:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:56:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:56:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:56:45.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:56:45.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:56:45.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:56:45.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:56:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:56:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:56:45.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:56:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1fd1af17-d141-4c6a-9c59-6cb787287a94 2021-01-15T17:56:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 1fd1af17-d141-4c6a-9c59-6cb787287a94' 2021-01-15T17:56:45.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:56:45.463 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 1fd1af17-d141-4c6a-9c59-6cb787287a94 2021-01-15T17:56:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBd1wFgNxv1GxAAEtLwu3eApA1BZTkkX7Daog== 2021-01-15T17:56:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBd1wFgNxv1GxAAEtLwu3eApA1BZTkkX7Daog=="}' 2021-01-15T17:56:45.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1fd1af17-d141-4c6a-9c59-6cb787287a94 -i td/osd-backfill-stats/3/new.json 2021-01-15T17:56:46.079 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:56:46.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:56:46.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBd1wFgNxv1GxAAEtLwu3eApA1BZTkkX7Daog== --osd-uuid 1fd1af17-d141-4c6a-9c59-6cb787287a94 2021-01-15T17:56:46.118 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.117+0000 7fe5b59f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:46.119 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.118+0000 7fe5b59f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:46.120 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.119+0000 7fe5b59f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:46.121 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.119+0000 7fe5b59f0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:56:46.121 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.119+0000 7fe5b59f0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:56:46.121 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.119+0000 7fe5b59f0080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:56:46.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:46.156+0000 7fe5b59f0080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:56:50.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:56:50.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:56:50.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:56:50.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:56:50.685 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:56:51.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:56:51.211 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:56:51.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:56:51.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:56:51.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:56:51.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:56:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:51.233+0000 7ff1c4869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:51.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:51.257+0000 7ff1c4869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:51.259 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:51.259+0000 7ff1c4869080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:56:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:56:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:56:51.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:56:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:56:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:56:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:56:51.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:51.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:56:51.623 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:56:51.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:51.775+0000 7ff1c4869080 -1 Falling back to public interface 2021-01-15T17:56:52.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:56:52.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:56:52.551+0000 7ff1c4869080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:56:53.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:56:53.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:56:53.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:56:53.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:56:53.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:56:53.023 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:56:53.428 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/624568,v1:127.0.0.1:6827/624568] [v2:127.0.0.1:6828/624568,v1:127.0.0.1:6829/624568] exists,up 1fd1af17-d141-4c6a-9c59-6cb787287a94 2021-01-15T17:56:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:56:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:56:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:56:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:451: TEST_backfill_remapped: create_pool test 1 1 2021-01-15T17:56:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T17:56:54.030 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:56:54.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:56:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:452: TEST_backfill_remapped: ceph osd pool set test size 3 2021-01-15T17:56:55.710 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T17:56:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:453: TEST_backfill_remapped: sleep 5 2021-01-15T17:57:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:455: TEST_backfill_remapped: wait_for_clean 2021-01-15T17:57:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:57:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:57:00.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:57:00.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:57:00.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:57:00.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:57:00.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:57:00.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:57:00.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:57:00.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:57:01.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:57:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:57:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:57:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T17:57:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:57:01.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:01.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:57:01.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2021-01-15T17:57:01.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2021-01-15T17:57:01.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2021-01-15T17:57:01.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:01.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:57:01.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672967 2021-01-15T17:57:01.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672967 2021-01-15T17:57:01.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967' 2021-01-15T17:57:01.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:01.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:57:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T17:57:01.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T17:57:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446' 2021-01-15T17:57:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:01.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:57:01.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345924 2021-01-15T17:57:01.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345924 2021-01-15T17:57:01.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672967 2-64424509446 3-85899345924' 2021-01-15T17:57:01.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:01.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2021-01-15T17:57:01.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:01.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:57:01.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2021-01-15T17:57:01.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:01.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2021-01-15T17:57:01.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-01-15T17:57:01.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:57:01.775 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2021-01-15T17:57:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836489 2021-01-15T17:57:02.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:02.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672967 2021-01-15T17:57:02.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:02.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:57:02.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672967 2021-01-15T17:57:02.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:02.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672967 2021-01-15T17:57:02.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-01-15T17:57:02.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:57:02.167 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672967 2021-01-15T17:57:02.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672967 -lt 42949672967 2021-01-15T17:57:02.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:02.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T17:57:02.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:57:02.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T17:57:02.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:02.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T17:57:02.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T17:57:02.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:57:02.568 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T17:57:02.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509446 2021-01-15T17:57:02.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:57:03.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:57:03.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:57:04.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509446 2021-01-15T17:57:04.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:04.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345924 2021-01-15T17:57:04.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:57:04.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345924 2021-01-15T17:57:04.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345924 2021-01-15T17:57:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345924' 2021-01-15T17:57:04.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:57:04.359 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345924 2021-01-15T17:57:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345924 -lt 85899345924 2021-01-15T17:57:04.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:57:04.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:04.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:05.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:57:05.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:05.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:05.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:05.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:05.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:05.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:57:05.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:05.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:05.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:06.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:57:06.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:57:06.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:57:06.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: seq 1 200 2021-01-15T17:57:06.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj1 /dev/null 2021-01-15T17:57:06.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj2 /dev/null 2021-01-15T17:57:06.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj3 /dev/null 2021-01-15T17:57:06.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj4 /dev/null 2021-01-15T17:57:06.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj5 /dev/null 2021-01-15T17:57:06.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj6 /dev/null 2021-01-15T17:57:06.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj7 /dev/null 2021-01-15T17:57:06.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:06.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj8 /dev/null 2021-01-15T17:57:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj9 /dev/null 2021-01-15T17:57:07.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj10 /dev/null 2021-01-15T17:57:07.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj11 /dev/null 2021-01-15T17:57:07.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj12 /dev/null 2021-01-15T17:57:07.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj13 /dev/null 2021-01-15T17:57:07.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj14 /dev/null 2021-01-15T17:57:07.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj15 /dev/null 2021-01-15T17:57:07.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:07.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj16 /dev/null 2021-01-15T17:57:08.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj17 /dev/null 2021-01-15T17:57:08.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj18 /dev/null 2021-01-15T17:57:08.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj19 /dev/null 2021-01-15T17:57:08.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj20 /dev/null 2021-01-15T17:57:08.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj21 /dev/null 2021-01-15T17:57:08.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj22 /dev/null 2021-01-15T17:57:08.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj23 /dev/null 2021-01-15T17:57:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj24 /dev/null 2021-01-15T17:57:08.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj25 /dev/null 2021-01-15T17:57:08.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:08.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj26 /dev/null 2021-01-15T17:57:09.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj27 /dev/null 2021-01-15T17:57:09.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj28 /dev/null 2021-01-15T17:57:09.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj29 /dev/null 2021-01-15T17:57:09.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj30 /dev/null 2021-01-15T17:57:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj31 /dev/null 2021-01-15T17:57:09.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj32 /dev/null 2021-01-15T17:57:09.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj33 /dev/null 2021-01-15T17:57:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj34 /dev/null 2021-01-15T17:57:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:09.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj35 /dev/null 2021-01-15T17:57:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj36 /dev/null 2021-01-15T17:57:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj37 /dev/null 2021-01-15T17:57:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj38 /dev/null 2021-01-15T17:57:10.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj39 /dev/null 2021-01-15T17:57:10.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj40 /dev/null 2021-01-15T17:57:10.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj41 /dev/null 2021-01-15T17:57:10.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj42 /dev/null 2021-01-15T17:57:10.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj43 /dev/null 2021-01-15T17:57:10.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:10.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj44 /dev/null 2021-01-15T17:57:11.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj45 /dev/null 2021-01-15T17:57:11.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj46 /dev/null 2021-01-15T17:57:11.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj47 /dev/null 2021-01-15T17:57:11.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj48 /dev/null 2021-01-15T17:57:11.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj49 /dev/null 2021-01-15T17:57:11.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj50 /dev/null 2021-01-15T17:57:11.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj51 /dev/null 2021-01-15T17:57:11.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj52 /dev/null 2021-01-15T17:57:11.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:11.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj53 /dev/null 2021-01-15T17:57:12.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj54 /dev/null 2021-01-15T17:57:12.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj55 /dev/null 2021-01-15T17:57:12.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj56 /dev/null 2021-01-15T17:57:12.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj57 /dev/null 2021-01-15T17:57:12.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj58 /dev/null 2021-01-15T17:57:12.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj59 /dev/null 2021-01-15T17:57:12.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj60 /dev/null 2021-01-15T17:57:12.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj61 /dev/null 2021-01-15T17:57:12.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj62 /dev/null 2021-01-15T17:57:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:12.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj63 /dev/null 2021-01-15T17:57:13.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj64 /dev/null 2021-01-15T17:57:13.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj65 /dev/null 2021-01-15T17:57:13.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj66 /dev/null 2021-01-15T17:57:13.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj67 /dev/null 2021-01-15T17:57:13.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj68 /dev/null 2021-01-15T17:57:13.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj69 /dev/null 2021-01-15T17:57:13.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj70 /dev/null 2021-01-15T17:57:13.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj71 /dev/null 2021-01-15T17:57:13.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:13.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj72 /dev/null 2021-01-15T17:57:14.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj73 /dev/null 2021-01-15T17:57:14.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj74 /dev/null 2021-01-15T17:57:14.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj75 /dev/null 2021-01-15T17:57:14.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj76 /dev/null 2021-01-15T17:57:14.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj77 /dev/null 2021-01-15T17:57:14.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj78 /dev/null 2021-01-15T17:57:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:14.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj79 /dev/null 2021-01-15T17:57:15.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj80 /dev/null 2021-01-15T17:57:15.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj81 /dev/null 2021-01-15T17:57:15.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj82 /dev/null 2021-01-15T17:57:15.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj83 /dev/null 2021-01-15T17:57:15.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj84 /dev/null 2021-01-15T17:57:15.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj85 /dev/null 2021-01-15T17:57:15.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj86 /dev/null 2021-01-15T17:57:15.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:15.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj87 /dev/null 2021-01-15T17:57:16.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj88 /dev/null 2021-01-15T17:57:16.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj89 /dev/null 2021-01-15T17:57:16.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj90 /dev/null 2021-01-15T17:57:16.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj91 /dev/null 2021-01-15T17:57:16.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj92 /dev/null 2021-01-15T17:57:16.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj93 /dev/null 2021-01-15T17:57:16.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj94 /dev/null 2021-01-15T17:57:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj95 /dev/null 2021-01-15T17:57:16.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj96 /dev/null 2021-01-15T17:57:16.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:16.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj97 /dev/null 2021-01-15T17:57:17.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj98 /dev/null 2021-01-15T17:57:17.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj99 /dev/null 2021-01-15T17:57:17.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj100 /dev/null 2021-01-15T17:57:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj101 /dev/null 2021-01-15T17:57:17.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj102 /dev/null 2021-01-15T17:57:17.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj103 /dev/null 2021-01-15T17:57:17.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj104 /dev/null 2021-01-15T17:57:17.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:17.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj105 /dev/null 2021-01-15T17:57:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj106 /dev/null 2021-01-15T17:57:18.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj107 /dev/null 2021-01-15T17:57:18.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj108 /dev/null 2021-01-15T17:57:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj109 /dev/null 2021-01-15T17:57:18.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj110 /dev/null 2021-01-15T17:57:18.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj111 /dev/null 2021-01-15T17:57:18.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj112 /dev/null 2021-01-15T17:57:18.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj113 /dev/null 2021-01-15T17:57:18.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:18.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj114 /dev/null 2021-01-15T17:57:19.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj115 /dev/null 2021-01-15T17:57:19.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj116 /dev/null 2021-01-15T17:57:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj117 /dev/null 2021-01-15T17:57:19.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj118 /dev/null 2021-01-15T17:57:19.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj119 /dev/null 2021-01-15T17:57:19.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj120 /dev/null 2021-01-15T17:57:19.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj121 /dev/null 2021-01-15T17:57:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj122 /dev/null 2021-01-15T17:57:19.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:19.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj123 /dev/null 2021-01-15T17:57:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj124 /dev/null 2021-01-15T17:57:20.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj125 /dev/null 2021-01-15T17:57:20.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj126 /dev/null 2021-01-15T17:57:20.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj127 /dev/null 2021-01-15T17:57:20.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj128 /dev/null 2021-01-15T17:57:20.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj129 /dev/null 2021-01-15T17:57:20.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj130 /dev/null 2021-01-15T17:57:20.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj131 /dev/null 2021-01-15T17:57:20.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:20.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj132 /dev/null 2021-01-15T17:57:21.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj133 /dev/null 2021-01-15T17:57:21.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj134 /dev/null 2021-01-15T17:57:21.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj135 /dev/null 2021-01-15T17:57:21.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj136 /dev/null 2021-01-15T17:57:21.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj137 /dev/null 2021-01-15T17:57:21.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj138 /dev/null 2021-01-15T17:57:21.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj139 /dev/null 2021-01-15T17:57:21.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj140 /dev/null 2021-01-15T17:57:21.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:21.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj141 /dev/null 2021-01-15T17:57:22.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj142 /dev/null 2021-01-15T17:57:22.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj143 /dev/null 2021-01-15T17:57:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj144 /dev/null 2021-01-15T17:57:22.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj145 /dev/null 2021-01-15T17:57:22.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj146 /dev/null 2021-01-15T17:57:22.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj147 /dev/null 2021-01-15T17:57:22.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj148 /dev/null 2021-01-15T17:57:22.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj149 /dev/null 2021-01-15T17:57:22.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:22.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj150 /dev/null 2021-01-15T17:57:23.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj151 /dev/null 2021-01-15T17:57:23.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj152 /dev/null 2021-01-15T17:57:23.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj153 /dev/null 2021-01-15T17:57:23.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj154 /dev/null 2021-01-15T17:57:23.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj155 /dev/null 2021-01-15T17:57:23.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj156 /dev/null 2021-01-15T17:57:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj157 /dev/null 2021-01-15T17:57:23.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj158 /dev/null 2021-01-15T17:57:23.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:23.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj159 /dev/null 2021-01-15T17:57:24.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj160 /dev/null 2021-01-15T17:57:24.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj161 /dev/null 2021-01-15T17:57:24.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj162 /dev/null 2021-01-15T17:57:24.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj163 /dev/null 2021-01-15T17:57:24.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj164 /dev/null 2021-01-15T17:57:24.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj165 /dev/null 2021-01-15T17:57:24.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj166 /dev/null 2021-01-15T17:57:24.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj167 /dev/null 2021-01-15T17:57:24.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:24.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj168 /dev/null 2021-01-15T17:57:25.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj169 /dev/null 2021-01-15T17:57:25.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj170 /dev/null 2021-01-15T17:57:25.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj171 /dev/null 2021-01-15T17:57:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj172 /dev/null 2021-01-15T17:57:25.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj173 /dev/null 2021-01-15T17:57:25.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj174 /dev/null 2021-01-15T17:57:25.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj175 /dev/null 2021-01-15T17:57:25.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj176 /dev/null 2021-01-15T17:57:25.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:25.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj177 /dev/null 2021-01-15T17:57:26.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj178 /dev/null 2021-01-15T17:57:26.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj179 /dev/null 2021-01-15T17:57:26.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj180 /dev/null 2021-01-15T17:57:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj181 /dev/null 2021-01-15T17:57:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj182 /dev/null 2021-01-15T17:57:26.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj183 /dev/null 2021-01-15T17:57:26.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj184 /dev/null 2021-01-15T17:57:26.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj185 /dev/null 2021-01-15T17:57:26.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:26.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj186 /dev/null 2021-01-15T17:57:27.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj187 /dev/null 2021-01-15T17:57:27.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj188 /dev/null 2021-01-15T17:57:27.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj189 /dev/null 2021-01-15T17:57:27.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj190 /dev/null 2021-01-15T17:57:27.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj191 /dev/null 2021-01-15T17:57:27.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj192 /dev/null 2021-01-15T17:57:27.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj193 /dev/null 2021-01-15T17:57:27.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj194 /dev/null 2021-01-15T17:57:27.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:27.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj195 /dev/null 2021-01-15T17:57:28.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:28.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj196 /dev/null 2021-01-15T17:57:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj197 /dev/null 2021-01-15T17:57:28.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:28.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj198 /dev/null 2021-01-15T17:57:28.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:28.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj199 /dev/null 2021-01-15T17:57:28.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:457: TEST_backfill_remapped: for i in $(seq 1 $objects) 2021-01-15T17:57:28.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:459: TEST_backfill_remapped: rados -p test put obj200 /dev/null 2021-01-15T17:57:28.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:462: TEST_backfill_remapped: get_pg test obj1 2021-01-15T17:57:28.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T17:57:28.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T17:57:28.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T17:57:28.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T17:57:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:462: TEST_backfill_remapped: local PG=1.0 2021-01-15T17:57:28.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:464: TEST_backfill_remapped: get_primary test obj1 2021-01-15T17:57:28.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:57:28.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:57:28.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:57:28.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:57:29.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:464: TEST_backfill_remapped: local primary=1 2021-01-15T17:57:29.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:465: TEST_backfill_remapped: get_not_primary test obj1 2021-01-15T17:57:29.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T17:57:29.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T17:57:29.396 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T17:57:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:57:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:57:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:57:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:57:29.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T17:57:29.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T17:57:29.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T17:57:30.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:465: TEST_backfill_remapped: local otherosd=0 2021-01-15T17:57:30.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:467: TEST_backfill_remapped: ceph osd set nobackfill 2021-01-15T17:57:30.895 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:57:30.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:468: TEST_backfill_remapped: ceph osd out osd.0 2021-01-15T17:57:31.538 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T17:57:31.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: get_osds test obj1 2021-01-15T17:57:31.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2021-01-15T17:57:31.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2021-01-15T17:57:31.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2021-01-15T17:57:31.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T17:57:31.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2021-01-15T17:57:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:57:31.950 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T17:57:31.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 2 3 2021-01-15T17:57:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: for i in $(get_osds $poolname obj1) 2021-01-15T17:57:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:471: TEST_backfill_remapped: '[' 1 = 1 -o 1 = 0 ']' 2021-01-15T17:57:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:473: TEST_backfill_remapped: continue 2021-01-15T17:57:31.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:469: TEST_backfill_remapped: for i in $(get_osds $poolname obj1) 2021-01-15T17:57:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:471: TEST_backfill_remapped: '[' 2 = 1 -o 2 = 0 ']' 2021-01-15T17:57:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:475: TEST_backfill_remapped: ceph osd out osd.2 2021-01-15T17:57:32.705 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T17:57:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:476: TEST_backfill_remapped: break 2021-01-15T17:57:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:478: TEST_backfill_remapped: ceph osd out osd.1 2021-01-15T17:57:33.568 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T17:57:33.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:479: TEST_backfill_remapped: ceph osd pool set test size 2 2021-01-15T17:57:34.531 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T17:57:34.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:480: TEST_backfill_remapped: sleep 2 2021-01-15T17:57:36.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:484: TEST_backfill_remapped: get_primary test obj1 2021-01-15T17:57:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T17:57:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T17:57:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T17:57:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T17:57:36.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:484: TEST_backfill_remapped: primary=0 2021-01-15T17:57:36.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:486: TEST_backfill_remapped: ceph osd unset nobackfill 2021-01-15T17:57:37.813 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:57:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:487: TEST_backfill_remapped: ceph tell osd.0 get_latest_osdmap 2021-01-15T17:57:37.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:488: TEST_backfill_remapped: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T17:57:38.090 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T17:57:38.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:490: TEST_backfill_remapped: sleep 2 2021-01-15T17:57:40.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:492: TEST_backfill_remapped: wait_for_clean 2021-01-15T17:57:40.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:57:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:57:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:57:40.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:57:40.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:57:40.103 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:57:40.103 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:57:40.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:57:40.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:57:40.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:57:40.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:57:40.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:57:40.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:57:40.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:57:40.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:57:40.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:57:40.600 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:57:40.600 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:57:40.600 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T17:57:40.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:57:40.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:40.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:57:40.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2021-01-15T17:57:40.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2021-01-15T17:57:40.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2021-01-15T17:57:40.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:40.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:57:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672977 2021-01-15T17:57:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672977 2021-01-15T17:57:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977' 2021-01-15T17:57:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:40.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:57:41.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509456 2021-01-15T17:57:41.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509456 2021-01-15T17:57:41.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509456' 2021-01-15T17:57:41.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:57:41.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:57:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345934 2021-01-15T17:57:41.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345934 2021-01-15T17:57:41.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672977 2-64424509456 3-85899345934' 2021-01-15T17:57:41.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:41.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2021-01-15T17:57:41.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:41.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:57:41.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2021-01-15T17:57:41.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:41.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2021-01-15T17:57:41.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-01-15T17:57:41.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:57:41.155 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836499 2021-01-15T17:57:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836499 2021-01-15T17:57:41.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:57:42.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:57:42.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:57:42.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836499 2021-01-15T17:57:42.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:42.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672977 2021-01-15T17:57:42.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:42.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:57:42.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672977 2021-01-15T17:57:42.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672977 2021-01-15T17:57:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2021-01-15T17:57:42.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:57:42.955 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672977 2021-01-15T17:57:43.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672977 -lt 42949672977 2021-01-15T17:57:43.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:43.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509456 2021-01-15T17:57:43.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:43.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:57:43.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-01-15T17:57:43.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:43.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509456 2021-01-15T17:57:43.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-01-15T17:57:43.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:57:43.354 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509456 2021-01-15T17:57:43.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509456 -lt 64424509456 2021-01-15T17:57:43.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:57:43.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345934 2021-01-15T17:57:43.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:57:43.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:57:43.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345934 2021-01-15T17:57:43.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:57:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345934 2021-01-15T17:57:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2021-01-15T17:57:43.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:57:43.772 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345934 2021-01-15T17:57:44.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345934 -lt 85899345934 2021-01-15T17:57:44.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:57:44.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:44.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:57:44.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:44.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:44.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:44.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:44.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:44.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:44.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:45.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:45.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:45.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:45.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:45.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:45.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:57:45.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:57:45.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:57:45.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:45.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:45.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:45.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:45.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:45.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:45.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:45.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:45.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:45.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:46.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:46.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:46.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:46.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:46.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:46.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:46.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:46.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:46.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:46.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:47.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:57:47.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:57:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:47.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:47.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:47.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:47.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:47.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:47.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:47.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:47.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:47.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:47.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:47.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:47.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:48.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:48.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:48.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:48.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:48.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:48.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:48.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:48.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:48.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:48.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:57:48.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:57:48.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:48.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:48.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:48.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:48.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:48.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:48.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:48.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:48.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:48.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:48.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:49.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:49.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:49.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:49.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:49.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:49.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:49.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:49.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:49.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:49.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:49.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:49.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:49.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:57:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:57:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:50.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:50.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:50.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:50.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:50.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:50.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:50.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:50.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:51.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:51.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:51.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:51.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:51.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:51.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:51.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:51.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:51.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:51.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:57:51.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:57:51.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:51.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:51.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:51.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:51.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:51.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:51.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:51.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:51.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:51.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:51.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:52.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:52.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:52.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:52.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:52.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:52.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:52.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:52.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:52.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:57:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:57:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:53.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:53.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:53.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:53.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:53.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:53.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:53.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:53.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:53.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:53.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:54.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:54.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:54.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:54.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:54.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:54.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:54.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:54.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:54.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:57:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:57:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:54.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:54.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:54.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:54.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:54.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:54.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:54.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:54.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:54.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:54.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:55.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:55.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:55.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:55.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:55.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:55.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:55.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:55.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:55.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:55.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:55.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:55.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:55.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:56.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2021-01-15T17:57:56.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2021-01-15T17:57:56.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:56.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:56.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:56.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:56.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:56.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:56.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:56.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:56.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:56.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:56.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:56.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:57.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:57.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:57.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:57:57.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:57:57.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:57.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:57.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:58.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:58.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:58.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:58.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:58.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:58.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:58.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:57:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:57:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:57:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:57:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:57:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:57:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:57:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:57:58.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:57:58.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:57:59.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:57:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:57:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:57:59.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:57:59.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:57:59.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:57:59.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:57:59.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:57:59.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:57:59.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:57:59.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:57:59.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:57:59.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:57:59.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:57:59.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:57:59.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:57:59.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:00.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:00.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:00.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:00.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:00.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:00.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:00.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:00.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:00.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:01.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:58:01.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:58:01.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:01.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:01.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:01.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:01.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:01.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:01.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:01.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:01.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:01.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:01.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:01.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:01.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:01.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:02.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:02.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:02.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:02.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:02.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:02.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:02.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:02.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:02.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:02.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:58:02.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:58:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:02.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:02.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:02.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:02.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:02.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:02.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:02.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:03.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:03.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:03.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:03.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:03.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:03.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:03.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:03.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:04.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:58:04.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:58:04.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:04.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:04.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:04.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:04.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:04.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:04.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:04.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:04.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:04.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:04.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:04.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:04.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:04.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:04.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:05.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:05.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:05.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:58:05.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:58:05.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:05.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:05.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:05.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:05.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:05.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:05.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:05.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:05.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:05.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:05.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:06.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:06.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:06.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:06.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:06.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:06.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:06.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:06.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:06.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:06.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:06.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:06.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:07.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T17:58:07.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T17:58:07.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:07.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:07.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:07.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:07.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:07.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:07.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:07.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:07.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:07.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:07.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:07.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:07.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:07.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:07.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:08.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:08.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:08.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:08.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:08.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:08.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:08.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:58:08.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:58:08.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:08.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:08.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:08.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:08.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:08.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:08.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:08.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:08.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:08.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:08.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:09.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:58:09.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:09.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:09.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:09.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:58:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:58:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:58:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:58:09.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:58:09.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:58:09.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:58:09.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:58:09.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:58:10.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:58:10.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:58:10.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:58:10.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:58:10.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:58:10.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:58:10.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:58:10.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:58:10.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:58:10.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:58:10.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:58:10.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:58:10.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:58:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:58:10.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:58:10.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:58:10.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:58:11.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:58:11.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:58:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:58:11.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:494: TEST_backfill_remapped: expr 200 '*' 2 2021-01-15T17:58:11.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:494: TEST_backfill_remapped: local misplaced=400 2021-01-15T17:58:11.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:496: TEST_backfill_remapped: check td/osd-backfill-stats 1.0 0 replicated 0 0 400 200 '' '' false 2021-01-15T17:58:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T17:58:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T17:58:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=0 2021-01-15T17:58:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T17:58:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=0 2021-01-15T17:58:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T17:58:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2021-01-15T17:58:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2021-01-15T17:58:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T17:58:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T17:58:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=false 2021-01-15T17:58:11.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' false = true ']' 2021-01-15T17:58:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T17:58:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T17:58:11.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T17:58:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.203 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T17:58:11.203 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T17:58:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[3]/[0,1]' 2021-01-15T17:58:11.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[3]/[0,1]' ']' 2021-01-15T17:58:11.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T17:58:11.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:58:11.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [3]/[0,1] ' 2021-01-15T17:58:11.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T17:58:11.213 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T17:58:11.213 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:58:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=0 2021-01-15T17:58:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 0 ']' 2021-01-15T17:58:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 0 0 2021-01-15T17:58:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T17:58:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:58:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T17:58:11.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:58:11.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:58:11.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T17:58:11.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:58:11.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=0 2021-01-15T17:58:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 0 ']' 2021-01-15T17:58:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 0 0 2021-01-15T17:58:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T17:58:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:58:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T17:58:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:58:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T17:58:11.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:58:11.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [3]/[0,1] ' 2021-01-15T17:58:11.224 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T17:58:11.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T17:58:11.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:58:11.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2021-01-15T17:58:11.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2021-01-15T17:58:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2021-01-15T17:58:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T17:58:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T17:58:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T17:58:11.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T17:58:11.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T17:58:11.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.0.log 2021-01-15T17:58:11.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T17:58:11.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T17:58:11.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=201 2021-01-15T17:58:11.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 201 ']' 2021-01-15T17:58:11.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 201 200 2021-01-15T17:58:11.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=201 2021-01-15T17:58:11.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T17:58:11.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2021-01-15T17:58:11.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T17:58:11.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T17:58:11.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:498: TEST_backfill_remapped: delete_pool test 2021-01-15T17:58:11.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T17:58:11.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T17:58:11.806 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T17:58:11.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:499: TEST_backfill_remapped: kill_daemons td/osd-backfill-stats 2021-01-15T17:58:11.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:58:11.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:58:11.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:58:11.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:58:11.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:58:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:58:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T17:58:12.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:58:12.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:58:12.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:58:12.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:58:12.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:58:12.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:58:12.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:58:12.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:58:12.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:58:12.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:58:12.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:58:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:58:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:58:12.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:58:12.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:58:12.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:58:12.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:58:12.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:58:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:58:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:58:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:58:12.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:58:12.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:58:12.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:58:12.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:58:12.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T17:58:12.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T17:58:12.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T17:58:12.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T17:58:12.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T17:58:12.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T17:58:12.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T17:58:12.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T17:58:12.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T17:58:12.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T17:58:12.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T17:58:12.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T17:58:12.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T17:58:12.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T17:58:12.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T17:58:12.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T17:58:12.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T17:58:12.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:58:12.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T17:58:12.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T17:58:12.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T17:58:12.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T17:58:12.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T17:58:12.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T17:58:12.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T17:58:12.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T17:58:12.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T17:58:12.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T17:58:12.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T17:58:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T17:58:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T17:58:12.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T17:58:12.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T17:58:12.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T17:58:12.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T17:58:12.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup td/osd-backfill-stats 2021-01-15T17:58:12.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:133: TEST_backfill_sizeup: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:135: TEST_backfill_sizeup: run_mon td/osd-backfill-stats a 2021-01-15T17:58:12.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T17:58:12.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T17:58:12.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T17:58:12.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T17:58:12.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T17:58:12.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T17:58:12.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:12.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:12.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:12.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:12.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T17:58:12.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T17:58:12.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T17:58:12.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:58:12.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:58:12.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T17:58:12.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T17:58:12.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:58:12.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:58:12.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:58:12.622 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:58:12.623 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.623 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.623 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:58:12.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:58:12.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T17:58:12.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T17:58:12.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T17:58:12.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T17:58:12.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T17:58:12.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T17:58:12.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T17:58:12.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T17:58:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T17:58:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T17:58:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T17:58:12.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T17:58:12.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T17:58:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:136: TEST_backfill_sizeup: run_mgr td/osd-backfill-stats x 2021-01-15T17:58:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T17:58:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T17:58:12.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T17:58:12.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T17:58:12.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T17:58:12.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T17:58:13.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T17:58:13.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:13.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:13.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:13.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:13.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:13.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:13.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:58:13.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T17:58:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:137: TEST_backfill_sizeup: export CEPH_ARGS 2021-01-15T17:58:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:138: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 0 2021-01-15T17:58:13.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T17:58:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:13.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T17:58:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:13.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:13.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T17:58:13.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T17:58:13.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:13.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:13.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:13.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:13.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:13.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:13.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:13.353 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:13.353 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:13.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:13.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:13.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:13.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:13.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:13.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:13.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:13.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:13.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:13.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:13.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:13.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:13.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T17:58:13.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:13.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4fbff541-cd42-4d59-b275-0f67cdb49033 2021-01-15T17:58:13.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 4fbff541-cd42-4d59-b275-0f67cdb49033' 2021-01-15T17:58:13.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:13.360 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 4fbff541-cd42-4d59-b275-0f67cdb49033 2021-01-15T17:58:13.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC11wFgtaavFRAAphVFZjpKNTJ3iuM0ga7Zow== 2021-01-15T17:58:13.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC11wFgtaavFRAAphVFZjpKNTJ3iuM0ga7Zow=="}' 2021-01-15T17:58:13.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4fbff541-cd42-4d59-b275-0f67cdb49033 -i td/osd-backfill-stats/0/new.json 2021-01-15T17:58:13.704 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:13.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T17:58:13.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC11wFgtaavFRAAphVFZjpKNTJ3iuM0ga7Zow== --osd-uuid 4fbff541-cd42-4d59-b275-0f67cdb49033 2021-01-15T17:58:13.740 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.740+0000 7f8017920080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:13.742 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.742+0000 7f8017920080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:13.744 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.744+0000 7f8017920080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:13.745 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.745+0000 7f8017920080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:58:13.745 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.745+0000 7f8017920080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:58:13.746 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.745+0000 7f8017920080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T17:58:13.785 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:13.785+0000 7f8017920080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T17:58:18.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T17:58:18.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:18.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T17:58:18.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:18.388 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T17:58:18.909 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T17:58:18.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T17:58:18.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:18.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:18.932 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:18.931+0000 7f530910a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:18.950+0000 7f530910a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:18.952 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:18.952+0000 7f530910a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:19.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T17:58:19.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:19.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T17:58:19.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:19.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:19.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:19.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:19.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:19.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:58:19.311 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:19.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:19.467+0000 7f530910a080 -1 Falling back to public interface 2021-01-15T17:58:19.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:58:20.277 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:20.277+0000 7f530910a080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T17:58:20.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:58:20.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:20.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:58:20.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:20.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T17:58:20.711 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:21.137 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/632811,v1:127.0.0.1:6803/632811] [v2:127.0.0.1:6804/632811,v1:127.0.0.1:6805/632811] exists,up 4fbff541-cd42-4d59-b275-0f67cdb49033 2021-01-15T17:58:21.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:58:21.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:58:21.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:58:21.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:139: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 1 2021-01-15T17:58:21.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:21.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:21.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T17:58:21.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:21.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T17:58:21.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:21.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:21.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:21.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:21.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T17:58:21.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T17:58:21.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:21.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:21.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:21.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:21.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:21.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:21.142 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:21.142 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:21.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:21.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:21.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:21.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:21.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:21.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:21.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:21.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:21.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:21.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:21.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:21.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:21.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:21.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T17:58:21.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:21.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7046bb92-07a0-4364-8fa0-e712a6f16f74 2021-01-15T17:58:21.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 7046bb92-07a0-4364-8fa0-e712a6f16f74' 2021-01-15T17:58:21.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:21.147 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 7046bb92-07a0-4364-8fa0-e712a6f16f74 2021-01-15T17:58:21.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC91wFgCk8bCRAAWkAbJPJ2rdCkIARWxDBnCQ== 2021-01-15T17:58:21.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC91wFgCk8bCRAAWkAbJPJ2rdCkIARWxDBnCQ=="}' 2021-01-15T17:58:21.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7046bb92-07a0-4364-8fa0-e712a6f16f74 -i td/osd-backfill-stats/1/new.json 2021-01-15T17:58:21.678 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T17:58:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC91wFgCk8bCRAAWkAbJPJ2rdCkIARWxDBnCQ== --osd-uuid 7046bb92-07a0-4364-8fa0-e712a6f16f74 2021-01-15T17:58:21.716 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.716+0000 7fa73b058080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:21.718 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.717+0000 7fa73b058080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:21.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.718+0000 7fa73b058080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:21.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.719+0000 7fa73b058080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:58:21.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.719+0000 7fa73b058080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:58:21.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.719+0000 7fa73b058080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T17:58:21.764 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:21.763+0000 7fa73b058080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T17:58:26.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T17:58:26.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:26.268 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T17:58:26.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T17:58:26.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:26.796 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T17:58:26.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T17:58:26.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:26.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:26.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:26.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:26.819 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:26.819+0000 7fc7c20f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:26.961 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:26.961+0000 7fc7c20f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:26.963 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:26.963+0000 7fc7c20f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:27.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T17:58:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T17:58:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:27.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:27.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:27.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:27.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:27.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:58:27.264 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:27.478 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:27.477+0000 7fc7c20f0080 -1 Falling back to public interface 2021-01-15T17:58:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:58:28.249 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:28.249+0000 7fc7c20f0080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T17:58:28.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:58:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:28.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:58:28.671 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:28.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:28.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T17:58:29.088 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/634093,v1:127.0.0.1:6811/634093] [v2:127.0.0.1:6812/634093,v1:127.0.0.1:6813/634093] exists,up 7046bb92-07a0-4364-8fa0-e712a6f16f74 2021-01-15T17:58:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:58:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:58:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:58:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:140: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 2 2021-01-15T17:58:29.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T17:58:29.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T17:58:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:29.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T17:58:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T17:58:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:29.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:29.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:29.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:29.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:29.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:29.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:29.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:29.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:29.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:29.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:29.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:29.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:29.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:29.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:29.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:29.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:29.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:29.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T17:58:29.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:29.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=13bd4869-845a-4d2f-8921-de23dadfe70e 2021-01-15T17:58:29.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 13bd4869-845a-4d2f-8921-de23dadfe70e' 2021-01-15T17:58:29.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:29.102 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 13bd4869-845a-4d2f-8921-de23dadfe70e 2021-01-15T17:58:29.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDF1wFgSOMyBhAAakSDh4ypMe7+IqNt5G0BPg== 2021-01-15T17:58:29.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDF1wFgSOMyBhAAakSDh4ypMe7+IqNt5G0BPg=="}' 2021-01-15T17:58:29.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 13bd4869-845a-4d2f-8921-de23dadfe70e -i td/osd-backfill-stats/2/new.json 2021-01-15T17:58:29.658 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T17:58:29.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T17:58:29.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDF1wFgSOMyBhAAakSDh4ypMe7+IqNt5G0BPg== --osd-uuid 13bd4869-845a-4d2f-8921-de23dadfe70e 2021-01-15T17:58:29.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.693+0000 7f8466fbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:29.695 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.695+0000 7f8466fbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:29.696 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.696+0000 7f8466fbb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:29.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.696+0000 7f8466fbb080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:58:29.697 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.696+0000 7f8466fbb080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:58:29.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.696+0000 7f8466fbb080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T17:58:29.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:29.748+0000 7f8466fbb080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T17:58:34.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T17:58:34.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:34.270 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T17:58:34.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T17:58:34.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:34.797 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T17:58:34.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T17:58:34.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:34.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:34.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:34.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:34.821 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:34.820+0000 7f43c883c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:34.838 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:34.837+0000 7f43c883c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:34.839 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:34.839+0000 7f43c883c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:35.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T17:58:35.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:35.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T17:58:35.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:35.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:35.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:35.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:35.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:35.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:58:35.211 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:35.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:35.359+0000 7f43c883c080 -1 Falling back to public interface 2021-01-15T17:58:35.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:58:36.182 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:36.181+0000 7f43c883c080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T17:58:36.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:58:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:58:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T17:58:36.612 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:37.004 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/635347,v1:127.0.0.1:6819/635347] [v2:127.0.0.1:6820/635347,v1:127.0.0.1:6821/635347] exists,up 13bd4869-845a-4d2f-8921-de23dadfe70e 2021-01-15T17:58:37.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:58:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:58:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:58:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:141: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 3 2021-01-15T17:58:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T17:58:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T17:58:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T17:58:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T17:58:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:37.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:37.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:37.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:37.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:37.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:37.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:37.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:37.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:37.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:37.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:37.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:37.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T17:58:37.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1f89486a-17d9-471e-9231-23fc9083332f 2021-01-15T17:58:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 1f89486a-17d9-471e-9231-23fc9083332f' 2021-01-15T17:58:37.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:37.013 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 1f89486a-17d9-471e-9231-23fc9083332f 2021-01-15T17:58:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDN1wFgrQgtARAAwLFvfbmLHEH6D/TbaECHyA== 2021-01-15T17:58:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDN1wFgrQgtARAAwLFvfbmLHEH6D/TbaECHyA=="}' 2021-01-15T17:58:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1f89486a-17d9-471e-9231-23fc9083332f -i td/osd-backfill-stats/3/new.json 2021-01-15T17:58:37.616 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T17:58:37.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T17:58:37.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDN1wFgrQgtARAAwLFvfbmLHEH6D/TbaECHyA== --osd-uuid 1f89486a-17d9-471e-9231-23fc9083332f 2021-01-15T17:58:37.653 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.652+0000 7f04dd63f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:37.655 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.654+0000 7f04dd63f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:37.656 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.655+0000 7f04dd63f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:37.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.656+0000 7f04dd63f080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:58:37.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.656+0000 7f04dd63f080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:58:37.657 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.656+0000 7f04dd63f080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T17:58:37.701 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:37.700+0000 7f04dd63f080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T17:58:42.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T17:58:42.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:42.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T17:58:42.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:42.466 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T17:58:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T17:58:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:42.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:42.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:42.998 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T17:58:43.019 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:43.018+0000 7fba883cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:43.020+0000 7fba883cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:43.023 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:43.022+0000 7fba883cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:43.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T17:58:43.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:43.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T17:58:43.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:43.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:58:43.396 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:43.538 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:43.538+0000 7fba883cc080 -1 Falling back to public interface 2021-01-15T17:58:43.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:58:44.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:44.315+0000 7fba883cc080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T17:58:44.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:58:44.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:44.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:58:44.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:44.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T17:58:44.788 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:45.220 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/636611,v1:127.0.0.1:6827/636611] [v2:127.0.0.1:6828/636611,v1:127.0.0.1:6829/636611] exists,up 1f89486a-17d9-471e-9231-23fc9083332f 2021-01-15T17:58:45.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:58:45.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:58:45.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:58:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:142: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 4 2021-01-15T17:58:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T17:58:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T17:58:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T17:58:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T17:58:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:45.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:45.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:45.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:45.225 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:45.225 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:45.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:45.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:45.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:45.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:45.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T17:58:45.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:45.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=96cf2b07-1535-47c4-b229-d5e612798f0c 2021-01-15T17:58:45.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 96cf2b07-1535-47c4-b229-d5e612798f0c' 2021-01-15T17:58:45.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:45.230 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 96cf2b07-1535-47c4-b229-d5e612798f0c 2021-01-15T17:58:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDV1wFgn/AbDhAA1Ww7G650P0DzcKi05d8v3w== 2021-01-15T17:58:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDV1wFgn/AbDhAA1Ww7G650P0DzcKi05d8v3w=="}' 2021-01-15T17:58:45.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 96cf2b07-1535-47c4-b229-d5e612798f0c -i td/osd-backfill-stats/4/new.json 2021-01-15T17:58:45.811 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T17:58:45.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T17:58:45.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDV1wFgn/AbDhAA1Ww7G650P0DzcKi05d8v3w== --osd-uuid 96cf2b07-1535-47c4-b229-d5e612798f0c 2021-01-15T17:58:45.848 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.847+0000 7f04b85b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.849+0000 7f04b85b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:45.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.849+0000 7f04b85b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:45.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.850+0000 7f04b85b2080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:58:45.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.850+0000 7f04b85b2080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:58:45.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.850+0000 7f04b85b2080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T17:58:45.895 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:45.894+0000 7f04b85b2080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T17:58:50.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T17:58:50.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:50.605 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T17:58:50.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T17:58:50.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:51.131 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T17:58:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T17:58:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:51.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:51.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:51.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:51.153+0000 7fc555367080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:51.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:51.172+0000 7fc555367080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:51.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:51.174+0000 7fc555367080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:51.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T17:58:51.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:51.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T17:58:51.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:51.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:51.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:51.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:51.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:51.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:58:51.559 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:51.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:51.693+0000 7fc555367080 -1 Falling back to public interface 2021-01-15T17:58:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:58:52.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:52.507+0000 7fc555367080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T17:58:52.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:58:52.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:52.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:58:52.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:52.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T17:58:52.950 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:58:53.408 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/637875,v1:127.0.0.1:6835/637875] [v2:127.0.0.1:6836/637875,v1:127.0.0.1:6837/637875] exists,up 96cf2b07-1535-47c4-b229-d5e612798f0c 2021-01-15T17:58:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:58:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:58:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:58:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:143: TEST_backfill_sizeup: run_osd td/osd-backfill-stats 5 2021-01-15T17:58:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T17:58:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T17:58:53.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T17:58:53.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T17:58:53.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T17:58:53.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T17:58:53.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T17:58:53.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T17:58:53.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T17:58:53.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T17:58:53.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T17:58:53.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T17:58:53.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T17:58:53.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T17:58:53.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T17:58:53.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T17:58:53.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T17:58:53.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T17:58:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T17:58:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T17:58:53.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T17:58:53.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T17:58:53.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T17:58:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T17:58:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T17:58:53.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T17:58:53.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T17:58:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0b42fff6-9f4c-4ba0-8e55-62028348a0b7 2021-01-15T17:58:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 0b42fff6-9f4c-4ba0-8e55-62028348a0b7' 2021-01-15T17:58:53.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T17:58:53.418 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 0b42fff6-9f4c-4ba0-8e55-62028348a0b7 2021-01-15T17:58:53.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDd1wFg2uNFGRAA1z6+BemOxr0BVC0MDZu2Jg== 2021-01-15T17:58:53.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDd1wFg2uNFGRAA1z6+BemOxr0BVC0MDZu2Jg=="}' 2021-01-15T17:58:53.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0b42fff6-9f4c-4ba0-8e55-62028348a0b7 -i td/osd-backfill-stats/5/new.json 2021-01-15T17:58:53.952 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T17:58:53.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T17:58:53.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDd1wFg2uNFGRAA1z6+BemOxr0BVC0MDZu2Jg== --osd-uuid 0b42fff6-9f4c-4ba0-8e55-62028348a0b7 2021-01-15T17:58:53.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:53.988+0000 7f832778e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:54.007 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.006+0000 7f832778e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.008+0000 7f832778e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:54.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.009+0000 7f832778e080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:58:54.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.009+0000 7f832778e080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:58:54.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.009+0000 7f832778e080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T17:58:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:54.073+0000 7f832778e080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T17:58:58.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T17:58:58.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T17:58:58.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T17:58:58.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T17:58:58.759 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T17:58:59.286 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T17:58:59.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T17:58:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T17:58:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T17:58:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T17:58:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T17:58:59.308 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:59.307+0000 7f7e8fd99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:59.309 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:59.309+0000 7f7e8fd99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:59.311 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:59.311+0000 7f7e8fd99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T17:58:59.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T17:58:59.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T17:58:59.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T17:58:59.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T17:58:59.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T17:58:59.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:58:59.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T17:58:59.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:58:59.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:58:59.684 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T17:58:59.826 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:58:59.825+0000 7f7e8fd99080 -1 Falling back to public interface 2021-01-15T17:59:00.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T17:59:00.617 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T17:59:00.616+0000 7f7e8fd99080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T17:59:01.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T17:59:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T17:59:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T17:59:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T17:59:01.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T17:59:01.071 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T17:59:01.482 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/639139,v1:127.0.0.1:6843/639139] [v2:127.0.0.1:6844/639139,v1:127.0.0.1:6845/639139] exists,up 0b42fff6-9f4c-4ba0-8e55-62028348a0b7 2021-01-15T17:59:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T17:59:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T17:59:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T17:59:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:145: TEST_backfill_sizeup: create_pool test 1 1 2021-01-15T17:59:01.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T17:59:02.131 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T17:59:02.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T17:59:03.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:146: TEST_backfill_sizeup: ceph osd pool set test size 1 --yes-i-really-mean-it 2021-01-15T17:59:03.779 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T17:59:03.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:148: TEST_backfill_sizeup: wait_for_clean 2021-01-15T17:59:03.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:59:03.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:59:03.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:59:03.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:59:03.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:59:03.790 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:59:03.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:59:03.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:59:03.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:59:03.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:59:03.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:59:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:59:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:59:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:59:03.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:59:04.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:59:04.298 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:59:04.299 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:59:04.299 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:59:04.299 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:59:04.299 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:59:04.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:59:04.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:59:04.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T17:59:04.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T17:59:04.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T17:59:04.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:59:04.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2021-01-15T17:59:04.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2021-01-15T17:59:04.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672969' 2021-01-15T17:59:04.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:59:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T17:59:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T17:59:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448' 2021-01-15T17:59:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:59:04.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345926 2021-01-15T17:59:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345926 2021-01-15T17:59:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345926' 2021-01-15T17:59:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:59:04.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T17:59:04.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T17:59:04.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345926 4-107374182405' 2021-01-15T17:59:04.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:04.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:59:05.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018883 2021-01-15T17:59:05.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018883 2021-01-15T17:59:05.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672969 2-64424509448 3-85899345926 4-107374182405 5-128849018883' 2021-01-15T17:59:05.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:05.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T17:59:05.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:05.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:59:05.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T17:59:05.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:05.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T17:59:05.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T17:59:05.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:59:05.127 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T17:59:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2021-01-15T17:59:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:05.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2021-01-15T17:59:05.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:05.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:59:05.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-01-15T17:59:05.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2021-01-15T17:59:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-01-15T17:59:05.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:59:05.524 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672969 2021-01-15T17:59:05.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672969 2021-01-15T17:59:05.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:05.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T17:59:05.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:59:05.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T17:59:05.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:05.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T17:59:05.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T17:59:05.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:59:05.938 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T17:59:06.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T17:59:06.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:06.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345926 2021-01-15T17:59:06.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:06.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:59:06.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345926 2021-01-15T17:59:06.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345926 2021-01-15T17:59:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2021-01-15T17:59:06.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:59:06.356 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345926 2021-01-15T17:59:06.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345926 -lt 85899345926 2021-01-15T17:59:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:06.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T17:59:06.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:59:06.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T17:59:06.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:06.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T17:59:06.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T17:59:06.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:59:06.751 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T17:59:07.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T17:59:07.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:07.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018883 2021-01-15T17:59:07.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:07.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:59:07.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018883 2021-01-15T17:59:07.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:07.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018883 2021-01-15T17:59:07.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-01-15T17:59:07.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:59:07.150 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018883 2021-01-15T17:59:07.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018884 -lt 128849018883 2021-01-15T17:59:07.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:59:07.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:07.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:08.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:59:08.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:08.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:08.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:08.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:08.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:08.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:08.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:08.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T17:59:08.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:08.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:08.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T17:59:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T17:59:08.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T17:59:08.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: seq 1 200 2021-01-15T17:59:08.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:08.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj1 /dev/null 2021-01-15T17:59:09.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj2 /dev/null 2021-01-15T17:59:09.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj3 /dev/null 2021-01-15T17:59:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj4 /dev/null 2021-01-15T17:59:09.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj5 /dev/null 2021-01-15T17:59:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj6 /dev/null 2021-01-15T17:59:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj7 /dev/null 2021-01-15T17:59:09.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj8 /dev/null 2021-01-15T17:59:09.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj9 /dev/null 2021-01-15T17:59:09.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj10 /dev/null 2021-01-15T17:59:09.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj11 /dev/null 2021-01-15T17:59:09.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj12 /dev/null 2021-01-15T17:59:09.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj13 /dev/null 2021-01-15T17:59:09.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj14 /dev/null 2021-01-15T17:59:09.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:09.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj15 /dev/null 2021-01-15T17:59:10.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj16 /dev/null 2021-01-15T17:59:10.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj17 /dev/null 2021-01-15T17:59:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj18 /dev/null 2021-01-15T17:59:10.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj19 /dev/null 2021-01-15T17:59:10.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj20 /dev/null 2021-01-15T17:59:10.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj21 /dev/null 2021-01-15T17:59:10.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj22 /dev/null 2021-01-15T17:59:10.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj23 /dev/null 2021-01-15T17:59:10.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj24 /dev/null 2021-01-15T17:59:10.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj25 /dev/null 2021-01-15T17:59:10.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj26 /dev/null 2021-01-15T17:59:10.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj27 /dev/null 2021-01-15T17:59:10.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj28 /dev/null 2021-01-15T17:59:10.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj29 /dev/null 2021-01-15T17:59:10.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj30 /dev/null 2021-01-15T17:59:10.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj31 /dev/null 2021-01-15T17:59:10.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:10.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj32 /dev/null 2021-01-15T17:59:11.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj33 /dev/null 2021-01-15T17:59:11.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj34 /dev/null 2021-01-15T17:59:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj35 /dev/null 2021-01-15T17:59:11.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj36 /dev/null 2021-01-15T17:59:11.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj37 /dev/null 2021-01-15T17:59:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj38 /dev/null 2021-01-15T17:59:11.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj39 /dev/null 2021-01-15T17:59:11.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj40 /dev/null 2021-01-15T17:59:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj41 /dev/null 2021-01-15T17:59:11.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj42 /dev/null 2021-01-15T17:59:11.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj43 /dev/null 2021-01-15T17:59:11.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj44 /dev/null 2021-01-15T17:59:11.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj45 /dev/null 2021-01-15T17:59:11.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:11.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj46 /dev/null 2021-01-15T17:59:12.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj47 /dev/null 2021-01-15T17:59:12.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj48 /dev/null 2021-01-15T17:59:12.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj49 /dev/null 2021-01-15T17:59:12.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj50 /dev/null 2021-01-15T17:59:12.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj51 /dev/null 2021-01-15T17:59:12.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj52 /dev/null 2021-01-15T17:59:12.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj53 /dev/null 2021-01-15T17:59:12.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj54 /dev/null 2021-01-15T17:59:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj55 /dev/null 2021-01-15T17:59:12.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj56 /dev/null 2021-01-15T17:59:12.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj57 /dev/null 2021-01-15T17:59:12.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj58 /dev/null 2021-01-15T17:59:12.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj59 /dev/null 2021-01-15T17:59:12.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj60 /dev/null 2021-01-15T17:59:12.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj61 /dev/null 2021-01-15T17:59:12.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:12.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj62 /dev/null 2021-01-15T17:59:13.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj63 /dev/null 2021-01-15T17:59:13.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj64 /dev/null 2021-01-15T17:59:13.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj65 /dev/null 2021-01-15T17:59:13.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj66 /dev/null 2021-01-15T17:59:13.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj67 /dev/null 2021-01-15T17:59:13.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj68 /dev/null 2021-01-15T17:59:13.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj69 /dev/null 2021-01-15T17:59:13.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj70 /dev/null 2021-01-15T17:59:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj71 /dev/null 2021-01-15T17:59:13.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj72 /dev/null 2021-01-15T17:59:13.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj73 /dev/null 2021-01-15T17:59:13.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj74 /dev/null 2021-01-15T17:59:13.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj75 /dev/null 2021-01-15T17:59:13.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:13.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj76 /dev/null 2021-01-15T17:59:14.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj77 /dev/null 2021-01-15T17:59:14.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj78 /dev/null 2021-01-15T17:59:14.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj79 /dev/null 2021-01-15T17:59:14.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj80 /dev/null 2021-01-15T17:59:14.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj81 /dev/null 2021-01-15T17:59:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj82 /dev/null 2021-01-15T17:59:14.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj83 /dev/null 2021-01-15T17:59:14.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj84 /dev/null 2021-01-15T17:59:14.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj85 /dev/null 2021-01-15T17:59:14.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj86 /dev/null 2021-01-15T17:59:14.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj87 /dev/null 2021-01-15T17:59:14.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj88 /dev/null 2021-01-15T17:59:14.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj89 /dev/null 2021-01-15T17:59:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj90 /dev/null 2021-01-15T17:59:14.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj91 /dev/null 2021-01-15T17:59:14.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj92 /dev/null 2021-01-15T17:59:14.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:14.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj93 /dev/null 2021-01-15T17:59:15.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj94 /dev/null 2021-01-15T17:59:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj95 /dev/null 2021-01-15T17:59:15.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj96 /dev/null 2021-01-15T17:59:15.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj97 /dev/null 2021-01-15T17:59:15.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj98 /dev/null 2021-01-15T17:59:15.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj99 /dev/null 2021-01-15T17:59:15.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj100 /dev/null 2021-01-15T17:59:15.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj101 /dev/null 2021-01-15T17:59:15.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj102 /dev/null 2021-01-15T17:59:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj103 /dev/null 2021-01-15T17:59:15.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj104 /dev/null 2021-01-15T17:59:15.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj105 /dev/null 2021-01-15T17:59:15.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj106 /dev/null 2021-01-15T17:59:15.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj107 /dev/null 2021-01-15T17:59:15.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:15.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj108 /dev/null 2021-01-15T17:59:16.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj109 /dev/null 2021-01-15T17:59:16.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj110 /dev/null 2021-01-15T17:59:16.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj111 /dev/null 2021-01-15T17:59:16.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj112 /dev/null 2021-01-15T17:59:16.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj113 /dev/null 2021-01-15T17:59:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj114 /dev/null 2021-01-15T17:59:16.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj115 /dev/null 2021-01-15T17:59:16.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj116 /dev/null 2021-01-15T17:59:16.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj117 /dev/null 2021-01-15T17:59:16.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj118 /dev/null 2021-01-15T17:59:16.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj119 /dev/null 2021-01-15T17:59:16.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj120 /dev/null 2021-01-15T17:59:16.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj121 /dev/null 2021-01-15T17:59:16.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj122 /dev/null 2021-01-15T17:59:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj123 /dev/null 2021-01-15T17:59:17.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj124 /dev/null 2021-01-15T17:59:17.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj125 /dev/null 2021-01-15T17:59:17.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj126 /dev/null 2021-01-15T17:59:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj127 /dev/null 2021-01-15T17:59:17.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj128 /dev/null 2021-01-15T17:59:17.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj129 /dev/null 2021-01-15T17:59:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj130 /dev/null 2021-01-15T17:59:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj131 /dev/null 2021-01-15T17:59:17.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj132 /dev/null 2021-01-15T17:59:17.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj133 /dev/null 2021-01-15T17:59:17.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj134 /dev/null 2021-01-15T17:59:17.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj135 /dev/null 2021-01-15T17:59:17.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj136 /dev/null 2021-01-15T17:59:17.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:17.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj137 /dev/null 2021-01-15T17:59:18.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj138 /dev/null 2021-01-15T17:59:18.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj139 /dev/null 2021-01-15T17:59:18.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj140 /dev/null 2021-01-15T17:59:18.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj141 /dev/null 2021-01-15T17:59:18.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj142 /dev/null 2021-01-15T17:59:18.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj143 /dev/null 2021-01-15T17:59:18.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj144 /dev/null 2021-01-15T17:59:18.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj145 /dev/null 2021-01-15T17:59:18.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj146 /dev/null 2021-01-15T17:59:18.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj147 /dev/null 2021-01-15T17:59:18.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj148 /dev/null 2021-01-15T17:59:18.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj149 /dev/null 2021-01-15T17:59:18.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj150 /dev/null 2021-01-15T17:59:18.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj151 /dev/null 2021-01-15T17:59:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj152 /dev/null 2021-01-15T17:59:18.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:18.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj153 /dev/null 2021-01-15T17:59:19.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj154 /dev/null 2021-01-15T17:59:19.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj155 /dev/null 2021-01-15T17:59:19.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj156 /dev/null 2021-01-15T17:59:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj157 /dev/null 2021-01-15T17:59:19.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj158 /dev/null 2021-01-15T17:59:19.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj159 /dev/null 2021-01-15T17:59:19.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj160 /dev/null 2021-01-15T17:59:19.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj161 /dev/null 2021-01-15T17:59:19.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj162 /dev/null 2021-01-15T17:59:19.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj163 /dev/null 2021-01-15T17:59:19.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj164 /dev/null 2021-01-15T17:59:19.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj165 /dev/null 2021-01-15T17:59:19.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj166 /dev/null 2021-01-15T17:59:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:19.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj167 /dev/null 2021-01-15T17:59:20.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj168 /dev/null 2021-01-15T17:59:20.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj169 /dev/null 2021-01-15T17:59:20.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj170 /dev/null 2021-01-15T17:59:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj171 /dev/null 2021-01-15T17:59:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj172 /dev/null 2021-01-15T17:59:20.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj173 /dev/null 2021-01-15T17:59:20.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj174 /dev/null 2021-01-15T17:59:20.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj175 /dev/null 2021-01-15T17:59:20.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj176 /dev/null 2021-01-15T17:59:20.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj177 /dev/null 2021-01-15T17:59:20.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj178 /dev/null 2021-01-15T17:59:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj179 /dev/null 2021-01-15T17:59:20.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj180 /dev/null 2021-01-15T17:59:20.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj181 /dev/null 2021-01-15T17:59:20.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:20.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj182 /dev/null 2021-01-15T17:59:21.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj183 /dev/null 2021-01-15T17:59:21.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj184 /dev/null 2021-01-15T17:59:21.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj185 /dev/null 2021-01-15T17:59:21.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj186 /dev/null 2021-01-15T17:59:21.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj187 /dev/null 2021-01-15T17:59:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj188 /dev/null 2021-01-15T17:59:21.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj189 /dev/null 2021-01-15T17:59:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj190 /dev/null 2021-01-15T17:59:21.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj191 /dev/null 2021-01-15T17:59:21.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj192 /dev/null 2021-01-15T17:59:21.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj193 /dev/null 2021-01-15T17:59:21.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj194 /dev/null 2021-01-15T17:59:21.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj195 /dev/null 2021-01-15T17:59:21.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj196 /dev/null 2021-01-15T17:59:21.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:21.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj197 /dev/null 2021-01-15T17:59:22.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:22.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj198 /dev/null 2021-01-15T17:59:22.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:22.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj199 /dev/null 2021-01-15T17:59:22.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:150: TEST_backfill_sizeup: for i in $(seq 1 $objects) 2021-01-15T17:59:22.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:152: TEST_backfill_sizeup: rados -p test put obj200 /dev/null 2021-01-15T17:59:22.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:155: TEST_backfill_sizeup: ceph osd set nobackfill 2021-01-15T17:59:22.870 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T17:59:22.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:156: TEST_backfill_sizeup: ceph osd pool set test size 3 2021-01-15T17:59:23.850 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T17:59:23.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:157: TEST_backfill_sizeup: sleep 2 2021-01-15T17:59:25.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:158: TEST_backfill_sizeup: ceph osd unset nobackfill 2021-01-15T17:59:26.558 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T17:59:26.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:160: TEST_backfill_sizeup: wait_for_clean 2021-01-15T17:59:26.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T17:59:26.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T17:59:26.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T17:59:26.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T17:59:26.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T17:59:26.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T17:59:26.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T17:59:26.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T17:59:26.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T17:59:26.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T17:59:26.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T17:59:26.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T17:59:26.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T17:59:26.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T17:59:26.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T17:59:27.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T17:59:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T17:59:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T17:59:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T17:59:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T17:59:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T17:59:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T17:59:27.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T17:59:27.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2021-01-15T17:59:27.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2021-01-15T17:59:27.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2021-01-15T17:59:27.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T17:59:27.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672976 2021-01-15T17:59:27.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672976 2021-01-15T17:59:27.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976' 2021-01-15T17:59:27.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T17:59:27.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509454 2021-01-15T17:59:27.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509454 2021-01-15T17:59:27.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454' 2021-01-15T17:59:27.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T17:59:27.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345933 2021-01-15T17:59:27.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345933 2021-01-15T17:59:27.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933' 2021-01-15T17:59:27.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T17:59:27.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182411 2021-01-15T17:59:27.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182411 2021-01-15T17:59:27.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933 4-107374182411' 2021-01-15T17:59:27.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T17:59:27.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T17:59:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018890 2021-01-15T17:59:27.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018890 2021-01-15T17:59:27.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672976 2-64424509454 3-85899345933 4-107374182411 5-128849018890' 2021-01-15T17:59:27.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:27.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2021-01-15T17:59:27.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:27.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T17:59:27.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2021-01-15T17:59:27.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:27.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2021-01-15T17:59:27.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-01-15T17:59:27.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:59:27.911 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836498 2021-01-15T17:59:28.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836496 -lt 21474836498 2021-01-15T17:59:28.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T17:59:29.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T17:59:29.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T17:59:29.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836498 2021-01-15T17:59:29.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:29.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672976 2021-01-15T17:59:29.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:29.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T17:59:29.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672976 2021-01-15T17:59:29.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:29.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672976 2021-01-15T17:59:29.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-01-15T17:59:29.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T17:59:29.770 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672976 2021-01-15T17:59:30.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672976 -lt 42949672976 2021-01-15T17:59:30.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:30.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509454 2021-01-15T17:59:30.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:30.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T17:59:30.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509454 2021-01-15T17:59:30.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:30.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509454 2021-01-15T17:59:30.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-01-15T17:59:30.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T17:59:30.168 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509454 2021-01-15T17:59:30.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509455 -lt 64424509454 2021-01-15T17:59:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:30.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345933 2021-01-15T17:59:30.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:30.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T17:59:30.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345933 2021-01-15T17:59:30.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:30.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345933 2021-01-15T17:59:30.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345933' 2021-01-15T17:59:30.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T17:59:30.569 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345933 2021-01-15T17:59:30.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345933 -lt 85899345933 2021-01-15T17:59:30.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:30.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182411 2021-01-15T17:59:30.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:30.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T17:59:30.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182411 2021-01-15T17:59:30.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:30.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182411 2021-01-15T17:59:30.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182411' 2021-01-15T17:59:30.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T17:59:30.964 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182411 2021-01-15T17:59:31.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182412 -lt 107374182411 2021-01-15T17:59:31.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T17:59:31.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018890 2021-01-15T17:59:31.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T17:59:31.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T17:59:31.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018890 2021-01-15T17:59:31.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T17:59:31.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018890 2021-01-15T17:59:31.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2021-01-15T17:59:31.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T17:59:31.390 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018890 2021-01-15T17:59:31.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018890 -lt 128849018890 2021-01-15T17:59:31.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T17:59:31.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:31.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:32.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T17:59:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:32.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:32.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:32.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:32.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:32.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:32.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:32.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:32.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:32.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:32.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:33.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:33.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T17:59:33.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T17:59:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T17:59:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:33.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:33.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:33.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:33.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:33.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:33.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:33.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:33.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:34.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:34.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:34.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:34.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:34.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:34.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:34.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:34.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:34.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2021-01-15T17:59:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2021-01-15T17:59:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:34.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:34.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:34.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:34.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:34.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:34.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:34.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:34.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:34.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:34.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:35.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:35.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:35.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:35.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:35.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:35.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:35.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:35.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:35.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:35.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:35.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:35.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:35.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:36.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:36.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:36.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:36.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:36.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:36.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:36.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:36.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:36.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:36.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:36.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:36.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:36.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:36.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:36.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:36.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:36.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:37.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:37.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:37.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:37.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:37.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T17:59:37.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T17:59:37.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:37.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:37.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:37.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:37.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:37.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:37.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:37.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:37.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:37.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:37.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:38.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:38.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:38.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:38.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:38.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:38.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:38.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:38.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:38.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:38.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:38.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:38.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:39.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:59:39.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:59:39.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:39.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:39.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:39.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:39.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:39.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:39.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:39.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:39.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:40.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:40.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:40.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:40.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:40.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:40.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:40.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:40.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:40.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:40.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T17:59:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T17:59:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:40.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:41.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:41.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:41.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:41.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:41.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:41.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:41.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:41.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:41.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:41.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:41.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:41.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:41.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:41.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:41.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:41.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:41.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:41.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:41.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:41.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:41.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T17:59:42.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T17:59:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:42.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:42.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:42.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:42.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:42.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:42.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:42.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:42.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:42.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:42.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:42.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:42.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:42.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:43.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:43.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:43.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:43.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:43.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:43.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:43.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:43.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:43.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:44.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:44.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:44.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:44.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:44.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:44.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:44.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:44.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:44.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:44.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:44.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:44.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:44.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:45.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:45.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:45.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:45.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:45.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:45.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:45.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:45.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:45.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:59:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:59:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:45.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:45.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:45.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:45.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:45.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:45.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:45.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:45.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:46.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:46.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:46.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:46.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:46.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:46.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:46.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:46.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:46.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:46.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:47.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:59:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:59:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:47.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:47.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:47.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:47.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:47.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:47.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:47.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:47.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:47.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:47.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:47.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:47.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:48.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:48.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:48.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:48.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:48.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:48.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:48.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:48.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:48.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:48.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:48.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:48.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:49.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:49.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:49.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:49.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:49.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:49.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:49.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:49.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:49.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:49.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:50.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:50.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:50.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:50.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:50.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:50.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:50.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:50.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:50.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:50.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:50.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:50.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:50.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:50.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:50.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:50.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:50.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:51.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:51.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:51.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:51.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:51.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:51.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:51.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:59:51.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:59:51.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:51.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:51.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:51.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:51.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:51.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:51.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:51.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:51.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:51.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:51.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:52.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:52.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:52.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:52.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:52.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:52.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:52.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:52.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:52.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:52.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:52.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:52.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:52.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:53.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:53.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:53.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:53.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:53.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:53.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:53.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:53.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:53.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:53.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:53.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:53.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:53.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:53.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:53.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:53.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:53.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:54.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:54.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:54.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:54.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:54.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:54.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:54.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:54.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:54.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:54.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:54.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:54.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:54.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:54.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:54.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:54.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:54.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:54.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:55.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:55.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:55.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:55.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:55.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:55.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:55.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:55.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:55.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:55.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:55.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:55.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:55.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:56.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T17:59:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T17:59:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:56.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:56.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:56.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:56.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:56.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:56.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:56.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:56.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:56.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:56.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:56.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:56.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:56.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:57.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:57.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:57.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:57.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:57.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:57.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:57.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:57.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:57.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:57.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:57.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:57.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:58.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:58.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:58.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:58.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:58.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:58.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:58.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:58.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:58.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:58.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:58.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:58.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T17:59:58.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T17:59:58.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T17:59:58.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T17:59:58.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T17:59:58.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T17:59:58.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T17:59:58.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T17:59:58.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T17:59:58.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T17:59:59.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T17:59:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T17:59:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T17:59:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T17:59:59.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T17:59:59.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T17:59:59.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T17:59:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T17:59:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T17:59:59.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T17:59:59.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T17:59:59.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T17:59:59.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T17:59:59.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T17:59:59.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T17:59:59.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T17:59:59.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:00.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:00.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:00.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:00.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:00.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:00.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:00:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:00:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:00.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:01.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:01.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:01.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:01.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:01.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:01.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:01.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:01.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:02.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:02.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:02.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:02.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:02.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:02.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:00:02.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:00:02.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:02.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:02.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:02.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:02.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:02.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:02.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:02.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:02.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:02.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:02.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:03.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:03.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:03.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:03.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:03.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:03.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:03.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:03.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:03.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:03.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:03.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:03.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:03.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:04.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:00:04.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:00:04.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:04.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:04.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:04.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:04.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:04.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:04.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:04.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:04.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:04.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:04.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:04.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:04.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:04.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:04.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:05.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:05.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:05.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:00:05.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:00:05.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:05.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:05.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:05.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:05.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:05.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:05.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:05.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:05.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:05.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:05.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:06.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:06.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:06.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:06.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:06.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:06.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:06.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:06.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:06.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:06.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:06.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:00:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:00:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:07.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:07.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:07.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:07.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:07.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:07.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:07.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:07.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:07.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:07.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:07.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:07.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:07.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:08.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:08.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:08.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:08.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:08.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:08.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:08.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:08.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:08.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:08.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:00:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:00:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:08.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:08.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:08.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:08.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:08.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:08.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:08.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:08.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:08.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:09.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:09.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:09.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:09.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:09.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:09.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:09.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:09.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:09.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:09.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:09.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:09.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:09.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:10.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:00:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:00:10.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:10.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:10.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:10.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:10.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:10.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:10.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:10.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:10.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:00:10.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:10.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:10.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:11.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:00:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:00:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:00:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:00:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:00:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:00:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:00:11.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:00:11.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:00:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:00:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:00:11.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:00:11.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:00:11.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:00:11.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:00:11.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:00:11.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:00:11.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:00:11.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:00:11.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:00:11.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:00:11.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:00:12.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:00:12.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:00:12.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:00:12.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:00:12.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:00:12.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:00:12.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:00:12.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:162: TEST_backfill_sizeup: get_primary test obj1 2021-01-15T18:00:12.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:00:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:00:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:00:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:00:13.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:162: TEST_backfill_sizeup: local primary=1 2021-01-15T18:00:13.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:163: TEST_backfill_sizeup: get_pg test obj1 2021-01-15T18:00:13.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:00:13.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:00:13.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:00:13.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:00:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:163: TEST_backfill_sizeup: local PG=1.0 2021-01-15T18:00:13.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:165: TEST_backfill_sizeup: expr 200 '*' 2 2021-01-15T18:00:13.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:165: TEST_backfill_sizeup: local degraded=400 2021-01-15T18:00:13.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:166: TEST_backfill_sizeup: check td/osd-backfill-stats 1.0 1 replicated 400 0 0 0 2021-01-15T18:00:13.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T18:00:13.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T18:00:13.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T18:00:13.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T18:00:13.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2021-01-15T18:00:13.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T18:00:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=0 2021-01-15T18:00:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T18:00:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T18:00:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T18:00:13.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T18:00:13.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T18:00:13.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T18:00:13.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T18:00:13.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T18:00:13.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T18:00:13.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T18:00:13.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:00:13.569 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T18:00:13.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:00:13.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[1,0,2]/[1]' 2021-01-15T18:00:13.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[1,0,2]/[1]' ']' 2021-01-15T18:00:13.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T18:00:13.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:00:13.580 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.580 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [1,0,2]/[1] ' 2021-01-15T18:00:13.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T18:00:13.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T18:00:13.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:00:13.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2021-01-15T18:00:13.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2021-01-15T18:00:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2021-01-15T18:00:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T18:00:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:00:13.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T18:00:13.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:00:13.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:00:13.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T18:00:13.587 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:00:13.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2021-01-15T18:00:13.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2021-01-15T18:00:13.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2021-01-15T18:00:13.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2021-01-15T18:00:13.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:00:13.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T18:00:13.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:00:13.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T18:00:13.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:00:13.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [1,0,2]/[1] ' 2021-01-15T18:00:13.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T18:00:13.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T18:00:13.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:00:13.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=0 2021-01-15T18:00:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 0 ']' 2021-01-15T18:00:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 0 0 2021-01-15T18:00:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=0 2021-01-15T18:00:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:00:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=0 2021-01-15T18:00:13.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:00:13.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:00:13.600 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:00:13.600 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T18:00:13.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:00:13.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=0 2021-01-15T18:00:13.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 0 ']' 2021-01-15T18:00:13.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 0 0 2021-01-15T18:00:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=0 2021-01-15T18:00:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:00:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T18:00:13.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:00:13.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T18:00:13.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:168: TEST_backfill_sizeup: delete_pool test 2021-01-15T18:00:13.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:00:13.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:00:14.121 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:00:14.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:169: TEST_backfill_sizeup: kill_daemons td/osd-backfill-stats 2021-01-15T18:00:14.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:00:14.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:00:14.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:00:14.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:00:14.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:00:14.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:00:14.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T18:00:14.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:00:14.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:00:14.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:00:14.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:00:14.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:00:14.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:00:14.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:00:14.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:00:14.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:00:14.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:00:14.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:00:14.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:00:14.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:00:14.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:00:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:00:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:00:14.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:00:14.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:00:14.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:00:14.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:00:14.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:00:14.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:00:14.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:00:14.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:00:14.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:00:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T18:00:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T18:00:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T18:00:14.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:00:14.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:00:14.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:00:14.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:00:14.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:00:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:00:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:00:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:00:14.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:00:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:00:14.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:00:14.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:00:14.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:00:14.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:00:14.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:00:14.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:00:14.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:00:14.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:00:14.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:00:14.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:00:14.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:00:14.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:00:14.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:00:14.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:00:14.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:00:14.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T18:00:14.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:00:14.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T18:00:14.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:00:14.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:00:14.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:00:14.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T18:00:14.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup4_allout td/osd-backfill-stats 2021-01-15T18:00:14.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:389: TEST_backfill_sizeup4_allout: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:391: TEST_backfill_sizeup4_allout: run_mon td/osd-backfill-stats a 2021-01-15T18:00:14.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:00:14.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:00:14.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:00:14.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T18:00:14.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T18:00:14.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:00:14.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:14.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:14.572 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:14.572 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.573 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:14.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:00:14.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:00:14.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.760 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T18:00:14.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:00:14.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T18:00:14.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:00:14.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:00:14.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:00:14.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:00:14.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:00:14.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:00:14.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:00:14.862 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:00:14.862 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:00:14.863 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:14.863 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:14.863 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T18:00:14.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:00:14.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T18:00:14.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:392: TEST_backfill_sizeup4_allout: run_mgr td/osd-backfill-stats x 2021-01-15T18:00:14.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T18:00:14.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:00:14.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:00:14.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:00:14.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T18:00:14.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:00:15.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:00:15.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:15.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:15.501 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:15.501 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:15.501 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:15.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:15.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:00:15.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:00:15.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:393: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 0 2021-01-15T18:00:15.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:00:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:00:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:00:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:00:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T18:00:15.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:00:15.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:00:15.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:00:15.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:00:15.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T18:00:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T18:00:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:00:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:00:15.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:00:15.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:00:15.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:15.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:15.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:15.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:15.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:00:15.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:00:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:00:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:00:15.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:00:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:00:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:00:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:15.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:00:15.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:00:15.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T18:00:15.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:00:15.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4795b52b-d291-459f-b4f7-cd8493758d11 2021-01-15T18:00:15.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 4795b52b-d291-459f-b4f7-cd8493758d11' 2021-01-15T18:00:15.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:00:15.537 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 4795b52b-d291-459f-b4f7-cd8493758d11 2021-01-15T18:00:15.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAv2AFgSEBRIBAA0FdwNq8m3Lu0n+h4IFO6rw== 2021-01-15T18:00:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAv2AFgSEBRIBAA0FdwNq8m3Lu0n+h4IFO6rw=="}' 2021-01-15T18:00:15.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4795b52b-d291-459f-b4f7-cd8493758d11 -i td/osd-backfill-stats/0/new.json 2021-01-15T18:00:15.907 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:15.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T18:00:15.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAv2AFgSEBRIBAA0FdwNq8m3Lu0n+h4IFO6rw== --osd-uuid 4795b52b-d291-459f-b4f7-cd8493758d11 2021-01-15T18:00:15.942 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.941+0000 7fd3699e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:15.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.943+0000 7fd3699e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:15.946 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.946+0000 7fd3699e8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:15.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.946+0000 7fd3699e8080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:00:15.948 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.946+0000 7fd3699e8080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:00:15.948 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.946+0000 7fd3699e8080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:00:15.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:15.988+0000 7fd3699e8080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T18:00:20.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T18:00:20.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:00:20.482 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:00:20.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:00:20.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:00:21.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:00:21.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:21.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:00:21.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:00:21.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:00:21.008 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:00:21.030 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:21.029+0000 7f8841794080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:21.054 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:21.053+0000 7f8841794080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:21.055 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:21.054+0000 7f8841794080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:21.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:00:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:00:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:00:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:00:21.408 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:21.572 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:21.572+0000 7f8841794080 -1 Falling back to public interface 2021-01-15T18:00:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:00:22.364 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:22.363+0000 7f8841794080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:00:22.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:00:22.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:22.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:00:22.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:22.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:00:22.805 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:23.231 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/648083,v1:127.0.0.1:6803/648083] [v2:127.0.0.1:6804/648083,v1:127.0.0.1:6805/648083] exists,up 4795b52b-d291-459f-b4f7-cd8493758d11 2021-01-15T18:00:23.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:00:23.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:00:23.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:00:23.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:394: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 1 2021-01-15T18:00:23.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:00:23.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:00:23.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:00:23.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:00:23.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T18:00:23.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:00:23.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:00:23.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:00:23.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:00:23.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T18:00:23.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T18:00:23.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:00:23.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:00:23.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:00:23.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:00:23.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:23.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:23.237 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:23.237 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:23.237 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:23.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:23.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:23.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:00:23.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:00:23.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:00:23.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:00:23.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:00:23.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:00:23.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:00:23.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:23.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:00:23.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:00:23.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T18:00:23.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:00:23.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2a727e56-67ee-4b58-b38d-19ea32305c45 2021-01-15T18:00:23.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 2a727e56-67ee-4b58-b38d-19ea32305c45' 2021-01-15T18:00:23.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:00:23.242 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 2a727e56-67ee-4b58-b38d-19ea32305c45 2021-01-15T18:00:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA32AFg3Qi5DhAAbbH50upqATwHNC+/Id2sFA== 2021-01-15T18:00:23.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA32AFg3Qi5DhAAbbH50upqATwHNC+/Id2sFA=="}' 2021-01-15T18:00:23.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2a727e56-67ee-4b58-b38d-19ea32305c45 -i td/osd-backfill-stats/1/new.json 2021-01-15T18:00:23.761 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:23.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T18:00:23.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA32AFg3Qi5DhAAbbH50upqATwHNC+/Id2sFA== --osd-uuid 2a727e56-67ee-4b58-b38d-19ea32305c45 2021-01-15T18:00:23.798 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.797+0000 7f5d716bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:23.799 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.798+0000 7f5d716bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:23.801 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.800+0000 7f5d716bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:23.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.800+0000 7f5d716bc080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:00:23.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.800+0000 7f5d716bc080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:00:23.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.800+0000 7f5d716bc080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:00:23.853 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:23.852+0000 7f5d716bc080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T18:00:28.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T18:00:28.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:00:28.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:00:28.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:00:28.439 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:00:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:00:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:28.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:00:28.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:00:28.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:00:28.969 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:00:28.991 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:28.990+0000 7f6f7627e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:29.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:29.008+0000 7f6f7627e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:29.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:29.009+0000 7f6f7627e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:29.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:00:29.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:00:29.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:00:29.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:00:29.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:00:29.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:29.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:00:29.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:29.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:00:29.363 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:29.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:29.526+0000 7f6f7627e080 -1 Falling back to public interface 2021-01-15T18:00:29.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:00:30.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:30.310+0000 7f6f7627e080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:00:30.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:00:30.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:30.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:00:30.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:30.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:00:30.774 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:31.178 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/649347,v1:127.0.0.1:6811/649347] [v2:127.0.0.1:6812/649347,v1:127.0.0.1:6813/649347] exists,up 2a727e56-67ee-4b58-b38d-19ea32305c45 2021-01-15T18:00:31.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:00:31.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:00:31.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:00:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:395: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 2 2021-01-15T18:00:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:00:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:00:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:00:31.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:00:31.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T18:00:31.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:00:31.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:00:31.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:00:31.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:00:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T18:00:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T18:00:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:00:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:00:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:00:31.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:00:31.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:31.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:31.183 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:31.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:31.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:31.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:31.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:31.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:00:31.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:00:31.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:00:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:00:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:00:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:00:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:00:31.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:31.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:00:31.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:00:31.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T18:00:31.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:00:31.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9a3baca1-66bf-44b7-a58e-4008677f1986 2021-01-15T18:00:31.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 9a3baca1-66bf-44b7-a58e-4008677f1986' 2021-01-15T18:00:31.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:00:31.194 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 9a3baca1-66bf-44b7-a58e-4008677f1986 2021-01-15T18:00:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA/2AFg49+SCxAADGogZ1QeWbZoV+mJzxpN8g== 2021-01-15T18:00:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA/2AFg49+SCxAADGogZ1QeWbZoV+mJzxpN8g=="}' 2021-01-15T18:00:31.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9a3baca1-66bf-44b7-a58e-4008677f1986 -i td/osd-backfill-stats/2/new.json 2021-01-15T18:00:31.802 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:00:31.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T18:00:31.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA/2AFg49+SCxAADGogZ1QeWbZoV+mJzxpN8g== --osd-uuid 9a3baca1-66bf-44b7-a58e-4008677f1986 2021-01-15T18:00:31.839 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.838+0000 7f9f589a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:31.841 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.840+0000 7f9f589a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:31.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.841+0000 7f9f589a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:31.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.841+0000 7f9f589a6080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:00:31.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.841+0000 7f9f589a6080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:00:31.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.841+0000 7f9f589a6080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:00:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:31.899+0000 7f9f589a6080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T18:00:36.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T18:00:36.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:00:36.402 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:00:36.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:00:36.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:00:36.928 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:00:36.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:00:36.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:36.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:00:36.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:00:36.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:00:36.950 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:36.950+0000 7f3a41fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:36.974 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:36.973+0000 7f3a41fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:36.975 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:36.974+0000 7f3a41fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:00:37.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:00:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:00:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:00:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:00:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:37.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:00:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:37.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:00:37.322 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:37.495 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:37.495+0000 7f3a41fd4080 -1 Falling back to public interface 2021-01-15T18:00:37.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:00:38.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:38.291+0000 7f3a41fd4080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:00:38.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:00:38.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:38.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:00:38.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:38.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:00:38.764 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:39.178 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/650619,v1:127.0.0.1:6819/650619] [v2:127.0.0.1:6820/650619,v1:127.0.0.1:6821/650619] exists,up 9a3baca1-66bf-44b7-a58e-4008677f1986 2021-01-15T18:00:39.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:00:39.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:00:39.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:00:39.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:396: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 3 2021-01-15T18:00:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:00:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:00:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:00:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:00:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T18:00:39.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:00:39.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:00:39.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:00:39.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:00:39.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T18:00:39.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T18:00:39.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:00:39.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:00:39.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:00:39.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:00:39.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:39.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:39.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:39.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:39.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:00:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:00:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:00:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:00:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:00:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:00:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:00:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:00:39.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:00:39.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T18:00:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:00:39.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=43e108b4-9e96-4be5-b294-971579690fc6 2021-01-15T18:00:39.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 43e108b4-9e96-4be5-b294-971579690fc6' 2021-01-15T18:00:39.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:00:39.188 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 43e108b4-9e96-4be5-b294-971579690fc6 2021-01-15T18:00:39.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBH2AFgcZCSCxAA0/Z3GsktkG7bhBEGuZLmjg== 2021-01-15T18:00:39.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBH2AFgcZCSCxAA0/Z3GsktkG7bhBEGuZLmjg=="}' 2021-01-15T18:00:39.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 43e108b4-9e96-4be5-b294-971579690fc6 -i td/osd-backfill-stats/3/new.json 2021-01-15T18:00:39.734 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:00:39.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T18:00:39.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBH2AFgcZCSCxAA0/Z3GsktkG7bhBEGuZLmjg== --osd-uuid 43e108b4-9e96-4be5-b294-971579690fc6 2021-01-15T18:00:39.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.769+0000 7f10b4a20080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:39.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.771+0000 7f10b4a20080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:39.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.772+0000 7f10b4a20080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.773+0000 7f10b4a20080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:00:39.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.773+0000 7f10b4a20080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:00:39.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.773+0000 7f10b4a20080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:00:39.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:39.815+0000 7f10b4a20080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T18:00:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T18:00:44.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:00:44.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:00:44.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:00:44.578 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:00:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:00:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:00:45.107 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:00:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:45.127+0000 7f6ee0b2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:45.148 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:45.148+0000 7f6ee0b2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:45.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:45.149+0000 7f6ee0b2b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:45.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:00:45.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:00:45.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:00:45.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:00:45.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:00:45.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:45.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:00:45.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:00:45.513 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:45.664 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:45.664+0000 7f6ee0b2b080 -1 Falling back to public interface 2021-01-15T18:00:45.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:00:46.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:46.484+0000 7f6ee0b2b080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:00:46.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:00:46.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:46.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:00:46.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:46.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:00:46.910 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:47.317 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/651883,v1:127.0.0.1:6827/651883] [v2:127.0.0.1:6828/651883,v1:127.0.0.1:6829/651883] exists,up 43e108b4-9e96-4be5-b294-971579690fc6 2021-01-15T18:00:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:00:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:00:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:00:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:397: TEST_backfill_sizeup4_allout: run_osd td/osd-backfill-stats 4 2021-01-15T18:00:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:00:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:00:47.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:00:47.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:00:47.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T18:00:47.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:00:47.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:00:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:00:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:00:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T18:00:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T18:00:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:00:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:00:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:00:47.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:00:47.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:00:47.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:00:47.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:00:47.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:00:47.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:00:47.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:47.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:00:47.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:00:47.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:00:47.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:00:47.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:00:47.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:00:47.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:00:47.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:00:47.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:47.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:00:47.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:00:47.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T18:00:47.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:00:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=210d62bc-f606-40fd-8c8d-d5319e135a77 2021-01-15T18:00:47.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 210d62bc-f606-40fd-8c8d-d5319e135a77' 2021-01-15T18:00:47.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:00:47.327 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 210d62bc-f606-40fd-8c8d-d5319e135a77 2021-01-15T18:00:47.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBP2AFgVpnZExAAiW7jjCmEjFoadwmEzAUpCA== 2021-01-15T18:00:47.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBP2AFgVpnZExAAiW7jjCmEjFoadwmEzAUpCA=="}' 2021-01-15T18:00:47.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 210d62bc-f606-40fd-8c8d-d5319e135a77 -i td/osd-backfill-stats/4/new.json 2021-01-15T18:00:47.893 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:00:47.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T18:00:47.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBP2AFgVpnZExAAiW7jjCmEjFoadwmEzAUpCA== --osd-uuid 210d62bc-f606-40fd-8c8d-d5319e135a77 2021-01-15T18:00:47.930 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.929+0000 7f4d933b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:47.931 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.931+0000 7f4d933b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:47.933 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.933+0000 7f4d933b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:47.934 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.933+0000 7f4d933b1080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:00:47.934 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.933+0000 7f4d933b1080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:00:47.934 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.933+0000 7f4d933b1080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:00:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:47.981+0000 7f4d933b1080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T18:00:52.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T18:00:52.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:00:52.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:00:52.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:00:52.726 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:00:53.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:00:53.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:00:53.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:00:53.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:00:53.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:00:53.255 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:00:53.277 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:53.276+0000 7f9dab235080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:53.290 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:53.289+0000 7f9dab235080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:53.291 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:53.290+0000 7f9dab235080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:00:53.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:00:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:00:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:00:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:00:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:00:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:00:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:00:53.674 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:00:53.809 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:53.809+0000 7f9dab235080 -1 Falling back to public interface 2021-01-15T18:00:54.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:00:54.600 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:00:54.599+0000 7f9dab235080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:00:55.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:00:55.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:00:55.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:00:55.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:00:55.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:00:55.066 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:00:55.465 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/653147,v1:127.0.0.1:6835/653147] [v2:127.0.0.1:6836/653147,v1:127.0.0.1:6837/653147] exists,up 210d62bc-f606-40fd-8c8d-d5319e135a77 2021-01-15T18:00:55.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:00:55.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:00:55.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:00:55.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:399: TEST_backfill_sizeup4_allout: create_pool test 1 1 2021-01-15T18:00:55.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:00:56.055 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:00:56.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:00:57.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:400: TEST_backfill_sizeup4_allout: ceph osd pool set test size 2 2021-01-15T18:00:57.942 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:00:57.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:402: TEST_backfill_sizeup4_allout: wait_for_clean 2021-01-15T18:00:57.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:00:57.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:00:57.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:00:57.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:00:57.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:00:57.955 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:00:57.955 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:00:57.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:00:57.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:00:57.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:00:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:00:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:00:58.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:00:58.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:00:58.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:00:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:00:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:00:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:00:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:00:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:00:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:00:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:00:58.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:00:58.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836489 2021-01-15T18:00:58.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836489 2021-01-15T18:00:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489' 2021-01-15T18:00:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:00:58.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:00:58.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2021-01-15T18:00:58.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2021-01-15T18:00:58.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2021-01-15T18:00:58.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:00:58.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:00:58.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T18:00:58.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T18:00:58.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509446' 2021-01-15T18:00:58.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:00:58.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:00:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345925 2021-01-15T18:00:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345925 2021-01-15T18:00:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509446 3-85899345925' 2021-01-15T18:00:58.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:00:58.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:00:59.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182403 2021-01-15T18:00:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182403 2021-01-15T18:00:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-64424509446 3-85899345925 4-107374182403' 2021-01-15T18:00:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:00:59.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2021-01-15T18:00:59.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:00:59.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:00:59.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836489 2021-01-15T18:00:59.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:00:59.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836489 2021-01-15T18:00:59.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2021-01-15T18:00:59.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:00:59.130 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836489 2021-01-15T18:00:59.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836489 2021-01-15T18:00:59.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:00:59.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2021-01-15T18:00:59.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:00:59.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:00:59.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-01-15T18:00:59.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:00:59.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2021-01-15T18:00:59.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-01-15T18:00:59.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:00:59.532 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672969 2021-01-15T18:00:59.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672969 2021-01-15T18:00:59.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:00:59.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T18:00:59.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:00:59.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:00:59.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T18:00:59.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:00:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T18:00:59.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T18:00:59.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:00:59.948 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T18:01:00.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509446 2021-01-15T18:01:00.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:00.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345925 2021-01-15T18:01:00.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:01:00.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345925 2021-01-15T18:01:00.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345925 2021-01-15T18:01:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345925' 2021-01-15T18:01:00.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:01:00.345 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345925 2021-01-15T18:01:00.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345925 -lt 85899345925 2021-01-15T18:01:00.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:00.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182403 2021-01-15T18:01:00.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:00.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:01:00.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182403 2021-01-15T18:01:00.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182403 2021-01-15T18:01:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182403' 2021-01-15T18:01:00.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:01:00.742 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182403 2021-01-15T18:01:01.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182403 -lt 107374182403 2021-01-15T18:01:01.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:01:01.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:01.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:01.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:01:01.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:01.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:01.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:01.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:01.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:01.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:01.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:02.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:02.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:02.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:02.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:02.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:02.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:01:02.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:01:02.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:01:02.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:02.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:02.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:02.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:02.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:02.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:02.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:02.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:02.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:02.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:03.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:03.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:03.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:03.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:03.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:03.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:03.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:03.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:03.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:03.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:03.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:03.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:03.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T18:01:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T18:01:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2021-01-15T18:01:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:04.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T18:01:04.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:04.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:04.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:04.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:04.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:04.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:04.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:04.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:04.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:01:04.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:04.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:04.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:05.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:01:05.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:01:05.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:01:05.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: seq 1 200 2021-01-15T18:01:05.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj1 /dev/null 2021-01-15T18:01:05.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj2 /dev/null 2021-01-15T18:01:05.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj3 /dev/null 2021-01-15T18:01:05.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj4 /dev/null 2021-01-15T18:01:05.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj5 /dev/null 2021-01-15T18:01:05.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj6 /dev/null 2021-01-15T18:01:05.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj7 /dev/null 2021-01-15T18:01:05.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj8 /dev/null 2021-01-15T18:01:05.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:05.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj9 /dev/null 2021-01-15T18:01:06.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj10 /dev/null 2021-01-15T18:01:06.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj11 /dev/null 2021-01-15T18:01:06.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj12 /dev/null 2021-01-15T18:01:06.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj13 /dev/null 2021-01-15T18:01:06.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj14 /dev/null 2021-01-15T18:01:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj15 /dev/null 2021-01-15T18:01:06.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj16 /dev/null 2021-01-15T18:01:06.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj17 /dev/null 2021-01-15T18:01:06.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj18 /dev/null 2021-01-15T18:01:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj19 /dev/null 2021-01-15T18:01:07.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj20 /dev/null 2021-01-15T18:01:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj21 /dev/null 2021-01-15T18:01:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj22 /dev/null 2021-01-15T18:01:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj23 /dev/null 2021-01-15T18:01:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj24 /dev/null 2021-01-15T18:01:07.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj25 /dev/null 2021-01-15T18:01:07.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj26 /dev/null 2021-01-15T18:01:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj27 /dev/null 2021-01-15T18:01:07.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:07.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj28 /dev/null 2021-01-15T18:01:08.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj29 /dev/null 2021-01-15T18:01:08.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj30 /dev/null 2021-01-15T18:01:08.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj31 /dev/null 2021-01-15T18:01:08.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj32 /dev/null 2021-01-15T18:01:08.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj33 /dev/null 2021-01-15T18:01:08.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj34 /dev/null 2021-01-15T18:01:08.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj35 /dev/null 2021-01-15T18:01:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj36 /dev/null 2021-01-15T18:01:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj37 /dev/null 2021-01-15T18:01:08.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:08.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj38 /dev/null 2021-01-15T18:01:09.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj39 /dev/null 2021-01-15T18:01:09.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj40 /dev/null 2021-01-15T18:01:09.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj41 /dev/null 2021-01-15T18:01:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj42 /dev/null 2021-01-15T18:01:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj43 /dev/null 2021-01-15T18:01:09.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj44 /dev/null 2021-01-15T18:01:09.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj45 /dev/null 2021-01-15T18:01:09.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj46 /dev/null 2021-01-15T18:01:09.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj47 /dev/null 2021-01-15T18:01:09.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj48 /dev/null 2021-01-15T18:01:09.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:09.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj49 /dev/null 2021-01-15T18:01:10.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj50 /dev/null 2021-01-15T18:01:10.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj51 /dev/null 2021-01-15T18:01:10.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj52 /dev/null 2021-01-15T18:01:10.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj53 /dev/null 2021-01-15T18:01:10.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj54 /dev/null 2021-01-15T18:01:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj55 /dev/null 2021-01-15T18:01:10.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj56 /dev/null 2021-01-15T18:01:10.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj57 /dev/null 2021-01-15T18:01:10.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj58 /dev/null 2021-01-15T18:01:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:10.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj59 /dev/null 2021-01-15T18:01:11.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj60 /dev/null 2021-01-15T18:01:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj61 /dev/null 2021-01-15T18:01:11.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj62 /dev/null 2021-01-15T18:01:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj63 /dev/null 2021-01-15T18:01:11.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj64 /dev/null 2021-01-15T18:01:11.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj65 /dev/null 2021-01-15T18:01:11.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj66 /dev/null 2021-01-15T18:01:11.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj67 /dev/null 2021-01-15T18:01:11.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj68 /dev/null 2021-01-15T18:01:11.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:11.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj69 /dev/null 2021-01-15T18:01:12.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj70 /dev/null 2021-01-15T18:01:12.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj71 /dev/null 2021-01-15T18:01:12.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj72 /dev/null 2021-01-15T18:01:12.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj73 /dev/null 2021-01-15T18:01:12.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj74 /dev/null 2021-01-15T18:01:12.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj75 /dev/null 2021-01-15T18:01:12.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj76 /dev/null 2021-01-15T18:01:12.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj77 /dev/null 2021-01-15T18:01:12.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj78 /dev/null 2021-01-15T18:01:12.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:12.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj79 /dev/null 2021-01-15T18:01:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj80 /dev/null 2021-01-15T18:01:13.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj81 /dev/null 2021-01-15T18:01:13.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj82 /dev/null 2021-01-15T18:01:13.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj83 /dev/null 2021-01-15T18:01:13.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj84 /dev/null 2021-01-15T18:01:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj85 /dev/null 2021-01-15T18:01:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj86 /dev/null 2021-01-15T18:01:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj87 /dev/null 2021-01-15T18:01:13.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj88 /dev/null 2021-01-15T18:01:13.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:13.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj89 /dev/null 2021-01-15T18:01:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj90 /dev/null 2021-01-15T18:01:14.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj91 /dev/null 2021-01-15T18:01:14.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj92 /dev/null 2021-01-15T18:01:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj93 /dev/null 2021-01-15T18:01:14.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj94 /dev/null 2021-01-15T18:01:14.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj95 /dev/null 2021-01-15T18:01:14.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj96 /dev/null 2021-01-15T18:01:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj97 /dev/null 2021-01-15T18:01:14.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj98 /dev/null 2021-01-15T18:01:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:14.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj99 /dev/null 2021-01-15T18:01:15.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj100 /dev/null 2021-01-15T18:01:15.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj101 /dev/null 2021-01-15T18:01:15.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj102 /dev/null 2021-01-15T18:01:15.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj103 /dev/null 2021-01-15T18:01:15.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj104 /dev/null 2021-01-15T18:01:15.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj105 /dev/null 2021-01-15T18:01:15.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj106 /dev/null 2021-01-15T18:01:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj107 /dev/null 2021-01-15T18:01:15.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:15.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj108 /dev/null 2021-01-15T18:01:16.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj109 /dev/null 2021-01-15T18:01:16.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj110 /dev/null 2021-01-15T18:01:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj111 /dev/null 2021-01-15T18:01:16.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj112 /dev/null 2021-01-15T18:01:16.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj113 /dev/null 2021-01-15T18:01:16.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj114 /dev/null 2021-01-15T18:01:16.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj115 /dev/null 2021-01-15T18:01:16.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj116 /dev/null 2021-01-15T18:01:16.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj117 /dev/null 2021-01-15T18:01:16.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:16.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj118 /dev/null 2021-01-15T18:01:17.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj119 /dev/null 2021-01-15T18:01:17.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj120 /dev/null 2021-01-15T18:01:17.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj121 /dev/null 2021-01-15T18:01:17.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj122 /dev/null 2021-01-15T18:01:17.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj123 /dev/null 2021-01-15T18:01:17.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj124 /dev/null 2021-01-15T18:01:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj125 /dev/null 2021-01-15T18:01:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj126 /dev/null 2021-01-15T18:01:18.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj127 /dev/null 2021-01-15T18:01:18.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj128 /dev/null 2021-01-15T18:01:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj129 /dev/null 2021-01-15T18:01:18.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj130 /dev/null 2021-01-15T18:01:18.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj131 /dev/null 2021-01-15T18:01:18.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj132 /dev/null 2021-01-15T18:01:18.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj133 /dev/null 2021-01-15T18:01:18.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj134 /dev/null 2021-01-15T18:01:18.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj135 /dev/null 2021-01-15T18:01:18.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:18.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj136 /dev/null 2021-01-15T18:01:19.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj137 /dev/null 2021-01-15T18:01:19.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj138 /dev/null 2021-01-15T18:01:19.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj139 /dev/null 2021-01-15T18:01:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj140 /dev/null 2021-01-15T18:01:19.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj141 /dev/null 2021-01-15T18:01:19.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj142 /dev/null 2021-01-15T18:01:19.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj143 /dev/null 2021-01-15T18:01:19.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj144 /dev/null 2021-01-15T18:01:19.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:19.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj145 /dev/null 2021-01-15T18:01:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj146 /dev/null 2021-01-15T18:01:20.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj147 /dev/null 2021-01-15T18:01:20.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj148 /dev/null 2021-01-15T18:01:20.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj149 /dev/null 2021-01-15T18:01:20.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj150 /dev/null 2021-01-15T18:01:20.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj151 /dev/null 2021-01-15T18:01:20.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj152 /dev/null 2021-01-15T18:01:20.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj153 /dev/null 2021-01-15T18:01:20.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj154 /dev/null 2021-01-15T18:01:20.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:20.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj155 /dev/null 2021-01-15T18:01:21.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj156 /dev/null 2021-01-15T18:01:21.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj157 /dev/null 2021-01-15T18:01:21.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj158 /dev/null 2021-01-15T18:01:21.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj159 /dev/null 2021-01-15T18:01:21.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj160 /dev/null 2021-01-15T18:01:21.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj161 /dev/null 2021-01-15T18:01:21.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj162 /dev/null 2021-01-15T18:01:21.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj163 /dev/null 2021-01-15T18:01:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj164 /dev/null 2021-01-15T18:01:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj165 /dev/null 2021-01-15T18:01:22.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj166 /dev/null 2021-01-15T18:01:22.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj167 /dev/null 2021-01-15T18:01:22.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj168 /dev/null 2021-01-15T18:01:22.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj169 /dev/null 2021-01-15T18:01:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj170 /dev/null 2021-01-15T18:01:22.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj171 /dev/null 2021-01-15T18:01:22.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj172 /dev/null 2021-01-15T18:01:22.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj173 /dev/null 2021-01-15T18:01:22.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj174 /dev/null 2021-01-15T18:01:22.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:22.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj175 /dev/null 2021-01-15T18:01:23.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj176 /dev/null 2021-01-15T18:01:23.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj177 /dev/null 2021-01-15T18:01:23.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj178 /dev/null 2021-01-15T18:01:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj179 /dev/null 2021-01-15T18:01:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj180 /dev/null 2021-01-15T18:01:23.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj181 /dev/null 2021-01-15T18:01:23.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj182 /dev/null 2021-01-15T18:01:23.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj183 /dev/null 2021-01-15T18:01:23.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj184 /dev/null 2021-01-15T18:01:23.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:23.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj185 /dev/null 2021-01-15T18:01:24.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj186 /dev/null 2021-01-15T18:01:24.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj187 /dev/null 2021-01-15T18:01:24.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj188 /dev/null 2021-01-15T18:01:24.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj189 /dev/null 2021-01-15T18:01:24.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj190 /dev/null 2021-01-15T18:01:24.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj191 /dev/null 2021-01-15T18:01:24.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj192 /dev/null 2021-01-15T18:01:24.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj193 /dev/null 2021-01-15T18:01:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj194 /dev/null 2021-01-15T18:01:24.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:24.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj195 /dev/null 2021-01-15T18:01:25.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:25.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj196 /dev/null 2021-01-15T18:01:25.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:25.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj197 /dev/null 2021-01-15T18:01:25.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:25.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj198 /dev/null 2021-01-15T18:01:25.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj199 /dev/null 2021-01-15T18:01:25.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:404: TEST_backfill_sizeup4_allout: for i in $(seq 1 $objects) 2021-01-15T18:01:25.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:406: TEST_backfill_sizeup4_allout: rados -p test put obj200 /dev/null 2021-01-15T18:01:25.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:409: TEST_backfill_sizeup4_allout: get_pg test obj1 2021-01-15T18:01:25.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:01:25.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:01:25.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:01:25.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:01:25.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:409: TEST_backfill_sizeup4_allout: local PG=1.0 2021-01-15T18:01:25.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:411: TEST_backfill_sizeup4_allout: get_primary test obj1 2021-01-15T18:01:25.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:01:25.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:01:25.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:01:25.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:01:26.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:411: TEST_backfill_sizeup4_allout: local primary=1 2021-01-15T18:01:26.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:412: TEST_backfill_sizeup4_allout: get_not_primary test obj1 2021-01-15T18:01:26.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:01:26.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:01:26.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:01:26.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:01:26.345 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:01:26.346 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:01:26.346 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:01:26.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:01:26.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:01:26.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:01:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:412: TEST_backfill_sizeup4_allout: local otherosd=0 2021-01-15T18:01:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:414: TEST_backfill_sizeup4_allout: ceph osd set nobackfill 2021-01-15T18:01:27.977 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T18:01:27.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:415: TEST_backfill_sizeup4_allout: ceph osd out osd.0 2021-01-15T18:01:28.620 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:01:28.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:416: TEST_backfill_sizeup4_allout: ceph osd out osd.1 2021-01-15T18:01:29.215 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T18:01:29.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:417: TEST_backfill_sizeup4_allout: ceph osd pool set test size 4 2021-01-15T18:01:30.030 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 4 2021-01-15T18:01:30.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:419: TEST_backfill_sizeup4_allout: sleep 2 2021-01-15T18:01:32.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:420: TEST_backfill_sizeup4_allout: get_primary test obj1 2021-01-15T18:01:32.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:01:32.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:01:32.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:01:32.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:01:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:420: TEST_backfill_sizeup4_allout: primary=1 2021-01-15T18:01:32.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:421: TEST_backfill_sizeup4_allout: ceph osd unset nobackfill 2021-01-15T18:01:33.272 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T18:01:33.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:422: TEST_backfill_sizeup4_allout: ceph tell osd.1 get_latest_osdmap 2021-01-15T18:01:33.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:423: TEST_backfill_sizeup4_allout: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T18:01:33.563 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:01:33.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:424: TEST_backfill_sizeup4_allout: sleep 2 2021-01-15T18:01:35.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:426: TEST_backfill_sizeup4_allout: wait_for_clean 2021-01-15T18:01:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:01:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:01:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:01:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:01:35.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:01:35.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:01:35.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:01:35.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:01:35.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:01:35.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:01:35.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:01:35.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:01:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:01:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:01:35.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:01:36.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:01:36.074 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:01:36.074 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:01:36.075 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:01:36.075 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:01:36.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:01:36.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:01:36.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:01:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2021-01-15T18:01:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2021-01-15T18:01:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2021-01-15T18:01:36.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:01:36.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:01:36.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2021-01-15T18:01:36.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2021-01-15T18:01:36.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2021-01-15T18:01:36.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:01:36.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:01:36.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509456 2021-01-15T18:01:36.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509456 2021-01-15T18:01:36.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509456' 2021-01-15T18:01:36.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:01:36.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:01:36.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345934 2021-01-15T18:01:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345934 2021-01-15T18:01:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509456 3-85899345934' 2021-01-15T18:01:36.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:01:36.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:01:36.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182413 2021-01-15T18:01:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182413 2021-01-15T18:01:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509456 3-85899345934 4-107374182413' 2021-01-15T18:01:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:36.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2021-01-15T18:01:36.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:36.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:01:36.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2021-01-15T18:01:36.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:36.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2021-01-15T18:01:36.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-01-15T18:01:36.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:01:36.769 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836499 2021-01-15T18:01:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836499 2021-01-15T18:01:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:01:38.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:01:38.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:01:38.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836499 -lt 21474836499 2021-01-15T18:01:38.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:38.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2021-01-15T18:01:38.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:38.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:01:38.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2021-01-15T18:01:38.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:38.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2021-01-15T18:01:38.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-01-15T18:01:38.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:01:38.565 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672978 2021-01-15T18:01:38.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672978 2021-01-15T18:01:38.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:38.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509456 2021-01-15T18:01:38.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:38.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:01:38.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-01-15T18:01:38.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:39.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509456 2021-01-15T18:01:39.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-01-15T18:01:39.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:01:39.001 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509456 2021-01-15T18:01:39.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509456 -lt 64424509456 2021-01-15T18:01:39.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:39.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345934 2021-01-15T18:01:39.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:39.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:01:39.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345934 2021-01-15T18:01:39.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:39.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345934 2021-01-15T18:01:39.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2021-01-15T18:01:39.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:01:39.437 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345934 2021-01-15T18:01:39.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345935 -lt 85899345934 2021-01-15T18:01:39.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:01:39.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182413 2021-01-15T18:01:39.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:01:39.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:01:39.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182413 2021-01-15T18:01:39.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:01:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182413 2021-01-15T18:01:39.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2021-01-15T18:01:39.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:01:39.838 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182413 2021-01-15T18:01:40.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182413 -lt 107374182413 2021-01-15T18:01:40.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:01:40.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:40.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:40.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:01:40.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:40.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:40.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:41.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:41.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:41.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:41.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:41.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:41.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:01:41.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:01:41.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:01:41.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:41.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:41.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:41.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:41.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:41.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:41.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:41.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:41.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:41.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:42.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:42.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:42.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:42.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:42.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:42.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:42.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:42.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:42.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:42.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:43.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2021-01-15T18:01:43.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2021-01-15T18:01:43.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:43.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:43.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:43.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:43.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:43.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:43.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:43.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:43.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:43.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:43.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:43.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:43.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:43.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:43.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:44.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:44.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:44.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:44.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:44.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:44.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:44.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:44.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:44.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:44.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:44.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:44.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:44.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:44.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:44.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:44.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:44.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:44.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:44.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:44.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:44.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:45.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:45.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:45.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:45.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:45.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:45.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:45.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:45.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:46.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:01:46.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:01:46.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:46.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:46.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:46.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:46.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:46.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:46.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:46.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:46.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:46.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:46.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:46.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:46.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:46.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:46.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:47.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:47.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:47.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:47.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:47.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:47.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:47.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:47.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:47.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:01:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:01:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:47.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:47.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:47.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:47.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:47.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:47.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:47.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:47.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:48.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:48.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:48.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:48.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:49.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:49.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:49.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:49.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:49.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:49.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:49.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:49.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:49.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:49.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:49.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:49.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:49.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:49.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:49.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:49.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:49.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:49.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:49.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:49.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:49.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:49.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:50.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:50.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:50.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:50.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:50.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:50.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:50.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:50.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:50.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:50.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:50.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:50.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:50.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:51.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:51.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:51.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:51.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:51.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:51.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:51.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:51.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:51.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:51.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:51.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:51.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:51.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:51.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:51.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:51.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:51.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:52.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:52.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:52.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:52.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:52.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:52.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:52.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:52.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:52.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:52.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:52.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:52.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:52.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:52.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:53.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:53.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:53.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:53.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:53.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:53.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:53.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:53.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:53.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:53.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:53.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:53.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:53.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:01:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:01:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:54.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:54.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:54.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:54.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:54.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:54.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:54.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:54.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:54.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:54.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:54.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:54.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:54.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:54.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:55.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:55.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:55.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:55.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:55.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:55.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:55.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:55.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:55.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:55.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:55.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:55.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:55.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:55.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:55.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:55.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:55.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:55.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:55.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:56.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:56.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:56.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:56.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:56.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:56.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:56.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:56.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:56.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:56.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:56.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:56.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:56.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:57.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:01:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:01:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:57.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:57.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:57.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:57.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:57.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:57.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:57.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:57.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:57.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:57.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:57.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:57.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:57.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:58.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:58.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:58.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:58.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:58.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:58.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:58.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:58.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:58.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:01:58.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:01:58.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:01:58.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:01:58.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:01:58.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:01:58.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:01:58.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:01:58.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:01:58.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:01:58.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:01:58.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:01:58.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:01:58.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:01:59.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:01:59.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:01:59.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:01:59.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:01:59.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:01:59.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:01:59.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:01:59.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:01:59.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:01:59.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:01:59.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:01:59.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:01:59.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:00.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:00.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:00.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:00.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:00.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:00.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:00.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:00.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:00.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:00.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:00.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:00.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:00.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:00.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:00.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:01.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:01.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:01.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:01.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:01.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:01.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:01.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:01.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:01.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:02.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:02.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:02.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:02.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:02.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:02.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:02.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:02.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:02.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:02.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:02.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:02.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:02.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:02.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:02.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:02.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:02.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:03.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:03.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:03.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:03.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:03.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:03.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:02:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:02:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:03.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:03.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:03.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:03.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:03.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:03.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:03.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:03.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:04.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:04.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:04.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:04.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:04.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:04.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:04.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:04.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:04.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:04.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:04.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:04.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:05.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:02:05.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:02:05.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:05.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:05.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:05.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:05.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:05.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:05.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:05.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:05.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:05.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:05.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:05.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:05.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:05.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:05.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:06.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:06.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:06.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:06.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:06.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:06.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:06.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:06.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:06.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:06.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:06.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:06.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:06.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:06.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:06.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:06.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:06.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:06.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:06.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:06.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:06.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:06.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:07.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:07.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:07.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:07.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:07.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:07.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:07.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:07.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:07.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:08.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:02:08.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:02:08.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:08.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:08.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:08.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:08.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:08.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:08.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:09.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:09.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:09.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:09.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:09.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:09.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:09.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:09.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:09.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:09.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:09.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:09.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:09.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:09.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:09.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:09.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:09.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:09.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:09.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:09.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:10.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:10.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:10.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:10.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:10.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:10.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:10.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:10.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:10.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:10.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:11.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:11.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:11.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:11.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:11.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:11.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:11.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:11.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:11.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:11.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:11.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:11.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:11.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:11.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:11.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:11.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:12.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:12.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:12.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:12.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:12.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:12.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:12.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:12.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:12.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:12.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:12.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:12.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:12.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:12.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:12.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:12.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:12.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:12.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:12.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:12.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:12.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:13.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:13.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:13.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:13.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:13.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:13.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:14.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:02:14.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:02:14.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:14.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:14.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:14.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:14.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:14.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:14.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:14.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:14.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:14.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:14.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:14.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:14.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:14.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:14.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:15.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:15.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:15.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:15.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:15.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:15.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:15.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:15.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:15.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:15.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:15.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:15.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:15.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:15.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:15.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:15.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:15.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:15.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:16.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:16.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:16.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:16.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:16.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:16.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:16.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:16.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:16.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:16.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:16.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:16.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:16.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:17.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:17.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:17.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:17.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:17.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:17.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:17.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:17.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:17.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:17.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:17.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:17.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:17.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:18.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:18.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:18.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:18.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:18.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:18.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:18.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:18.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:18.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:18.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:02:18.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:02:18.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:18.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:18.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:19.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:19.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:19.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:19.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:19.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:19.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:19.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:19.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:19.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:19.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:19.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:19.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:02:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:02:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:20.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:20.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:20.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:20.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:20.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:20.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:20.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:20.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:20.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:21.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:21.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:21.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:21.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:21.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:21.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:21.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:02:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:02:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:22.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:22.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:22.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:22.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:22.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:22.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:22.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:22.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:22.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:22.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:23.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:23.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:23.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:23.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:23.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:23.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:23.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:23.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:23.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:02:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:02:23.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:23.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:23.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:23.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:23.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:23.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:23.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:23.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:23.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:23.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:24.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:02:24.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:24.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:24.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:24.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:02:24.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:02:24.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:02:24.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:02:24.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:02:24.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:02:24.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:02:24.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:02:24.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:02:25.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:02:25.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:02:25.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:02:25.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:02:25.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:02:25.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:02:25.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:02:25.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:02:25.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:02:25.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:02:25.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:02:25.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:02:25.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:02:25.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:02:25.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:02:25.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:02:25.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:02:26.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:02:26.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:02:26.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:02:26.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:428: TEST_backfill_sizeup4_allout: expr 200 '*' 2 2021-01-15T18:02:26.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:428: TEST_backfill_sizeup4_allout: local misdeg=400 2021-01-15T18:02:26.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:429: TEST_backfill_sizeup4_allout: check td/osd-backfill-stats 1.0 1 replicated 400 0 400 200 2021-01-15T18:02:26.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T18:02:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T18:02:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T18:02:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2021-01-15T18:02:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T18:02:26.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=400 2021-01-15T18:02:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=200 2021-01-15T18:02:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T18:02:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T18:02:26.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T18:02:26.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T18:02:26.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log 2021-01-15T18:02:26.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T18:02:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T18:02:26.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T18:02:26.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T18:02:26.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:02:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T18:02:26.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:02:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[2,4,3]/[1,0]' 2021-01-15T18:02:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[2,4,3]/[1,0]' ']' 2021-01-15T18:02:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T18:02:26.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:02:26.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [2,4,3]/[1,0] ' 2021-01-15T18:02:26.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T18:02:26.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T18:02:26.212 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:02:26.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2021-01-15T18:02:26.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2021-01-15T18:02:26.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2021-01-15T18:02:26.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T18:02:26.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:02:26.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T18:02:26.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:02:26.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:02:26.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.218 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T18:02:26.218 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:02:26.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2021-01-15T18:02:26.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2021-01-15T18:02:26.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2021-01-15T18:02:26.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2021-01-15T18:02:26.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:02:26.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T18:02:26.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:02:26.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T18:02:26.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:02:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [2,4,3]/[1,0] ' 2021-01-15T18:02:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T18:02:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T18:02:26.231 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:02:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=400 2021-01-15T18:02:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 400 ']' 2021-01-15T18:02:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 400 400 2021-01-15T18:02:26.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T18:02:26.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:02:26.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T18:02:26.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:02:26.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:02:26.235 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:02:26.235 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T18:02:26.235 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:02:26.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=201 2021-01-15T18:02:26.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 201 ']' 2021-01-15T18:02:26.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 201 200 2021-01-15T18:02:26.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=201 2021-01-15T18:02:26.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:02:26.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=200 2021-01-15T18:02:26.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:02:26.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T18:02:26.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:431: TEST_backfill_sizeup4_allout: delete_pool test 2021-01-15T18:02:26.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:02:26.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:02:26.761 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:02:26.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:432: TEST_backfill_sizeup4_allout: kill_daemons td/osd-backfill-stats 2021-01-15T18:02:26.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:02:26.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:02:26.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:02:26.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:02:26.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:02:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:02:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T18:02:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:02:26.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:02:26.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:02:26.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:02:26.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:02:26.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:02:26.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:02:26.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:02:26.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:02:26.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:02:26.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:02:26.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:02:26.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:02:26.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:02:26.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:02:26.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:02:26.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:02:26.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:02:26.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:02:26.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:02:26.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:02:26.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:02:26.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:26.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:02:26.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:02:26.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:02:26.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:35: run: for func in $funcs 2021-01-15T18:02:26.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:36: run: setup td/osd-backfill-stats 2021-01-15T18:02:26.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-stats 2021-01-15T18:02:26.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:02:26.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:02:26.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:02:26.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:02:26.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:02:26.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:02:26.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:02:26.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:02:26.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:02:26.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:02:26.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:02:26.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:02:26.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:02:26.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:02:26.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:02:26.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:02:26.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:02:26.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:02:26.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:02:26.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:02:26.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:02:26.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:02:26.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:26.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:26.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:02:26.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:02:26.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:02:26.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-stats 2021-01-15T18:02:26.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:02:26.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:26.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:26.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.516267 2021-01-15T18:02:26.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:02:26.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:02:26.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:02:26.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-stats 1' TERM HUP INT 2021-01-15T18:02:26.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:37: run: TEST_backfill_sizeup_out td/osd-backfill-stats 2021-01-15T18:02:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:182: TEST_backfill_sizeup_out: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:184: TEST_backfill_sizeup_out: run_mon td/osd-backfill-stats a 2021-01-15T18:02:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-stats 2021-01-15T18:02:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:02:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:02:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:02:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-stats/a 2021-01-15T18:02:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-stats/a --run-dir=td/osd-backfill-stats 2021-01-15T18:02:27.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:02:27.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:27.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:27.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:27.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:27.211 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:27.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:27.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-stats/a '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-stats/log --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:02:27.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:02:27.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:02:27.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:02:27.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:02:27.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:02:27.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:02:27.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:02:27.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:02:27.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:02:27.381 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:02:27.382 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:27.382 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:27.382 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T18:02:27.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:02:27.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get fsid 2021-01-15T18:02:27.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:02:27.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:02:27.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:02:27.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:02:27.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:02:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:02:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:02:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:02:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:02:27.483 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:27.483 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:27.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.516267/ceph-mon.a.asok 2021-01-15T18:02:27.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:02:27.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.516267/ceph-mon.a.asok config get mon_host 2021-01-15T18:02:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:185: TEST_backfill_sizeup_out: run_mgr td/osd-backfill-stats x 2021-01-15T18:02:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-backfill-stats 2021-01-15T18:02:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:02:27.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:02:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:02:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-backfill-stats/x 2021-01-15T18:02:27.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:02:28.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:02:28.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:28.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:28.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:28.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:28.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:28.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:28.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:02:28.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-stats/x '--log-file=td/osd-backfill-stats/$name.log' '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --run-dir=td/osd-backfill-stats '--pid-file=td/osd-backfill-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:02:28.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:186: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 0 2021-01-15T18:02:28.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:02:28.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:02:28.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:02:28.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:02:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/0 2021-01-15T18:02:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:02:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:02:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:02:28.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:02:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/0' 2021-01-15T18:02:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/0/journal' 2021-01-15T18:02:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:02:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:02:28.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:02:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:02:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:28.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:28.119 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:28.119 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:28.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:02:28.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:02:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:02:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:02:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:02:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:02:28.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:02:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:02:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:02:28.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/0 2021-01-15T18:02:28.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:02:28.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=710b2eea-3dd8-4a69-acb3-42e7a0961ca5 2021-01-15T18:02:28.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 710b2eea-3dd8-4a69-acb3-42e7a0961ca5' 2021-01-15T18:02:28.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:02:28.124 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 710b2eea-3dd8-4a69-acb3-42e7a0961ca5 2021-01-15T18:02:28.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC02AFgBb7aBxAA40f/0qTWmD35kRrjZPDU9g== 2021-01-15T18:02:28.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC02AFgBb7aBxAA40f/0qTWmD35kRrjZPDU9g=="}' 2021-01-15T18:02:28.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 710b2eea-3dd8-4a69-acb3-42e7a0961ca5 -i td/osd-backfill-stats/0/new.json 2021-01-15T18:02:28.471 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:02:28.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/0/new.json 2021-01-15T18:02:28.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC02AFgBb7aBxAA40f/0qTWmD35kRrjZPDU9g== --osd-uuid 710b2eea-3dd8-4a69-acb3-42e7a0961ca5 2021-01-15T18:02:28.507 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.506+0000 7f9231962080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:28.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.508+0000 7f9231962080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:28.510 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.509+0000 7f9231962080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.510+0000 7f9231962080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:02:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.510+0000 7f9231962080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:02:28.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.510+0000 7f9231962080 -1 bluestore(td/osd-backfill-stats/0/block) _read_bdev_label failed to open td/osd-backfill-stats/0/block: (2) No such file or directory 2021-01-15T18:02:28.552 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:28.551+0000 7f9231962080 -1 bluestore(td/osd-backfill-stats/0) _read_fsid unparsable uuid 2021-01-15T18:02:33.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/0/keyring 2021-01-15T18:02:33.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:02:33.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:02:33.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:02:33.187 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:02:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:02:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/0 --osd-journal=td/osd-backfill-stats/0/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:33.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:02:33.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:02:33.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:02:33.715 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:02:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:33.736+0000 7f8a90bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:33.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:33.750+0000 7f8a90bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:33.753 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:33.752+0000 7f8a90bc7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:02:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:02:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:02:34.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:02:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:02:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:02:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:34.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:02:34.116 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:02:34.267 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:34.267+0000 7f8a90bc7080 -1 Falling back to public interface 2021-01-15T18:02:34.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:02:35.053 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:35.052+0000 7f8a90bc7080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:02:35.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:02:35.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:35.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:02:35.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:35.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:02:35.512 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:02:35.911 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/662523,v1:127.0.0.1:6803/662523] [v2:127.0.0.1:6804/662523,v1:127.0.0.1:6805/662523] exists,up 710b2eea-3dd8-4a69-acb3-42e7a0961ca5 2021-01-15T18:02:35.911 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 49da4b54-835e-4b37-989b-6daa1970f728 2021-01-15T18:02:35.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:02:35.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:02:35.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:02:35.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:187: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 1 2021-01-15T18:02:35.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:02:35.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:02:35.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:02:35.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:02:35.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/1 2021-01-15T18:02:35.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:02:35.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:02:35.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:02:35.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:02:35.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/1' 2021-01-15T18:02:35.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/1/journal' 2021-01-15T18:02:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:02:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:02:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:02:35.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:02:35.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:35.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:35.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:35.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:02:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:02:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:02:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:02:35.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:02:35.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:02:35.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:02:35.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:35.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:02:35.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:02:35.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/1 2021-01-15T18:02:35.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:02:35.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=49da4b54-835e-4b37-989b-6daa1970f728 2021-01-15T18:02:35.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 49da4b54-835e-4b37-989b-6daa1970f728' 2021-01-15T18:02:35.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:02:35.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC72AFg/FsuNxAARwknJ5vDsJkyTFwOiyU3rw== 2021-01-15T18:02:35.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC72AFg/FsuNxAARwknJ5vDsJkyTFwOiyU3rw=="}' 2021-01-15T18:02:35.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 49da4b54-835e-4b37-989b-6daa1970f728 -i td/osd-backfill-stats/1/new.json 2021-01-15T18:02:36.429 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:02:36.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/1/new.json 2021-01-15T18:02:36.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC72AFg/FsuNxAARwknJ5vDsJkyTFwOiyU3rw== --osd-uuid 49da4b54-835e-4b37-989b-6daa1970f728 2021-01-15T18:02:36.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.465+0000 7fc2c28b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:36.474 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.474+0000 7fc2c28b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:36.475 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.475+0000 7fc2c28b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.475+0000 7fc2c28b1080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:02:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.475+0000 7fc2c28b1080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:02:36.477 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.475+0000 7fc2c28b1080 -1 bluestore(td/osd-backfill-stats/1/block) _read_bdev_label failed to open td/osd-backfill-stats/1/block: (2) No such file or directory 2021-01-15T18:02:36.591 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:36.590+0000 7fc2c28b1080 -1 bluestore(td/osd-backfill-stats/1) _read_fsid unparsable uuid 2021-01-15T18:02:41.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/1/keyring 2021-01-15T18:02:41.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:02:41.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:02:41.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:02:41.167 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:02:41.689 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:02:41.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:02:41.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/1 --osd-journal=td/osd-backfill-stats/1/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:41.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:02:41.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:02:41.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:02:41.713 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:41.712+0000 7f2693229080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:41.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:41.729+0000 7f2693229080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:41.732 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:41.731+0000 7f2693229080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:42.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:02:42.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:02:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:02:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:02:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:02:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:02:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:02:42.083 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:02:42.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:42.246+0000 7f2693229080 -1 Falling back to public interface 2021-01-15T18:02:42.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:02:43.032 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:43.031+0000 7f2693229080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:02:43.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:02:43.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:02:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:43.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:02:43.497 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:02:43.916 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/663795,v1:127.0.0.1:6811/663795] [v2:127.0.0.1:6812/663795,v1:127.0.0.1:6813/663795] exists,up 49da4b54-835e-4b37-989b-6daa1970f728 2021-01-15T18:02:43.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:02:43.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:02:43.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:02:43.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:188: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 2 2021-01-15T18:02:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:02:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:02:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:02:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:02:43.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/2 2021-01-15T18:02:43.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:02:43.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:02:43.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:02:43.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:02:43.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/2' 2021-01-15T18:02:43.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/2/journal' 2021-01-15T18:02:43.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:02:43.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:02:43.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:02:43.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:02:43.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:43.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:43.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:43.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:43.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:43.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:43.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:43.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:02:43.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:02:43.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:02:43.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:02:43.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:02:43.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:02:43.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:02:43.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:43.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:02:43.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:02:43.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/2 2021-01-15T18:02:43.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:02:43.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d2bb8d3e-be02-4ccd-91c0-75df85be1ea2 2021-01-15T18:02:43.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 d2bb8d3e-be02-4ccd-91c0-75df85be1ea2' 2021-01-15T18:02:43.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:02:43.926 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 d2bb8d3e-be02-4ccd-91c0-75df85be1ea2 2021-01-15T18:02:43.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDD2AFgO5eQNxAA/OayR3M6gQN6sgZD53efgQ== 2021-01-15T18:02:43.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDD2AFgO5eQNxAA/OayR3M6gQN6sgZD53efgQ=="}' 2021-01-15T18:02:43.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d2bb8d3e-be02-4ccd-91c0-75df85be1ea2 -i td/osd-backfill-stats/2/new.json 2021-01-15T18:02:44.508 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:02:44.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/2/new.json 2021-01-15T18:02:44.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDD2AFgO5eQNxAA/OayR3M6gQN6sgZD53efgQ== --osd-uuid d2bb8d3e-be02-4ccd-91c0-75df85be1ea2 2021-01-15T18:02:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.544+0000 7f86ffbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:44.546 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.546+0000 7f86ffbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:44.548 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.547+0000 7f86ffbe4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:44.549 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.548+0000 7f86ffbe4080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:02:44.549 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.548+0000 7f86ffbe4080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:02:44.549 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.548+0000 7f86ffbe4080 -1 bluestore(td/osd-backfill-stats/2/block) _read_bdev_label failed to open td/osd-backfill-stats/2/block: (2) No such file or directory 2021-01-15T18:02:44.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:44.604+0000 7f86ffbe4080 -1 bluestore(td/osd-backfill-stats/2) _read_fsid unparsable uuid 2021-01-15T18:02:49.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/2/keyring 2021-01-15T18:02:49.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:02:49.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:02:49.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:02:49.142 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:02:49.676 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:02:49.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:02:49.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/2 --osd-journal=td/osd-backfill-stats/2/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:49.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:02:49.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:02:49.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:02:49.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:49.697+0000 7fdc46392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:49.711 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:49.711+0000 7fdc46392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:49.713 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:49.712+0000 7fdc46392080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:50.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:02:50.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:02:50.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:02:50.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:02:50.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:02:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:02:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:50.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:02:50.079 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:02:50.244 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:50.244+0000 7fdc46392080 -1 Falling back to public interface 2021-01-15T18:02:50.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:02:51.048 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:51.047+0000 7fdc46392080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:02:51.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:02:51.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:51.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:02:51.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:51.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:02:51.481 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:02:51.914 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/665059,v1:127.0.0.1:6819/665059] [v2:127.0.0.1:6820/665059,v1:127.0.0.1:6821/665059] exists,up d2bb8d3e-be02-4ccd-91c0-75df85be1ea2 2021-01-15T18:02:51.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:02:51.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:02:51.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:02:51.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:189: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 3 2021-01-15T18:02:51.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:02:51.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:02:51.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:02:51.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:02:51.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/3 2021-01-15T18:02:51.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:02:51.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:02:51.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:02:51.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:02:51.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/3' 2021-01-15T18:02:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/3/journal' 2021-01-15T18:02:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:02:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:02:51.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:02:51.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:02:51.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:02:51.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:02:51.919 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:02:51.919 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:02:51.920 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:02:51.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:51.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:02:51.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:02:51.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:02:51.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:02:51.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:02:51.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:02:51.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:02:51.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:02:51.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:51.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:02:51.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:02:51.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/3 2021-01-15T18:02:51.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:02:51.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1dfad18d-2b7d-496b-969c-a31d3a787b03 2021-01-15T18:02:51.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 1dfad18d-2b7d-496b-969c-a31d3a787b03' 2021-01-15T18:02:51.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:02:51.924 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 1dfad18d-2b7d-496b-969c-a31d3a787b03 2021-01-15T18:02:51.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDL2AFgOBtvNxAAz8qhZsCjLzacDXD/VZZBfg== 2021-01-15T18:02:51.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDL2AFgOBtvNxAAz8qhZsCjLzacDXD/VZZBfg=="}' 2021-01-15T18:02:51.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1dfad18d-2b7d-496b-969c-a31d3a787b03 -i td/osd-backfill-stats/3/new.json 2021-01-15T18:02:52.600 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:02:52.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/3/new.json 2021-01-15T18:02:52.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDL2AFgOBtvNxAAz8qhZsCjLzacDXD/VZZBfg== --osd-uuid 1dfad18d-2b7d-496b-969c-a31d3a787b03 2021-01-15T18:02:52.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.637+0000 7fcec4eaf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:52.639 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.638+0000 7fcec4eaf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:52.640 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.639+0000 7fcec4eaf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:52.640 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.640+0000 7fcec4eaf080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:02:52.641 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.640+0000 7fcec4eaf080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:02:52.641 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.640+0000 7fcec4eaf080 -1 bluestore(td/osd-backfill-stats/3/block) _read_bdev_label failed to open td/osd-backfill-stats/3/block: (2) No such file or directory 2021-01-15T18:02:52.689 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:52.689+0000 7fcec4eaf080 -1 bluestore(td/osd-backfill-stats/3) _read_fsid unparsable uuid 2021-01-15T18:02:57.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/3/keyring 2021-01-15T18:02:57.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:02:57.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:02:57.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:02:57.264 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:02:57.796 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:02:57.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:02:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/3 --osd-journal=td/osd-backfill-stats/3/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:02:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:02:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:02:57.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:02:57.819 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:57.818+0000 7fda0275b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:57.820 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:57.819+0000 7fda0275b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:57.822 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:57.821+0000 7fda0275b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:02:58.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:02:58.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:02:58.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:02:58.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:02:58.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:02:58.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:58.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:02:58.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:58.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:02:58.195 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:02:58.343 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:58.342+0000 7fda0275b080 -1 Falling back to public interface 2021-01-15T18:02:58.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:02:59.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:02:59.170+0000 7fda0275b080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:02:59.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:02:59.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:02:59.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:02:59.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:02:59.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:02:59.593 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:03:00.031 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/666323,v1:127.0.0.1:6827/666323] [v2:127.0.0.1:6828/666323,v1:127.0.0.1:6829/666323] exists,up 1dfad18d-2b7d-496b-969c-a31d3a787b03 2021-01-15T18:03:00.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:03:00.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:03:00.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:03:00.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:190: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 4 2021-01-15T18:03:00.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:03:00.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:03:00.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:03:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:03:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/4 2021-01-15T18:03:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:03:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:03:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:03:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:03:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/4' 2021-01-15T18:03:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/4/journal' 2021-01-15T18:03:00.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:03:00.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:03:00.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:03:00.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:03:00.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:03:00.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:03:00.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:03:00.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:03:00.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:03:00.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:03:00.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:03:00.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:03:00.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:03:00.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:03:00.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:03:00.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:03:00.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/4 2021-01-15T18:03:00.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:03:00.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=24d60be2-4d6d-4111-885d-030daa989ad9 2021-01-15T18:03:00.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 24d60be2-4d6d-4111-885d-030daa989ad9' 2021-01-15T18:03:00.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:03:00.040 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 24d60be2-4d6d-4111-885d-030daa989ad9 2021-01-15T18:03:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDU2AFgnEHLAhAArk7l24ccaa1AXROq4Rf4hw== 2021-01-15T18:03:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDU2AFgnEHLAhAArk7l24ccaa1AXROq4Rf4hw=="}' 2021-01-15T18:03:00.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 24d60be2-4d6d-4111-885d-030daa989ad9 -i td/osd-backfill-stats/4/new.json 2021-01-15T18:03:00.592 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:03:00.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/4/new.json 2021-01-15T18:03:00.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDU2AFgnEHLAhAArk7l24ccaa1AXROq4Rf4hw== --osd-uuid 24d60be2-4d6d-4111-885d-030daa989ad9 2021-01-15T18:03:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.628+0000 7fc8a1929080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:00.630 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.630+0000 7fc8a1929080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:00.632 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.631+0000 7fc8a1929080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:00.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.632+0000 7fc8a1929080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:03:00.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.632+0000 7fc8a1929080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:03:00.634 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.632+0000 7fc8a1929080 -1 bluestore(td/osd-backfill-stats/4/block) _read_bdev_label failed to open td/osd-backfill-stats/4/block: (2) No such file or directory 2021-01-15T18:03:00.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:00.686+0000 7fc8a1929080 -1 bluestore(td/osd-backfill-stats/4) _read_fsid unparsable uuid 2021-01-15T18:03:05.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/4/keyring 2021-01-15T18:03:05.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:03:05.455 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:03:05.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:03:05.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:03:06.014 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:03:06.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:03:06.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/4 --osd-journal=td/osd-backfill-stats/4/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:03:06.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:03:06.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:03:06.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:03:06.038 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:06.036+0000 7f9a11067080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:06.040 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:06.038+0000 7f9a11067080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:06.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:06.039+0000 7f9a11067080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:06.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:03:06.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:03:06.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:03:06.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:03:06.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:03:06.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:03:06.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:03:06.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:03:06.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:03:06.413 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:03:06.568 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:06.567+0000 7f9a11067080 -1 Falling back to public interface 2021-01-15T18:03:06.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:03:07.363 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:07.362+0000 7f9a11067080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:03:07.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:03:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:03:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:03:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:03:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:03:07.839 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:03:08.265 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/667588,v1:127.0.0.1:6835/667588] [v2:127.0.0.1:6836/667588,v1:127.0.0.1:6837/667588] exists,up 24d60be2-4d6d-4111-885d-030daa989ad9 2021-01-15T18:03:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:03:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:03:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:03:08.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:191: TEST_backfill_sizeup_out: run_osd td/osd-backfill-stats 5 2021-01-15T18:03:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-backfill-stats 2021-01-15T18:03:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:03:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:03:08.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:03:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-backfill-stats/5 2021-01-15T18:03:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-01-15T18:03:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:03:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:03:08.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:03:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-backfill-stats/5' 2021-01-15T18:03:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-stats/5/journal' 2021-01-15T18:03:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:03:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:03:08.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-backfill-stats' 2021-01-15T18:03:08.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:03:08.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:03:08.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:03:08.270 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:03:08.271 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:03:08.271 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:03:08.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:03:08.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' 2021-01-15T18:03:08.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:03:08.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:03:08.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:03:08.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-backfill-stats/$name.log' 2021-01-15T18:03:08.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-backfill-stats/$name.pid' 2021-01-15T18:03:08.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:03:08.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:03:08.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:03:08.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:03:08.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:03:08.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-backfill-stats/5 2021-01-15T18:03:08.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:03:08.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6eaeca31-b858-4fa3-9811-84584ddf21a5 2021-01-15T18:03:08.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 6eaeca31-b858-4fa3-9811-84584ddf21a5' 2021-01-15T18:03:08.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:03:08.276 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 6eaeca31-b858-4fa3-9811-84584ddf21a5 2021-01-15T18:03:08.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDc2AFgSDLAEBAACFAPVnTdgWtK/52h6JuiBw== 2021-01-15T18:03:08.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDc2AFgSDLAEBAACFAPVnTdgWtK/52h6JuiBw=="}' 2021-01-15T18:03:08.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6eaeca31-b858-4fa3-9811-84584ddf21a5 -i td/osd-backfill-stats/5/new.json 2021-01-15T18:03:08.789 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:03:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-backfill-stats/5/new.json 2021-01-15T18:03:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDc2AFgSDLAEBAACFAPVnTdgWtK/52h6JuiBw== --osd-uuid 6eaeca31-b858-4fa3-9811-84584ddf21a5 2021-01-15T18:03:08.826 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.825+0000 7f2d5e039080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:08.862 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.861+0000 7f2d5e039080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:08.864 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.863+0000 7f2d5e039080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:08.865 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.864+0000 7f2d5e039080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T18:03:08.865 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.864+0000 7f2d5e039080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T18:03:08.865 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.864+0000 7f2d5e039080 -1 bluestore(td/osd-backfill-stats/5/block) _read_bdev_label failed to open td/osd-backfill-stats/5/block: (2) No such file or directory 2021-01-15T18:03:08.928 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:08.927+0000 7f2d5e039080 -1 bluestore(td/osd-backfill-stats/5) _read_fsid unparsable uuid 2021-01-15T18:03:13.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-backfill-stats/5/keyring 2021-01-15T18:03:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:03:13.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:03:13.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-backfill-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:03:13.673 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:03:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:03:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=568d49d9-bc18-48c9-a0db-97983d506ab5 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-stats/5 --osd-journal=td/osd-backfill-stats/5/journal --chdir= --run-dir=td/osd-backfill-stats '--admin-socket=/tmp/ceph-asok.516267/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-stats/$name.log' '--pid-file=td/osd-backfill-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:03:14.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:03:14.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:03:14.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:03:14.202 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:03:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:14.223+0000 7fd79a044080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:14.225 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:14.225+0000 7fd79a044080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:14.227 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:14.226+0000 7fd79a044080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:03:14.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:03:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:03:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:03:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:03:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:03:14.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:03:14.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:03:14.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:03:14.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:03:14.594 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:03:14.748 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:14.747+0000 7fd79a044080 -1 Falling back to public interface 2021-01-15T18:03:15.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:03:15.549 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:03:15.548+0000 7fd79a044080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:03:16.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:03:16.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:03:16.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:03:16.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:03:16.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:03:16.009 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:03:16.429 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/668852,v1:127.0.0.1:6843/668852] [v2:127.0.0.1:6844/668852,v1:127.0.0.1:6845/668852] exists,up 6eaeca31-b858-4fa3-9811-84584ddf21a5 2021-01-15T18:03:16.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:03:16.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:03:16.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:03:16.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:193: TEST_backfill_sizeup_out: create_pool test 1 1 2021-01-15T18:03:16.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:03:17.134 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:03:17.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:03:18.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:194: TEST_backfill_sizeup_out: ceph osd pool set test size 1 --yes-i-really-mean-it 2021-01-15T18:03:19.023 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T18:03:19.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:196: TEST_backfill_sizeup_out: wait_for_clean 2021-01-15T18:03:19.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:03:19.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:03:19.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:03:19.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:03:19.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:03:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:03:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:03:19.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:03:19.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:03:19.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:03:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:03:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:03:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:03:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:03:19.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:03:19.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:03:19.557 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:03:19.557 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:03:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:19.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:03:19.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T18:03:19.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T18:03:19.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T18:03:19.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:19.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:03:19.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T18:03:19.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T18:03:19.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-01-15T18:03:19.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:19.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:03:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T18:03:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T18:03:19.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448' 2021-01-15T18:03:19.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:19.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:03:20.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345926 2021-01-15T18:03:20.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345926 2021-01-15T18:03:20.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345926' 2021-01-15T18:03:20.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:20.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:03:20.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T18:03:20.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T18:03:20.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345926 4-107374182405' 2021-01-15T18:03:20.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:20.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:03:20.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018883 2021-01-15T18:03:20.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018883 2021-01-15T18:03:20.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345926 4-107374182405 5-128849018883' 2021-01-15T18:03:20.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:20.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T18:03:20.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:20.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:03:20.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T18:03:20.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:20.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T18:03:20.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T18:03:20.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:03:20.386 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T18:03:20.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836491 2021-01-15T18:03:20.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:20.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T18:03:20.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:03:20.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T18:03:20.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T18:03:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T18:03:20.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:03:20.773 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T18:03:21.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T18:03:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:21.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T18:03:21.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:21.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:03:21.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T18:03:21.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:21.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T18:03:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T18:03:21.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:03:21.199 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T18:03:21.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509448 2021-01-15T18:03:21.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:03:22.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:03:22.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:03:23.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T18:03:23.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:23.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345926 2021-01-15T18:03:23.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:23.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:03:23.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345926 2021-01-15T18:03:23.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:23.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345926 2021-01-15T18:03:23.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2021-01-15T18:03:23.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:03:23.012 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345926 2021-01-15T18:03:23.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345926 2021-01-15T18:03:23.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:23.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T18:03:23.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:23.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:03:23.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T18:03:23.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:23.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T18:03:23.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T18:03:23.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:03:23.420 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T18:03:23.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T18:03:23.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:23.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018883 2021-01-15T18:03:23.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:23.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:03:23.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018883 2021-01-15T18:03:23.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:23.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018883 2021-01-15T18:03:23.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018883' 2021-01-15T18:03:23.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:03:23.836 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018883 2021-01-15T18:03:24.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018884 -lt 128849018883 2021-01-15T18:03:24.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:03:24.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:24.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:24.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:03:24.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:24.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:24.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:24.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:24.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:24.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:24.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:25.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:03:25.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:25.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:25.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:25.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:03:25.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:03:25.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:03:25.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: seq 1 200 2021-01-15T18:03:25.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:25.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj1 /dev/null 2021-01-15T18:03:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:25.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj2 /dev/null 2021-01-15T18:03:25.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:25.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj3 /dev/null 2021-01-15T18:03:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj4 /dev/null 2021-01-15T18:03:25.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:25.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj5 /dev/null 2021-01-15T18:03:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj6 /dev/null 2021-01-15T18:03:26.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj7 /dev/null 2021-01-15T18:03:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj8 /dev/null 2021-01-15T18:03:26.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj9 /dev/null 2021-01-15T18:03:26.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj10 /dev/null 2021-01-15T18:03:26.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj11 /dev/null 2021-01-15T18:03:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj12 /dev/null 2021-01-15T18:03:26.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj13 /dev/null 2021-01-15T18:03:26.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj14 /dev/null 2021-01-15T18:03:26.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj15 /dev/null 2021-01-15T18:03:26.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj16 /dev/null 2021-01-15T18:03:26.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj17 /dev/null 2021-01-15T18:03:26.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj18 /dev/null 2021-01-15T18:03:26.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj19 /dev/null 2021-01-15T18:03:26.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj20 /dev/null 2021-01-15T18:03:26.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:26.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj21 /dev/null 2021-01-15T18:03:27.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj22 /dev/null 2021-01-15T18:03:27.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj23 /dev/null 2021-01-15T18:03:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj24 /dev/null 2021-01-15T18:03:27.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj25 /dev/null 2021-01-15T18:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj26 /dev/null 2021-01-15T18:03:27.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj27 /dev/null 2021-01-15T18:03:27.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj28 /dev/null 2021-01-15T18:03:27.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj29 /dev/null 2021-01-15T18:03:27.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj30 /dev/null 2021-01-15T18:03:27.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj31 /dev/null 2021-01-15T18:03:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj32 /dev/null 2021-01-15T18:03:27.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj33 /dev/null 2021-01-15T18:03:27.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj34 /dev/null 2021-01-15T18:03:27.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj35 /dev/null 2021-01-15T18:03:27.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:27.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj36 /dev/null 2021-01-15T18:03:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj37 /dev/null 2021-01-15T18:03:28.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj38 /dev/null 2021-01-15T18:03:28.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj39 /dev/null 2021-01-15T18:03:28.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj40 /dev/null 2021-01-15T18:03:28.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj41 /dev/null 2021-01-15T18:03:28.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj42 /dev/null 2021-01-15T18:03:28.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj43 /dev/null 2021-01-15T18:03:28.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj44 /dev/null 2021-01-15T18:03:28.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj45 /dev/null 2021-01-15T18:03:28.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj46 /dev/null 2021-01-15T18:03:28.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj47 /dev/null 2021-01-15T18:03:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj48 /dev/null 2021-01-15T18:03:28.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj49 /dev/null 2021-01-15T18:03:28.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj50 /dev/null 2021-01-15T18:03:28.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj51 /dev/null 2021-01-15T18:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj52 /dev/null 2021-01-15T18:03:29.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj53 /dev/null 2021-01-15T18:03:29.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj54 /dev/null 2021-01-15T18:03:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj55 /dev/null 2021-01-15T18:03:29.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj56 /dev/null 2021-01-15T18:03:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj57 /dev/null 2021-01-15T18:03:29.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj58 /dev/null 2021-01-15T18:03:29.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj59 /dev/null 2021-01-15T18:03:29.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj60 /dev/null 2021-01-15T18:03:29.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj61 /dev/null 2021-01-15T18:03:29.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj62 /dev/null 2021-01-15T18:03:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj63 /dev/null 2021-01-15T18:03:29.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj64 /dev/null 2021-01-15T18:03:29.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj65 /dev/null 2021-01-15T18:03:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj66 /dev/null 2021-01-15T18:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj67 /dev/null 2021-01-15T18:03:30.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj68 /dev/null 2021-01-15T18:03:30.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj69 /dev/null 2021-01-15T18:03:30.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj70 /dev/null 2021-01-15T18:03:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj71 /dev/null 2021-01-15T18:03:30.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj72 /dev/null 2021-01-15T18:03:30.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj73 /dev/null 2021-01-15T18:03:30.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj74 /dev/null 2021-01-15T18:03:30.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj75 /dev/null 2021-01-15T18:03:30.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj76 /dev/null 2021-01-15T18:03:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj77 /dev/null 2021-01-15T18:03:30.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj78 /dev/null 2021-01-15T18:03:30.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj79 /dev/null 2021-01-15T18:03:30.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj80 /dev/null 2021-01-15T18:03:30.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj81 /dev/null 2021-01-15T18:03:30.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:30.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj82 /dev/null 2021-01-15T18:03:31.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj83 /dev/null 2021-01-15T18:03:31.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj84 /dev/null 2021-01-15T18:03:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj85 /dev/null 2021-01-15T18:03:31.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj86 /dev/null 2021-01-15T18:03:31.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj87 /dev/null 2021-01-15T18:03:31.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj88 /dev/null 2021-01-15T18:03:31.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj89 /dev/null 2021-01-15T18:03:31.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj90 /dev/null 2021-01-15T18:03:31.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj91 /dev/null 2021-01-15T18:03:31.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj92 /dev/null 2021-01-15T18:03:31.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj93 /dev/null 2021-01-15T18:03:31.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj94 /dev/null 2021-01-15T18:03:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj95 /dev/null 2021-01-15T18:03:32.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj96 /dev/null 2021-01-15T18:03:32.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj97 /dev/null 2021-01-15T18:03:32.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj98 /dev/null 2021-01-15T18:03:32.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj99 /dev/null 2021-01-15T18:03:32.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj100 /dev/null 2021-01-15T18:03:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj101 /dev/null 2021-01-15T18:03:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj102 /dev/null 2021-01-15T18:03:32.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj103 /dev/null 2021-01-15T18:03:32.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj104 /dev/null 2021-01-15T18:03:32.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj105 /dev/null 2021-01-15T18:03:32.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj106 /dev/null 2021-01-15T18:03:32.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj107 /dev/null 2021-01-15T18:03:32.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj108 /dev/null 2021-01-15T18:03:32.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj109 /dev/null 2021-01-15T18:03:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj110 /dev/null 2021-01-15T18:03:33.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj111 /dev/null 2021-01-15T18:03:33.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj112 /dev/null 2021-01-15T18:03:33.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj113 /dev/null 2021-01-15T18:03:33.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj114 /dev/null 2021-01-15T18:03:33.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj115 /dev/null 2021-01-15T18:03:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj116 /dev/null 2021-01-15T18:03:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj117 /dev/null 2021-01-15T18:03:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj118 /dev/null 2021-01-15T18:03:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj119 /dev/null 2021-01-15T18:03:33.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj120 /dev/null 2021-01-15T18:03:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj121 /dev/null 2021-01-15T18:03:33.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj122 /dev/null 2021-01-15T18:03:33.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj123 /dev/null 2021-01-15T18:03:33.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj124 /dev/null 2021-01-15T18:03:33.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj125 /dev/null 2021-01-15T18:03:33.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:33.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj126 /dev/null 2021-01-15T18:03:34.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj127 /dev/null 2021-01-15T18:03:34.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj128 /dev/null 2021-01-15T18:03:34.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj129 /dev/null 2021-01-15T18:03:34.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj130 /dev/null 2021-01-15T18:03:34.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj131 /dev/null 2021-01-15T18:03:34.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj132 /dev/null 2021-01-15T18:03:34.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj133 /dev/null 2021-01-15T18:03:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj134 /dev/null 2021-01-15T18:03:34.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj135 /dev/null 2021-01-15T18:03:34.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj136 /dev/null 2021-01-15T18:03:34.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj137 /dev/null 2021-01-15T18:03:34.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj138 /dev/null 2021-01-15T18:03:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj139 /dev/null 2021-01-15T18:03:34.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj140 /dev/null 2021-01-15T18:03:34.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:34.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj141 /dev/null 2021-01-15T18:03:35.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj142 /dev/null 2021-01-15T18:03:35.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj143 /dev/null 2021-01-15T18:03:35.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj144 /dev/null 2021-01-15T18:03:35.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj145 /dev/null 2021-01-15T18:03:35.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj146 /dev/null 2021-01-15T18:03:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj147 /dev/null 2021-01-15T18:03:35.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj148 /dev/null 2021-01-15T18:03:35.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj149 /dev/null 2021-01-15T18:03:35.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj150 /dev/null 2021-01-15T18:03:35.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj151 /dev/null 2021-01-15T18:03:35.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj152 /dev/null 2021-01-15T18:03:35.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj153 /dev/null 2021-01-15T18:03:35.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj154 /dev/null 2021-01-15T18:03:35.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj155 /dev/null 2021-01-15T18:03:35.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj156 /dev/null 2021-01-15T18:03:35.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:35.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj157 /dev/null 2021-01-15T18:03:36.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj158 /dev/null 2021-01-15T18:03:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj159 /dev/null 2021-01-15T18:03:36.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj160 /dev/null 2021-01-15T18:03:36.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj161 /dev/null 2021-01-15T18:03:36.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj162 /dev/null 2021-01-15T18:03:36.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj163 /dev/null 2021-01-15T18:03:36.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj164 /dev/null 2021-01-15T18:03:36.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj165 /dev/null 2021-01-15T18:03:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj166 /dev/null 2021-01-15T18:03:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj167 /dev/null 2021-01-15T18:03:36.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj168 /dev/null 2021-01-15T18:03:36.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj169 /dev/null 2021-01-15T18:03:36.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:36.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj170 /dev/null 2021-01-15T18:03:37.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj171 /dev/null 2021-01-15T18:03:37.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj172 /dev/null 2021-01-15T18:03:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj173 /dev/null 2021-01-15T18:03:37.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj174 /dev/null 2021-01-15T18:03:37.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj175 /dev/null 2021-01-15T18:03:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj176 /dev/null 2021-01-15T18:03:37.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj177 /dev/null 2021-01-15T18:03:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj178 /dev/null 2021-01-15T18:03:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj179 /dev/null 2021-01-15T18:03:37.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj180 /dev/null 2021-01-15T18:03:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj181 /dev/null 2021-01-15T18:03:37.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj182 /dev/null 2021-01-15T18:03:37.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj183 /dev/null 2021-01-15T18:03:37.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj184 /dev/null 2021-01-15T18:03:37.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:37.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj185 /dev/null 2021-01-15T18:03:38.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj186 /dev/null 2021-01-15T18:03:38.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj187 /dev/null 2021-01-15T18:03:38.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj188 /dev/null 2021-01-15T18:03:38.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj189 /dev/null 2021-01-15T18:03:38.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj190 /dev/null 2021-01-15T18:03:38.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj191 /dev/null 2021-01-15T18:03:38.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj192 /dev/null 2021-01-15T18:03:38.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj193 /dev/null 2021-01-15T18:03:38.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj194 /dev/null 2021-01-15T18:03:38.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj195 /dev/null 2021-01-15T18:03:38.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj196 /dev/null 2021-01-15T18:03:38.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj197 /dev/null 2021-01-15T18:03:38.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj198 /dev/null 2021-01-15T18:03:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj199 /dev/null 2021-01-15T18:03:38.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:198: TEST_backfill_sizeup_out: for i in $(seq 1 $objects) 2021-01-15T18:03:38.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:200: TEST_backfill_sizeup_out: rados -p test put obj200 /dev/null 2021-01-15T18:03:38.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:203: TEST_backfill_sizeup_out: get_pg test obj1 2021-01-15T18:03:38.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:03:38.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:03:38.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:03:38.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:03:39.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:203: TEST_backfill_sizeup_out: local PG=1.0 2021-01-15T18:03:39.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:205: TEST_backfill_sizeup_out: get_primary test obj1 2021-01-15T18:03:39.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:03:39.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:03:39.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:03:39.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:03:39.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:205: TEST_backfill_sizeup_out: local primary=1 2021-01-15T18:03:39.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:207: TEST_backfill_sizeup_out: ceph osd set nobackfill 2021-01-15T18:03:40.551 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is set 2021-01-15T18:03:40.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:208: TEST_backfill_sizeup_out: ceph osd out osd.1 2021-01-15T18:03:41.092 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T18:03:41.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:209: TEST_backfill_sizeup_out: ceph osd pool set test size 3 2021-01-15T18:03:41.889 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T18:03:41.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:210: TEST_backfill_sizeup_out: sleep 2 2021-01-15T18:03:43.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:211: TEST_backfill_sizeup_out: ceph osd unset nobackfill 2021-01-15T18:03:44.633 INFO:tasks.workunit.client.0.smithi145.stderr:nobackfill is unset 2021-01-15T18:03:44.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:213: TEST_backfill_sizeup_out: wait_for_clean 2021-01-15T18:03:44.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:03:44.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:03:44.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:03:44.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:03:44.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:03:44.646 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:03:44.646 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:03:44.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:03:44.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:03:44.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:03:44.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:03:44.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:03:44.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:03:44.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:03:44.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:03:45.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:03:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:03:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:03:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:03:45.128 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:03:45.129 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:03:45.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:03:45.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:03:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836498 2021-01-15T18:03:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836498 2021-01-15T18:03:45.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498' 2021-01-15T18:03:45.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:03:45.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672977 2021-01-15T18:03:45.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672977 2021-01-15T18:03:45.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977' 2021-01-15T18:03:45.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:03:45.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509455 2021-01-15T18:03:45.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509455 2021-01-15T18:03:45.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509455' 2021-01-15T18:03:45.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:03:45.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345934 2021-01-15T18:03:45.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345934 2021-01-15T18:03:45.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509455 3-85899345934' 2021-01-15T18:03:45.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:03:45.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182412 2021-01-15T18:03:45.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182412 2021-01-15T18:03:45.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509455 3-85899345934 4-107374182412' 2021-01-15T18:03:45.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:03:45.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:03:45.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018890 2021-01-15T18:03:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018890 2021-01-15T18:03:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836498 1-42949672977 2-64424509455 3-85899345934 4-107374182412 5-128849018890' 2021-01-15T18:03:45.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:45.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2021-01-15T18:03:45.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:45.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:03:45.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836498 2021-01-15T18:03:45.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:45.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836498 2021-01-15T18:03:45.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2021-01-15T18:03:45.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:03:45.954 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836498 2021-01-15T18:03:46.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836498 2021-01-15T18:03:46.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:46.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672977 2021-01-15T18:03:46.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:46.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:03:46.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672977 2021-01-15T18:03:46.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:46.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672977 2021-01-15T18:03:46.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2021-01-15T18:03:46.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:03:46.383 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672977 2021-01-15T18:03:46.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672977 -lt 42949672977 2021-01-15T18:03:46.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:46.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509455 2021-01-15T18:03:46.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:46.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:03:46.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509455 2021-01-15T18:03:46.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:46.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509455 2021-01-15T18:03:46.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2021-01-15T18:03:46.776 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509455 2021-01-15T18:03:46.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:03:47.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509455 -lt 64424509455 2021-01-15T18:03:47.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:47.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345934 2021-01-15T18:03:47.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:47.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:03:47.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345934 2021-01-15T18:03:47.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:47.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345934 2021-01-15T18:03:47.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2021-01-15T18:03:47.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:03:47.174 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345934 2021-01-15T18:03:47.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345934 -lt 85899345934 2021-01-15T18:03:47.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:47.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182412 2021-01-15T18:03:47.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:47.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:03:47.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182412 2021-01-15T18:03:47.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:47.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182412 2021-01-15T18:03:47.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182412' 2021-01-15T18:03:47.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:03:47.609 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182412 2021-01-15T18:03:48.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182412 -lt 107374182412 2021-01-15T18:03:48.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:03:48.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018890 2021-01-15T18:03:48.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:03:48.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:03:48.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018890 2021-01-15T18:03:48.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:03:48.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018890 2021-01-15T18:03:48.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2021-01-15T18:03:48.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:03:48.014 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018890 2021-01-15T18:03:48.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018891 -lt 128849018890 2021-01-15T18:03:48.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:03:48.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:48.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:48.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:03:48.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:48.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:48.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:48.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:48.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:48.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:48.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:49.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:49.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:49.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:49.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:03:49.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:03:49.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:03:49.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:49.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:49.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:49.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:49.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:49.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:49.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:50.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:50.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:50.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:50.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:50.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:50.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:50.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:50.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:50.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:50.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:50.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:50.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:50.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:51.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=0 2021-01-15T18:03:51.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 0 '!=' null 2021-01-15T18:03:51.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:51.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:51.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:51.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:51.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:51.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:51.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:51.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:51.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:51.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:51.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:51.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:51.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:51.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:51.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:52.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:52.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:52.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:52.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:52.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:52.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:52.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:52.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:52.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:53.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=0 2021-01-15T18:03:53.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 0 '!=' null 2021-01-15T18:03:53.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:53.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:53.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:53.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:53.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:53.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:53.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:53.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:53.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:53.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:53.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:53.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:54.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:54.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:54.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:54.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:54.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:54.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:54.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:54.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:54.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:54.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=0 2021-01-15T18:03:54.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 0 '!=' null 2021-01-15T18:03:54.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:54.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:54.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:54.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:54.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:54.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:54.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:54.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:54.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:54.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:54.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:55.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:55.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:55.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:55.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:55.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:55.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:55.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:55.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:55.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:56.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:03:56.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:03:56.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:56.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:56.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:56.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:56.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:56.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:56.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:56.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:56.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:56.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:56.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:56.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:56.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:56.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:56.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:57.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:57.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:57.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:57.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:57.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:57.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:57.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:57.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:57.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:03:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:03:57.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:57.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:57.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:57.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:57.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:58.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:58.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:58.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:58.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:03:58.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:03:58.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:03:58.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:03:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:03:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:03:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:03:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:03:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:03:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:03:59.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:03:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:03:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:03:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:03:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:03:59.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:03:59.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:03:59.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:03:59.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:03:59.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:03:59.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:03:59.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:03:59.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:03:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:03:59.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:03:59.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:03:59.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:00.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:00.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:00.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:00.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:00.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:00.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:00.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:00.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:00.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:00.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:04:00.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:04:00.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:00.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:00.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:00.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:00.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:00.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:00.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:00.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:00.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:00.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:01.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:01.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:01.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:01.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:01.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:01.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:01.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:01.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:01.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:01.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:01.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:01.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:01.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:02.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:02.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:02.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:02.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:02.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:02.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:02.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:02.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:02.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:03.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:03.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:03.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:03.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:03.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:03.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:03.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:03.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:03.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:03.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:03.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:03.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:03.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:03.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:03.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:03.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:03.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:03.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:03.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:04.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:04.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:04.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:04.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:04.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:05.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:05.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:05.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:05.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:05.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:05.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:05.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:05.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:05.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:05.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:05.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:05.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:05.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:05.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:05.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:05.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:05.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:06.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:06.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:06.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:06.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:06.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:06.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:06.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:06.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:06.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:07.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:07.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:07.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:07.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:07.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:07.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:07.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:07.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:07.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:07.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:07.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:07.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:07.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:07.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:07.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:07.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:07.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:08.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:08.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:08.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:08.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:08.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:08.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:08.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:08.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:08.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:08.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:08.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:08.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:08.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:08.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:08.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:08.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:09.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:09.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:09.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:09.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:09.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:09.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:09.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:09.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:10.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:10.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:10.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:10.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:10.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:10.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:10.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:10.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:10.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:10.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:10.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:10.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:10.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:10.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:10.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:10.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:10.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:11.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:11.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:11.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:11.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:11.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:11.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:11.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:11.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:11.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:11.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:11.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:11.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:11.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:11.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:11.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:11.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:11.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:11.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:11.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:11.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:11.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:11.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:12.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:12.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:12.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:12.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:12.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:12.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:12.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:12.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:12.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:12.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:12.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:12.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:12.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:13.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:13.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:13.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:13.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:13.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:13.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:13.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:13.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:13.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:13.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:13.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:13.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:13.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:13.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:13.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:13.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:13.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:14.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:14.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:14.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:14.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:14.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:14.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:14.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:14.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:14.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:14.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:14.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:14.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:14.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:14.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:14.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:14.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:14.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:14.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:14.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:14.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:14.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:14.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:15.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:15.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:15.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:15.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:15.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:15.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:15.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:15.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:15.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:15.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:15.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:15.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:15.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:16.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:16.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:16.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:16.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:16.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:16.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:16.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:16.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:16.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:16.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:16.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:16.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:16.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:16.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:16.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:17.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:17.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:17.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:17.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:17.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:17.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:17.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:17.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:17.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:17.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:17.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:17.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:17.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:17.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:17.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:17.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:17.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:17.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:17.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:17.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:17.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:17.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:18.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:18.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:18.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:18.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:18.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:18.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:18.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:18.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:18.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:18.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:18.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:18.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:18.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:19.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:19.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:19.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:19.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:19.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:19.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:19.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:19.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:20.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:20.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:20.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:20.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:20.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:20.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:20.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:20.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:20.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:20.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:20.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:21.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:21.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:21.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:21.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:21.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:21.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:22.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:22.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:22.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:22.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:22.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:22.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:22.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:22.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:22.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:22.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:22.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:22.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:22.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:22.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:23.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:23.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:23.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:23.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:23.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:23.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:23.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:23.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:23.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:23.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:23.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:23.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:23.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:23.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:24.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:24.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:24.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:24.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:24.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:24.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:24.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:24.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:24.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:24.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:24.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:24.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:24.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:24.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:24.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:24.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:24.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:24.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:24.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:24.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:24.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:25.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:25.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:25.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:25.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:25.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:25.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:25.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:25.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:25.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:25.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:25.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:25.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:25.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:25.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:25.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:25.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:25.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:26.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:26.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:26.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:26.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:26.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:26.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:26.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:26.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:26.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:27.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:27.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:27.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:27.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:27.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:27.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:27.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:27.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:27.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:27.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:27.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:27.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:27.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:27.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:27.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:28.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:28.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:28.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:28.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:28.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:28.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:28.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:28.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:28.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:28.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:28.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:29.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:29.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:29.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:29.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:29.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:29.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:29.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:29.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:29.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:29.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:29.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:30.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:30.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:30.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:30.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:30.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:30.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:30.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:30.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:30.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:30.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:30.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:30.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:30.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:31.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:31.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:31.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:31.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:31.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:31.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:31.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:31.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:31.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:31.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:31.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:31.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:31.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:31.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:32.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:32.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:32.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:32.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:32.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:32.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:32.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:32.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:33.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:33.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:33.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:33.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:33.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:33.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:33.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:33.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:33.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:33.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:33.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:33.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:33.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:33.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:33.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:33.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:33.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:34.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:34.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:34.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:34.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:34.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:34.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:34.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:34.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:34.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:04:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:04:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:34.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:34.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:34.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:34.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:34.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:34.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:34.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:34.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:34.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:35.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:35.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:35.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:35.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:35.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:35.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:35.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:35.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:35.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:35.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:35.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:35.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:36.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:36.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:36.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:36.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:36.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:36.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:36.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:36.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:36.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:36.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:36.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:36.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:36.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:36.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:04:36.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:36.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:36.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:37.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:04:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:04:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:04:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:04:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:04:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:04:37.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:04:37.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:04:37.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:04:37.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:04:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:04:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:04:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:04:37.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:04:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:04:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:04:37.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:04:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:04:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:04:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:04:37.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:04:37.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:04:38.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:04:38.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:04:38.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:04:38.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:04:38.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:04:38.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:04:38.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:04:38.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:215: TEST_backfill_sizeup_out: expr 200 '*' 2 2021-01-15T18:04:38.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:215: TEST_backfill_sizeup_out: local degraded=400 2021-01-15T18:04:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:216: TEST_backfill_sizeup_out: check td/osd-backfill-stats 1.0 1 replicated 400 0 200 0 2021-01-15T18:04:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:63: check: local dir=td/osd-backfill-stats 2021-01-15T18:04:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:64: check: local PG=1.0 2021-01-15T18:04:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:65: check: local primary=1 2021-01-15T18:04:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:66: check: local type=replicated 2021-01-15T18:04:38.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:67: check: local degraded_start=400 2021-01-15T18:04:38.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:68: check: local degraded_end=0 2021-01-15T18:04:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:69: check: local misplaced_start=200 2021-01-15T18:04:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:70: check: local misplaced_end=0 2021-01-15T18:04:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:71: check: local primary_start= 2021-01-15T18:04:38.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:72: check: local primary_end= 2021-01-15T18:04:38.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:73: check: local check_setup=true 2021-01-15T18:04:38.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: grep -l +backfilling td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:75: check: local log=td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:76: check: '[' true = true ']' 2021-01-15T18:04:38.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: grep -l +backfilling td/osd-backfill-stats/osd.0.log td/osd-backfill-stats/osd.1.log td/osd-backfill-stats/osd.2.log td/osd-backfill-stats/osd.3.log td/osd-backfill-stats/osd.4.log td/osd-backfill-stats/osd.5.log 2021-01-15T18:04:38.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:78: check: local alllogs=td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: echo td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: wc -w 2021-01-15T18:04:38.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:79: check: '[' 1 '!=' 1 ']' 2021-01-15T18:04:38.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:86: check: local 'addp= ' 2021-01-15T18:04:38.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:87: check: '[' replicated = erasure ']' 2021-01-15T18:04:38.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:04:38.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: tail -1 2021-01-15T18:04:38.836 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:04:38.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:92: check: UPACT='[0,2,4]/[1]' 2021-01-15T18:04:38.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:93: check: '[' -n '[0,2,4]/[1]' ']' 2021-01-15T18:04:38.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:96: check: local which=degraded 2021-01-15T18:04:38.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:04:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -F ' [0,2,4]/[1] ' 2021-01-15T18:04:38.858 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: grep -v est 2021-01-15T18:04:38.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: head -1 2021-01-15T18:04:38.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:04:38.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:97: check: FIRST=400 2021-01-15T18:04:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:98: check: '[' -n 400 ']' 2021-01-15T18:04:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:99: check: below_margin 400 400 2021-01-15T18:04:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=400 2021-01-15T18:04:38.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:04:38.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=400 2021-01-15T18:04:38.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:04:38.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:04:38.864 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats degraded ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.864 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: tail -1 2021-01-15T18:04:38.864 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:04:38.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:100: check: LAST=2 2021-01-15T18:04:38.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:101: check: '[' -n 2 ']' 2021-01-15T18:04:38.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:102: check: above_margin 2 0 2021-01-15T18:04:38.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=2 2021-01-15T18:04:38.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:04:38.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T18:04:38.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:04:38.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:105: check: which=misplaced 2021-01-15T18:04:38.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:04:38.875 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.875 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -F ' [0,2,4]/[1] ' 2021-01-15T18:04:38.875 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: grep -v est 2021-01-15T18:04:38.876 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: head -1 2021-01-15T18:04:38.876 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:04:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:106: check: FIRST=200 2021-01-15T18:04:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:107: check: '[' -n 200 ']' 2021-01-15T18:04:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:108: check: below_margin 200 200 2021-01-15T18:04:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:43: below_margin: local -i check=200 2021-01-15T18:04:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:44: below_margin: shift 2021-01-15T18:04:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:45: below_margin: local -i target=200 2021-01-15T18:04:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:47: below_margin: return 0 2021-01-15T18:04:38.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: eval grep '"pg[[]${PG}.*backfilling.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:04:38.879 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: grep 'pg[[]1.0.*backfilling.*update_calc_stats misplaced ' td/osd-backfill-stats/osd.1.log 2021-01-15T18:04:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: tail -1 2021-01-15T18:04:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:04:38.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:109: check: LAST=1 2021-01-15T18:04:38.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:110: check: '[' -n 1 ']' 2021-01-15T18:04:38.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:111: check: above_margin 1 0 2021-01-15T18:04:38.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:51: above_margin: local -i check=1 2021-01-15T18:04:38.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:52: above_margin: shift 2021-01-15T18:04:38.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:53: above_margin: local -i target=0 2021-01-15T18:04:38.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:55: above_margin: return 0 2021-01-15T18:04:38.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:114: check: '[' -n '' ']' 2021-01-15T18:04:38.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:218: TEST_backfill_sizeup_out: delete_pool test 2021-01-15T18:04:38.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:04:38.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:04:39.425 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:04:39.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:219: TEST_backfill_sizeup_out: kill_daemons td/osd-backfill-stats 2021-01-15T18:04:39.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:04:39.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:04:39.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:04:39.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:04:39.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:04:39.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:04:39.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-stats.sh:38: run: teardown td/osd-backfill-stats 2021-01-15T18:04:39.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:04:39.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:04:39.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:04:39.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:04:39.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:04:39.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:04:39.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:04:39.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:04:39.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:04:39.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:04:39.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:04:39.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:04:39.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:04:39.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:04:39.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:04:39.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:04:39.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:04:39.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:04:39.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:04:39.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:04:39.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:04:39.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:39.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:04:39.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:04:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:04:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:04:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:04:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-backfill-stats 0 2021-01-15T18:04:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-stats 2021-01-15T18:04:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:04:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-stats KILL 2021-01-15T18:04:39.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:04:39.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:04:39.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:04:39.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:04:39.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:04:39.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:04:39.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:04:39.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:04:39.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:04:39.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:04:39.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:04:39.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:04:39.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:04:39.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:04:39.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:04:39.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:04:39.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-stats 2021-01-15T18:04:39.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:04:39.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:39.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.516267 2021-01-15T18:04:39.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.516267 2021-01-15T18:04:39.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:04:39.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:04:39.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:04:39.587 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:04:39.588 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:04:39.623 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2021-01-15T18:04:39.624 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2021-01-15T18:04:39.684 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:04:39.691 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:04:39.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:04:39.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:04:39.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:04:39.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:04:39.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:04:39.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:04:39.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:04:39.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:04:39.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-bench 2021-01-15T18:04:39.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2021-01-15T18:04:39.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2021-01-15T18:04:39.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2021-01-15T18:04:39.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2021-01-15T18:04:39.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2021-01-15T18:04:39.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2021-01-15T18:04:39.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=8971b86d-2417-4635-8200-0c8037e06c5b --auth-supported=none ' 2021-01-15T18:04:39.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2021-01-15T18:04:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2021-01-15T18:04:39.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2021-01-15T18:04:39.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:04:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2021-01-15T18:04:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2021-01-15T18:04:39.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2021-01-15T18:04:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2021-01-15T18:04:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2021-01-15T18:04:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-01-15T18:04:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:04:39.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-01-15T18:04:39.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:04:39.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:04:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:04:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:04:39.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:04:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:04:39.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:04:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:04:39.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:04:39.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:04:39.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:04:39.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:04:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:04:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:04:39.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:04:39.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:04:39.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:04:39.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-01-15T18:04:39.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:04:39.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:39.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:39.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.677315 2021-01-15T18:04:39.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:04:39.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:04:39.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2021-01-15T18:04:39.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:04:39.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:39.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:39.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.677315 2021-01-15T18:04:39.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:04:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:04:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:04:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:04:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2021-01-15T18:04:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2021-01-15T18:04:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2021-01-15T18:04:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2021-01-15T18:04:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2021-01-15T18:04:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:04:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:04:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:04:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2021-01-15T18:04:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2021-01-15T18:04:39.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:04:39.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:04:39.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:04:39.997 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:04:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:39.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.677315/$cluster-$name.asok' 2021-01-15T18:04:39.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.677315/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:04:40.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:04:40.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:04:40.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:04:40.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:04:40.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:04:40.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:04:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:04:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:04:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:04:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:04:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.677315/ceph-mon.a.asok 2021-01-15T18:04:40.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:04:40.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.677315/ceph-mon.a.asok config get fsid 2021-01-15T18:04:40.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:04:40.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:04:40.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:04:40.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:04:40.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:04:40.276 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:04:40.276 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:04:40.276 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:04:40.277 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:04:40.277 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:40.277 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:40.277 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.677315/ceph-mon.a.asok 2021-01-15T18:04:40.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:04:40.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.677315/ceph-mon.a.asok config get mon_host 2021-01-15T18:04:40.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2021-01-15T18:04:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-bench 2021-01-15T18:04:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:04:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:04:40.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:04:40.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-bench/x 2021-01-15T18:04:40.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:04:41.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:04:41.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:04:41.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:04:41.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:04:41.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:41.262 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:41.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.677315/$cluster-$name.asok' 2021-01-15T18:04:41.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:04:41.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.677315/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:04:41.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2021-01-15T18:04:41.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bench 2021-01-15T18:04:41.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:04:41.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:04:41.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:04:41.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bench/0 2021-01-15T18:04:41.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8971b86d-2417-4635-8200-0c8037e06c5b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2021-01-15T18:04:41.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:04:41.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:04:41.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:04:41.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2021-01-15T18:04:41.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2021-01-15T18:04:41.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:04:41.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:04:41.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2021-01-15T18:04:41.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:04:41.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:04:41.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:04:41.287 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:04:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:04:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:04:41.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.677315/$cluster-$name.asok' 2021-01-15T18:04:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.677315/$cluster-$name.asok' 2021-01-15T18:04:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:04:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:04:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:04:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:04:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:04:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bench/0 2021-01-15T18:04:41.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:04:41.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb 2021-01-15T18:04:41.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb' 2021-01-15T18:04:41.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:04:41.291 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb 2021-01-15T18:04:41.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA52QFggH6nERAA/0SKKRJgqEfKCFYzV6tuuw== 2021-01-15T18:04:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA52QFggH6nERAA/0SKKRJgqEfKCFYzV6tuuw=="}' 2021-01-15T18:04:41.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb -i td/osd-bench/0/new.json 2021-01-15T18:04:41.632 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:04:41.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bench/0/new.json 2021-01-15T18:04:41.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8971b86d-2417-4635-8200-0c8037e06c5b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.677315/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA52QFggH6nERAA/0SKKRJgqEfKCFYzV6tuuw== --osd-uuid 483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb 2021-01-15T18:04:41.665 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.664+0000 7f4b3ed34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:41.667 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.666+0000 7f4b3ed34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:41.668 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.667+0000 7f4b3ed34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:41.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.668+0000 7f4b3ed34080 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-01-15T18:04:41.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.668+0000 7f4b3ed34080 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-01-15T18:04:41.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.668+0000 7f4b3ed34080 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2021-01-15T18:04:41.705 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:41.704+0000 7f4b3ed34080 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2021-01-15T18:04:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bench/0/keyring 2021-01-15T18:04:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:04:46.156 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:04:46.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:04:46.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:04:46.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:04:46.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8971b86d-2417-4635-8200-0c8037e06c5b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.677315/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:04:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:04:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:04:46.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:04:46.677 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:04:46.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:46.694+0000 7fa859363080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:46.721 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:46.721+0000 7fa859363080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:46.723 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:46.722+0000 7fa859363080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:04:47.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:04:47.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:04:47.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:04:47.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:04:47.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:04:47.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:04:47.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:04:47.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:04:47.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:04:47.080 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:04:47.239 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:47.238+0000 7fa859363080 -1 Falling back to public interface 2021-01-15T18:04:47.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:04:48.022 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:04:48.021+0000 7fa859363080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:04:48.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:04:48.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:04:48.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:04:48.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:04:48.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:04:48.467 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:04:48.904 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/678354,v1:127.0.0.1:6803/678354] [v2:127.0.0.1:6804/678354,v1:127.0.0.1:6805/678354] exists,up 483dc1d5-6db3-4aa2-85ea-5a31b5b8aabb 2021-01-15T18:04:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:04:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:04:48.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:04:48.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2021-01-15T18:04:48.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2021-01-15T18:04:48.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2021-01-15T18:04:48.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2021-01-15T18:04:48.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2021-01-15T18:04:48.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2021-01-15T18:04:48.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2021-01-15T18:04:48.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2021-01-15T18:04:48.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2021-01-15T18:04:48.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2021-01-15T18:04:48.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2021-01-15T18:04:48.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2021-01-15T18:04:48.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2021-01-15T18:04:48.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2021-01-15T18:04:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2021-01-15T18:04:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=osd_bench_max_block_size 2021-01-15T18:04:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2021-01-15T18:04:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2021-01-15T18:04:48.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 1024 67108865 2021-01-15T18:04:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2021-01-15T18:04:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2021-01-15T18:04:49.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2021-01-15T18:04:49.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2021-01-15T18:04:49.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2021-01-15T18:04:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2021-01-15T18:04:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2021-01-15T18:04:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2021-01-15T18:04:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2021-01-15T18:04:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=bench_small_size_max_iops 2021-01-15T18:04:49.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2021-01-15T18:04:49.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2021-01-15T18:04:49.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 3072001 1024 2021-01-15T18:04:49.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2021-01-15T18:04:49.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2021-01-15T18:04:49.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2021-01-15T18:04:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2021-01-15T18:04:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2021-01-15T18:04:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2021-01-15T18:04:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2021-01-15T18:04:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1832: expect_failure: local dir=td/osd-bench 2021-01-15T18:04:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: expect_failure: shift 2021-01-15T18:04:49.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1834: expect_failure: local expected=osd_bench_large_size_max_throughput 2021-01-15T18:04:49.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1835: expect_failure: shift 2021-01-15T18:04:49.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1836: expect_failure: local success 2021-01-15T18:04:49.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1838: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2021-01-15T18:04:49.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: expect_failure: success=false 2021-01-15T18:04:49.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: false 2021-01-15T18:04:49.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1844: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2021-01-15T18:04:49.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: expect_failure: return 0 2021-01-15T18:04:49.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2021-01-15T18:06:01.613 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:06:01.613 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_written": 1073741824, 2021-01-15T18:06:01.614 INFO:tasks.workunit.client.0.smithi145.stdout: "blocksize": 4194304, 2021-01-15T18:06:01.614 INFO:tasks.workunit.client.0.smithi145.stdout: "elapsed_sec": 42.769169204999997, 2021-01-15T18:06:01.615 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_per_sec": 25105510.440321401, 2021-01-15T18:06:01.615 INFO:tasks.workunit.client.0.smithi145.stdout: "iops": 5.9856201268008711 2021-01-15T18:06:01.615 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:06:01.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2021-01-15T18:06:03.249 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:06:03.249 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_written": 10, 2021-01-15T18:06:03.250 INFO:tasks.workunit.client.0.smithi145.stdout: "blocksize": 4444, 2021-01-15T18:06:03.250 INFO:tasks.workunit.client.0.smithi145.stdout: "elapsed_sec": 0.24253763, 2021-01-15T18:06:03.250 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_per_sec": 41.230715415170835, 2021-01-15T18:06:03.250 INFO:tasks.workunit.client.0.smithi145.stdout: "iops": 0.0092778387522886672 2021-01-15T18:06:03.250 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:06:03.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2021-01-15T18:06:04.051 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:06:04.052 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_written": 1, 2021-01-15T18:06:04.052 INFO:tasks.workunit.client.0.smithi145.stdout: "blocksize": 14456, 2021-01-15T18:06:04.052 INFO:tasks.workunit.client.0.smithi145.stdout: "elapsed_sec": 0.19896233299999999, 2021-01-15T18:06:04.052 INFO:tasks.workunit.client.0.smithi145.stdout: "bytes_per_sec": 5.0260769710616531, 2021-01-15T18:06:04.052 INFO:tasks.workunit.client.0.smithi145.stdout: "iops": 0.00034768103009557644 2021-01-15T18:06:04.053 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:06:04.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2021-01-15T18:06:04.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-01-15T18:06:04.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:06:04.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-01-15T18:06:04.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:06:04.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:06:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:06:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:06:04.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:06:04.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:06:04.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:06:04.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:06:04.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:06:04.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:06:04.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:06:04.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:06:04.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:06:04.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:06:04.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:06:04.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:06:04.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-01-15T18:06:04.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:06:04.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:06:04.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.677315 2021-01-15T18:06:04.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:06:04.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:06:04.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:06:04.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-bench 0 2021-01-15T18:06:04.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2021-01-15T18:06:04.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:06:04.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2021-01-15T18:06:04.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:06:04.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:06:04.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:06:04.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:06:04.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:06:04.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:06:04.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:06:04.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:06:04.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:06:04.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:06:04.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:06:04.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:06:04.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:06:04.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:06:04.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:06:04.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:06:04.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2021-01-15T18:06:04.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:06:04.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.677315 2021-01-15T18:06:04.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.677315 2021-01-15T18:06:04.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:06:04.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:06:04.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:06:04.209 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:06:04.210 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:06:04.275 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2021-01-15T18:06:04.275 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2021-01-15T18:06:04.336 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:06:04.345 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:06:04.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:06:04.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:06:04.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:06:04.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:06:04.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:06:04.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:06:04.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:06:04.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:06:04.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-bluefs-volume-ops 2021-01-15T18:06:04.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2021-01-15T18:06:04.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-01-15T18:06:04.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-01-15T18:06:04.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: export CEPH_ARGS 2021-01-15T18:06:04.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: uuidgen 2021-01-15T18:06:04.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: CEPH_ARGS+='--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none ' 2021-01-15T18:06:04.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-01-15T18:06:04.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2021-01-15T18:06:04.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:16: run: CEPH_ARGS+='--bluestore_block_db_create=true ' 2021-01-15T18:06:04.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:17: run: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2021-01-15T18:06:04.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:18: run: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2021-01-15T18:06:04.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:19: run: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2021-01-15T18:06:04.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: run: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2021-01-15T18:06:04.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: set 2021-01-15T18:06:04.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:06:04.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:21: run: local funcs=TEST_bluestore 2021-01-15T18:06:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: run: for func in $funcs 2021-01-15T18:06:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: run: setup td/osd-bluefs-volume-ops 2021-01-15T18:06:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2021-01-15T18:06:04.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:06:04.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-01-15T18:06:04.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:06:04.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:06:04.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:06:04.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:06:04.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:06:04.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:06:04.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:06:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:06:04.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:06:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:06:04.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:06:04.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:06:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:06:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:06:04.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:06:04.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:06:04.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:06:04.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-01-15T18:06:04.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:06:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:04.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679134 2021-01-15T18:06:04.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:06:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:06:04.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2021-01-15T18:06:04.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:06:04.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:04.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.679134 2021-01-15T18:06:04.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:06:04.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:06:04.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:06:04.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:06:04.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2021-01-15T18:06:04.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:24: run: TEST_bluestore td/osd-bluefs-volume-ops 2021-01-15T18:06:04.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: ulimit -n 2021-01-15T18:06:04.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: local flimit=4096 2021-01-15T18:06:04.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: '[' 4096 -lt 1536 ']' 2021-01-15T18:06:04.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2021-01-15T18:06:04.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:06:04.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:06:04.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:06:04.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2021-01-15T18:06:04.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:04.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:06:04.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:04.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:04.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:04.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.660 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:04.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:04.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:06:04.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:06:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:06:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:06:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:06:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:06:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:06:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:06:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:06:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:06:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:06:04.837 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.838 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:04.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679134/ceph-mon.a.asok 2021-01-15T18:06:04.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:06:04.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679134/ceph-mon.a.asok config get fsid 2021-01-15T18:06:04.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:06:04.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:06:04.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:06:04.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:06:04.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:06:04.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:06:04.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:06:04.938 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:06:04.938 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:06:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:04.939 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.679134/ceph-mon.a.asok 2021-01-15T18:06:04.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:06:04.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.679134/ceph-mon.a.asok config get mon_host 2021-01-15T18:06:05.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2021-01-15T18:06:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:06:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:06:05.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:06:05.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-bluefs-volume-ops/x 2021-01-15T18:06:05.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:06:05.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:06:05.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:05.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:05.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:05.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:05.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:05.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:05.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:06:05.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:06:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2021-01-15T18:06:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:06:05.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:06:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:06:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-01-15T18:06:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:06:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:06:05.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:06:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:06:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-01-15T18:06:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-01-15T18:06:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:06:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:06:05.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:06:05.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:06:05.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:05.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:05.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:05.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:05.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:05.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:05.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:05.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:06:05.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:06:05.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:06:05.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:06:05.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:06:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:06:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:06:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:06:05.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:06:05.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-01-15T18:06:05.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:06:05.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:06:05.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3' 2021-01-15T18:06:05.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:06:05.954 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:06:05.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCN2QFg3tIbORAAmpAtu712b+oeo3cwnJ/o4w== 2021-01-15T18:06:05.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCN2QFg3tIbORAAmpAtu712b+oeo3cwnJ/o4w=="}' 2021-01-15T18:06:05.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 -i td/osd-bluefs-volume-ops/0/new.json 2021-01-15T18:06:06.294 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:06:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2021-01-15T18:06:06.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCN2QFg3tIbORAAmpAtu712b+oeo3cwnJ/o4w== --osd-uuid 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:06:06.327 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.326+0000 7f8588813080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:06.329 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.328+0000 7f8588813080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:06.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.329+0000 7f8588813080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:06.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.329+0000 7f8588813080 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-01-15T18:06:06.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.329+0000 7f8588813080 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-01-15T18:06:06.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.329+0000 7f8588813080 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2021-01-15T18:06:06.367 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:06.366+0000 7f8588813080 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2021-01-15T18:06:14.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2021-01-15T18:06:14.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:06:14.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:06:14.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:06:14.402 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:06:14.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:06:14.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:14.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:06:14.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:06:14.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:06:14.929 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:06:14.947 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:14.946+0000 7f038165c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:14.964+0000 7f038165c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:14.967 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:14.966+0000 7f038165c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:06:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:06:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:06:15.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:06:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:06:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:06:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:15.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:06:15.325 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:06:15.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:15.983 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:15.982+0000 7f038165c080 -1 Falling back to public interface 2021-01-15T18:06:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:06:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:06:16.723 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:06:17.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:18.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:06:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:18.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:06:18.124 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:06:18.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:19.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:19.041+0000 7f038165c080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:06:19.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:19.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:19.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:06:19.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:19.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:06:19.523 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:06:19.965 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/680542,v1:127.0.0.1:6803/680542] [v2:127.0.0.1:6804/680542,v1:127.0.0.1:6805/680542] exists,up 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:06:19.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:06:19.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:06:19.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:06:19.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-01-15T18:06:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=680542 2021-01-15T18:06:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2021-01-15T18:06:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:06:19.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:06:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:06:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-01-15T18:06:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:06:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:06:19.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:06:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:06:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-01-15T18:06:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-01-15T18:06:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:06:19.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:06:19.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:06:19.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:06:19.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:19.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:19.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:19.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:19.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:19.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:19.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:19.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:06:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:06:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:06:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:06:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:06:19.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:06:19.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:06:19.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:19.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:06:19.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:06:19.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-01-15T18:06:19.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:06:19.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:06:19.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 93be79a4-4607-4171-8cc5-1d56128fbaec' 2021-01-15T18:06:19.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:06:19.977 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:06:19.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCb2QFgevGHOhAAy8GYQFQchJG1itN6F2FXYQ== 2021-01-15T18:06:19.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCb2QFgevGHOhAAy8GYQFQchJG1itN6F2FXYQ=="}' 2021-01-15T18:06:19.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 93be79a4-4607-4171-8cc5-1d56128fbaec -i td/osd-bluefs-volume-ops/1/new.json 2021-01-15T18:06:20.466 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:06:20.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2021-01-15T18:06:20.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCb2QFgevGHOhAAy8GYQFQchJG1itN6F2FXYQ== --osd-uuid 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:06:20.500 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.498+0000 7f8d406c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:20.501 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.500+0000 7f8d406c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.501+0000 7f8d406c9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:20.504 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.502+0000 7f8d406c9080 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-01-15T18:06:20.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.502+0000 7f8d406c9080 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-01-15T18:06:20.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.502+0000 7f8d406c9080 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2021-01-15T18:06:20.539 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:20.537+0000 7f8d406c9080 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2021-01-15T18:06:29.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2021-01-15T18:06:29.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:06:29.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:06:29.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:06:29.074 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:06:29.594 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:06:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:06:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:06:29.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:06:29.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:06:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:29.614+0000 7f521bad2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:29.646 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:29.644+0000 7f521bad2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:29.647 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:29.646+0000 7f521bad2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:29.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:06:29.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:06:29.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:06:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:06:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:06:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:29.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:06:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:29.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:06:29.987 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:06:30.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:30.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:30.669+0000 7f521bad2080 -1 Falling back to public interface 2021-01-15T18:06:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:06:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:31.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:06:31.377 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:06:31.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:06:32.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:32.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:06:32.786 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:06:33.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:33.731+0000 7f521bad2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:06:34.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:34.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:34.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:06:34.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:34.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:06:34.174 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:06:34.582 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/682585,v1:127.0.0.1:6811/682585] [v2:127.0.0.1:6812/682585,v1:127.0.0.1:6813/682585] exists,up 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:06:34.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:06:34.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:06:34.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:06:34.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-01-15T18:06:34.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=682585 2021-01-15T18:06:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2021-01-15T18:06:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:06:34.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:06:34.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:06:34.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-01-15T18:06:34.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:06:34.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:06:34.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:06:34.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:06:34.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-01-15T18:06:34.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-01-15T18:06:34.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:06:34.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:06:34.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:06:34.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:06:34.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:34.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:34.591 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:34.591 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:34.591 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:34.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:34.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:34.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:06:34.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:06:34.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:06:34.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:06:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:06:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:06:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:06:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:34.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:06:34.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:06:34.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-01-15T18:06:34.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:06:34.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:06:34.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 d46bab4e-10e9-487f-868e-560dd20ffd2b' 2021-01-15T18:06:34.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:06:34.596 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:06:34.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCq2QFg8vnLIxAAihxElOgnnc9X/nojE9u8mw== 2021-01-15T18:06:34.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCq2QFg8vnLIxAAihxElOgnnc9X/nojE9u8mw=="}' 2021-01-15T18:06:34.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d46bab4e-10e9-487f-868e-560dd20ffd2b -i td/osd-bluefs-volume-ops/2/new.json 2021-01-15T18:06:35.115 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:06:35.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2021-01-15T18:06:35.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCq2QFg8vnLIxAAihxElOgnnc9X/nojE9u8mw== --osd-uuid d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:06:35.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.149+0000 7f0f23744080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:35.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.151+0000 7f0f23744080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:35.153 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.152+0000 7f0f23744080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:35.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.153+0000 7f0f23744080 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-01-15T18:06:35.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.153+0000 7f0f23744080 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-01-15T18:06:35.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.153+0000 7f0f23744080 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2021-01-15T18:06:35.203 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:35.202+0000 7f0f23744080 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2021-01-15T18:06:43.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2021-01-15T18:06:43.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:06:43.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:06:43.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:06:43.532 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:06:44.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:06:44.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:44.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:06:44.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:06:44.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:06:44.067 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:06:44.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:44.085+0000 7f349f026080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:44.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:44.087+0000 7f349f026080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:44.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:44.087+0000 7f349f026080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:44.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:06:44.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:06:44.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:06:44.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:06:44.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:06:44.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:44.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:06:44.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:44.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:06:44.456 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:06:44.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:45.104+0000 7f349f026080 -1 Falling back to public interface 2021-01-15T18:06:45.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:45.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:45.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:06:45.858 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:06:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:45.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:06:46.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:47.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:06:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:06:47.261 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:06:47.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:48.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:48.240+0000 7f349f026080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:06:48.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:06:48.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:48.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:06:48.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:48.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:06:48.659 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:06:49.063 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/684636,v1:127.0.0.1:6819/684636] [v2:127.0.0.1:6820/684636,v1:127.0.0.1:6821/684636] exists,up d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:06:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:06:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:06:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:06:49.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-01-15T18:06:49.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=684636 2021-01-15T18:06:49.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2021-01-15T18:06:49.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:06:49.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:06:49.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:06:49.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:06:49.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-01-15T18:06:49.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:06:49.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:06:49.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:06:49.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:06:49.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-01-15T18:06:49.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-01-15T18:06:49.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:06:49.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:06:49.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:06:49.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:06:49.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:06:49.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:06:49.075 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:06:49.075 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:06:49.075 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:06:49.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:49.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:06:49.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:06:49.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:06:49.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:06:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:06:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:06:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:06:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:06:49.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:49.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:06:49.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:06:49.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-01-15T18:06:49.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:06:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:06:49.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 2c65a15a-25a8-495b-ac17-67309590a354' 2021-01-15T18:06:49.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:06:49.080 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:06:49.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC52QFgXU3oBBAAvIK/SZjCf7v7fdTtXApELg== 2021-01-15T18:06:49.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC52QFgXU3oBBAAvIK/SZjCf7v7fdTtXApELg=="}' 2021-01-15T18:06:49.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2c65a15a-25a8-495b-ac17-67309590a354 -i td/osd-bluefs-volume-ops/3/new.json 2021-01-15T18:06:49.598 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:06:49.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2021-01-15T18:06:49.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC52QFgXU3oBBAAvIK/SZjCf7v7fdTtXApELg== --osd-uuid 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:06:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.631+0000 7f70415ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:49.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.632+0000 7f70415ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.634+0000 7f70415ad080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.635+0000 7f70415ad080 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-01-15T18:06:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.635+0000 7f70415ad080 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-01-15T18:06:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.635+0000 7f70415ad080 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2021-01-15T18:06:49.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:49.670+0000 7f70415ad080 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2021-01-15T18:06:57.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2021-01-15T18:06:57.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:06:57.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:06:57.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:06:57.926 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:06:58.446 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:06:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:06:58.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:06:58.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:06:58.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:06:58.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:06:58.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:58.465+0000 7f74dc6f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:58.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:58.467+0000 7f74dc6f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:58.468 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:58.468+0000 7f74dc6f0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:06:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:06:58.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:06:58.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:06:58.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:06:58.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:06:58.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:06:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:06:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:06:58.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:06:58.834 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:06:59.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:06:59.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:06:59.484+0000 7f74dc6f0080 -1 Falling back to public interface 2021-01-15T18:07:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:07:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:07:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:07:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:07:00.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:07:00.238 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:07:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:07:01.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:07:01.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:07:01.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:07:01.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:07:01.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:07:01.629 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:07:02.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:07:02.551 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:07:02.550+0000 7f74dc6f0080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:07:03.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:07:03.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:07:03.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:07:03.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:07:03.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:07:03.060 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:07:03.460 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/686679,v1:127.0.0.1:6827/686679] [v2:127.0.0.1:6828/686679,v1:127.0.0.1:6829/686679] exists,up 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:07:03.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:07:03.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:07:03.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:07:03.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-01-15T18:07:03.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=686679 2021-01-15T18:07:03.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2021-01-15T18:07:08.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2021-01-15T18:07:08.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 16 2021-01-15T18:07:09.000 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T18:07:09.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:07:10.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-01-15T18:07:10.039 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:07:10.039 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-01-15T18:07:10.039 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_687593 2021-01-15T18:07:10.039 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:07:10.040 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:07:11.038 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 45 29 0.113269 0.113281 0.326461 0.473469 2021-01-15T18:07:12.039 INFO:tasks.workunit.client.0.smithi145.stdout: 2 16 95 79 0.154279 0.195312 0.291579 0.359074 2021-01-15T18:07:13.039 INFO:tasks.workunit.client.0.smithi145.stdout: 3 16 159 143 0.186176 0.25 0.251296 0.316132 2021-01-15T18:07:14.039 INFO:tasks.workunit.client.0.smithi145.stdout: 4 16 219 203 0.198218 0.234375 0.290855 0.303504 2021-01-15T18:07:15.039 INFO:tasks.workunit.client.0.smithi145.stdout: 5 16 283 267 0.208566 0.25 0.260497 0.293008 2021-01-15T18:07:16.040 INFO:tasks.workunit.client.0.smithi145.stdout: 6 16 347 331 0.215466 0.25 0.276131 0.285904 2021-01-15T18:07:17.039 INFO:tasks.workunit.client.0.smithi145.stdout: 7 16 411 395 0.220394 0.25 0.250515 0.281684 2021-01-15T18:07:18.040 INFO:tasks.workunit.client.0.smithi145.stdout: 8 16 463 447 0.218232 0.203125 0.268308 0.279894 2021-01-15T18:07:19.040 INFO:tasks.workunit.client.0.smithi145.stdout: 9 16 527 511 0.221757 0.25 0.241418 0.276771 2021-01-15T18:07:20.040 INFO:tasks.workunit.client.0.smithi145.stdout: 10 16 587 571 0.223016 0.234375 0.293562 0.276071 2021-01-15T18:07:21.040 INFO:tasks.workunit.client.0.smithi145.stdout: 11 16 651 635 0.225466 0.25 0.262113 0.275113 2021-01-15T18:07:22.040 INFO:tasks.workunit.client.0.smithi145.stdout: 12 16 715 699 0.227507 0.25 0.28252 0.273151 2021-01-15T18:07:23.040 INFO:tasks.workunit.client.0.smithi145.stdout: 13 16 767 751 0.22563 0.203125 0.269748 0.272751 2021-01-15T18:07:24.040 INFO:tasks.workunit.client.0.smithi145.stdout: 14 16 827 811 0.226253 0.234375 0.260316 0.271311 2021-01-15T18:07:25.040 INFO:tasks.workunit.client.0.smithi145.stdout: 15 16 879 863 0.224709 0.203125 0.228215 0.274582 2021-01-15T18:07:26.040 INFO:tasks.workunit.client.0.smithi145.stdout: 16 16 943 927 0.226288 0.25 0.283511 0.272728 2021-01-15T18:07:27.041 INFO:tasks.workunit.client.0.smithi145.stdout: 17 16 1007 991 0.227681 0.25 0.246207 0.27156 2021-01-15T18:07:28.041 INFO:tasks.workunit.client.0.smithi145.stdout: 18 16 1071 1055 0.228918 0.25 0.266795 0.270978 2021-01-15T18:07:29.041 INFO:tasks.workunit.client.0.smithi145.stdout: 19 16 1131 1115 0.229204 0.234375 0.283767 0.270975 2021-01-15T18:07:30.041 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:07:30.041612+0000 min lat: 0.225436 max lat: 0.904001 avg lat: 0.270261 2021-01-15T18:07:30.042 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:07:30.042 INFO:tasks.workunit.client.0.smithi145.stdout: 20 16 1195 1179 0.230242 0.25 0.274873 0.270261 2021-01-15T18:07:31.041 INFO:tasks.workunit.client.0.smithi145.stdout: 21 16 1259 1243 0.231182 0.25 0.268843 0.269765 2021-01-15T18:07:32.042 INFO:tasks.workunit.client.0.smithi145.stdout: 22 16 1312 1296 0.230083 0.207031 0.28283 0.269176 2021-01-15T18:07:33.042 INFO:tasks.workunit.client.0.smithi145.stdout: 23 16 1376 1360 0.230947 0.25 0.251758 0.269063 2021-01-15T18:07:34.042 INFO:tasks.workunit.client.0.smithi145.stdout: 24 16 1435 1419 0.230927 0.230469 0.277505 0.269405 2021-01-15T18:07:35.042 INFO:tasks.workunit.client.0.smithi145.stdout: 25 16 1499 1483 0.231688 0.25 0.258193 0.269001 2021-01-15T18:07:36.042 INFO:tasks.workunit.client.0.smithi145.stdout: 26 16 1552 1536 0.230739 0.207031 0.29628 0.26916 2021-01-15T18:07:37.042 INFO:tasks.workunit.client.0.smithi145.stdout: 27 16 1616 1600 0.231451 0.25 0.266923 0.268857 2021-01-15T18:07:38.042 INFO:tasks.workunit.client.0.smithi145.stdout: 28 16 1675 1659 0.231415 0.230469 0.270356 0.2687 2021-01-15T18:07:39.042 INFO:tasks.workunit.client.0.smithi145.stdout: 29 16 1739 1723 0.232055 0.25 0.252976 0.268363 2021-01-15T18:07:40.042 INFO:tasks.workunit.client.0.smithi145.stdout: 30 16 1787 1771 0.230569 0.1875 0.241071 0.267959 2021-01-15T18:07:41.042 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 30.2243 2021-01-15T18:07:41.043 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 1787 2021-01-15T18:07:41.043 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:07:41.043 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:07:41.044 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.230955 2021-01-15T18:07:41.044 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0.0301267 2021-01-15T18:07:41.044 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.25 2021-01-15T18:07:41.044 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0.113281 2021-01-15T18:07:41.044 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 59 2021-01-15T18:07:41.045 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 7.71243 2021-01-15T18:07:41.045 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 64 2021-01-15T18:07:41.045 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 29 2021-01-15T18:07:41.045 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.270193 2021-01-15T18:07:41.046 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 0.0501283 2021-01-15T18:07:41.046 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 0.904001 2021-01-15T18:07:41.046 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.160199 2021-01-15T18:07:41.158 INFO:tasks.workunit.client.0.smithi145.stdout:after bench 2021-01-15T18:07:41.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2021-01-15T18:07:41.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 680542 2021-01-15T18:07:41.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2021-01-15T18:07:42.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 680542 2021-01-15T18:07:42.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (680542) - No such process 2021-01-15T18:07:42.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2021-01-15T18:07:42.578 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:07:42.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 682585 2021-01-15T18:07:42.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2021-01-15T18:07:43.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 682585 2021-01-15T18:07:43.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (682585) - No such process 2021-01-15T18:07:43.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2021-01-15T18:07:44.023 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T18:07:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 684636 2021-01-15T18:07:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2021-01-15T18:07:45.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 684636 2021-01-15T18:07:45.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (684636) - No such process 2021-01-15T18:07:45.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2021-01-15T18:07:45.446 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T18:07:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 686679 2021-01-15T18:07:45.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2021-01-15T18:07:46.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 686679 2021-01-15T18:07:46.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (686679) - No such process 2021-01-15T18:07:46.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2021-01-15T18:07:46.988 INFO:tasks.workunit.client.0.smithi145.stderr:osd.3 is already down. 2021-01-15T18:07:46.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:07:48.779 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:07:48.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:07:50.567 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:07:50.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:07:52.354 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:07:52.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:07:54.142 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:07:54.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2021-01-15T18:07:54.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2021-01-15T18:07:54.176 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:07:55.026 INFO:tasks.workunit.client.0.smithi145.stdout:0 : device size 0x1ffff000 : using 0xb00000(11 MiB) 2021-01-15T18:07:55.026 INFO:tasks.workunit.client.0.smithi145.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-01-15T18:07:55.027 INFO:tasks.workunit.client.0.smithi145.stdout:2 : device size 0x100000000 : using 0x80555000(2.0 GiB) 2021-01-15T18:07:55.027 INFO:tasks.workunit.client.0.smithi145.stdout:Expanding DB/WAL... 2021-01-15T18:07:55.027 INFO:tasks.workunit.client.0.smithi145.stdout:0 : expanding to 0x536870912 2021-01-15T18:07:55.045 INFO:tasks.workunit.client.0.smithi145.stdout:0 : size label updated to 536870912 2021-01-15T18:07:55.045 INFO:tasks.workunit.client.0.smithi145.stdout:1 : expanding to 0x1073741824 2021-01-15T18:07:55.070 INFO:tasks.workunit.client.0.smithi145.stdout:1 : size label updated to 1073741824 2021-01-15T18:07:55.070 INFO:tasks.workunit.client.0.smithi145.stdout:2 : expanding from 0x80000000 to 0x100000000 2021-01-15T18:07:55.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:07:55.254+0000 7f532c0a33c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-01-15T18:07:55.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:07:55.254+0000 7f532c0a33c0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2021-01-15T18:08:00.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2021-01-15T18:08:00.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2021-01-15T18:08:00.073 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:00.934 INFO:tasks.workunit.client.0.smithi145.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2021-01-15T18:08:00.934 INFO:tasks.workunit.client.0.smithi145.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-01-15T18:08:00.934 INFO:tasks.workunit.client.0.smithi145.stdout:2 : device size 0x101000000 : using 0x81594000(2.0 GiB) 2021-01-15T18:08:00.935 INFO:tasks.workunit.client.0.smithi145.stdout:Expanding DB/WAL... 2021-01-15T18:08:00.935 INFO:tasks.workunit.client.0.smithi145.stdout:0 : expanding to 0x536870912 2021-01-15T18:08:00.958 INFO:tasks.workunit.client.0.smithi145.stdout:0 : size label updated to 536870912 2021-01-15T18:08:00.958 INFO:tasks.workunit.client.0.smithi145.stdout:1 : expanding to 0x1073741824 2021-01-15T18:08:00.983 INFO:tasks.workunit.client.0.smithi145.stdout:1 : size label updated to 1073741824 2021-01-15T18:08:00.983 INFO:tasks.workunit.client.0.smithi145.stdout:2 : expanding from 0x80000000 to 0x101000000 2021-01-15T18:08:01.168 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:01.167+0000 7fd5fc5d03c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-01-15T18:08:01.168 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:01.167+0000 7fd5fc5d03c0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2021-01-15T18:08:05.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2021-01-15T18:08:05.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2021-01-15T18:08:05.974 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:06.829 INFO:tasks.workunit.client.0.smithi145.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2021-01-15T18:08:06.829 INFO:tasks.workunit.client.0.smithi145.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-01-15T18:08:06.829 INFO:tasks.workunit.client.0.smithi145.stdout:2 : device size 0x100020000 : using 0x80594000(2.0 GiB) 2021-01-15T18:08:06.830 INFO:tasks.workunit.client.0.smithi145.stdout:Expanding DB/WAL... 2021-01-15T18:08:06.830 INFO:tasks.workunit.client.0.smithi145.stdout:0 : expanding to 0x536870912 2021-01-15T18:08:06.846 INFO:tasks.workunit.client.0.smithi145.stdout:0 : size label updated to 536870912 2021-01-15T18:08:06.846 INFO:tasks.workunit.client.0.smithi145.stdout:1 : expanding to 0x1073741824 2021-01-15T18:08:06.871 INFO:tasks.workunit.client.0.smithi145.stdout:1 : size label updated to 1073741824 2021-01-15T18:08:06.871 INFO:tasks.workunit.client.0.smithi145.stdout:2 : expanding from 0x80000000 to 0x100020000 2021-01-15T18:08:07.055 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:07.054+0000 7f501337d3c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-01-15T18:08:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:07.054+0000 7f501337d3c0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2021-01-15T18:08:11.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2021-01-15T18:08:11.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2021-01-15T18:08:11.597 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:12.451 INFO:tasks.workunit.client.0.smithi145.stdout:0 : device size 0x1ffff000 : using 0xc00000(12 MiB) 2021-01-15T18:08:12.452 INFO:tasks.workunit.client.0.smithi145.stdout:1 : device size 0x3fffe000 : using 0x900000(9 MiB) 2021-01-15T18:08:12.452 INFO:tasks.workunit.client.0.smithi145.stdout:2 : device size 0xfff00000 : using 0x804a9000(2.0 GiB) 2021-01-15T18:08:12.452 INFO:tasks.workunit.client.0.smithi145.stdout:Expanding DB/WAL... 2021-01-15T18:08:12.453 INFO:tasks.workunit.client.0.smithi145.stdout:0 : expanding to 0x536870912 2021-01-15T18:08:12.475 INFO:tasks.workunit.client.0.smithi145.stdout:0 : size label updated to 536870912 2021-01-15T18:08:12.475 INFO:tasks.workunit.client.0.smithi145.stdout:1 : expanding to 0x1073741824 2021-01-15T18:08:12.500 INFO:tasks.workunit.client.0.smithi145.stdout:1 : size label updated to 1073741824 2021-01-15T18:08:12.500 INFO:tasks.workunit.client.0.smithi145.stdout:2 : expanding from 0x80000000 to 0xfff00000 2021-01-15T18:08:12.676 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:12.676+0000 7fb10fc5f3c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-01-15T18:08:12.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:12.676+0000 7fb10fc5f3c0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2021-01-15T18:08:17.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:08:19.285 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:19.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:08:21.073 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:21.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:08:22.865 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:22.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:08:24.651 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:24.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2021-01-15T18:08:24.666 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:26.445 INFO:tasks.workunit.client.0.smithi145.stdout:0 : device size 0x1ffff000 : using 0x500000(5 MiB) 2021-01-15T18:08:26.446 INFO:tasks.workunit.client.0.smithi145.stdout:1 : device size 0x3fffe000 : using 0xf00000(15 MiB) 2021-01-15T18:08:26.446 INFO:tasks.workunit.client.0.smithi145.stdout:2 : device size 0x100000000 : using 0x555000(5.3 MiB) 2021-01-15T18:08:26.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2021-01-15T18:08:26.461 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:27.981 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-01-15T18:08:27.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:08:29.267 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:29.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-01-15T18:08:29.283 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:31.054 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-01-15T18:08:31.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:08:32.340 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:32.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2021-01-15T18:08:32.355 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:34.125 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-01-15T18:08:34.125 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-01-15T18:08:34.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:08:34.911 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:34.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2021-01-15T18:08:34.926 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:34.931 INFO:tasks.workunit.client.0.smithi145.stderr:Migrate to WAL device isn't supported. 2021-01-15T18:08:34.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2021-01-15T18:08:34.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:08:36.716 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:36.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-01-15T18:08:36.733 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:38.254 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-01-15T18:08:38.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:08:39.543 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:39.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2021-01-15T18:08:39.581 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:08:40.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:08:42.115 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:08:42.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-01-15T18:08:42.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:08:42.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:08:42.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:08:42.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:08:42.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-01-15T18:08:42.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:08:42.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:08:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:08:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:08:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-01-15T18:08:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-01-15T18:08:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:08:42.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:08:42.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:08:42.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:08:42.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:08:42.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:08:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:08:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:08:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:08:42.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:42.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:08:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:08:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:08:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:08:42.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:08:42.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:42.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:08:42.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:08:42.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-01-15T18:08:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:08:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:42.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-01-15T18:08:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:08:42.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:08:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:08:42.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:08:42.129 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:08:42.144 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:42.143+0000 7f543535a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:42.146+0000 7f543535a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:42.149 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:42.148+0000 7f543535a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:08:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:08:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:08:42.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:08:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:08:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:08:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:42.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:08:42.525 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:08:42.920 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:42.919+0000 7f543535a080 -1 Falling back to public interface 2021-01-15T18:08:42.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:43.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:43.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:43.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:08:43.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:43.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:08:43.929 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:08:44.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:45.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:45.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:45.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:08:45.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:45.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:08:45.329 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:08:45.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:45.834 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:45.833+0000 7f543535a080 -1 osd.0 28 log_to_monitors {default=true} 2021-01-15T18:08:46.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:46.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:46.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:08:46.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:46.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:08:46.731 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:08:47.122 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/698983,v1:127.0.0.1:6803/698983] [v2:127.0.0.1:6804/698983,v1:127.0.0.1:6805/698983] exists,up 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:08:47.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:08:47.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:08:47.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:08:47.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-01-15T18:08:47.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=698983 2021-01-15T18:08:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-01-15T18:08:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:08:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:08:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T18:08:47.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:08:47.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-01-15T18:08:47.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:08:47.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:08:47.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:08:47.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:08:47.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-01-15T18:08:47.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-01-15T18:08:47.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:08:47.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:08:47.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:08:47.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:08:47.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:08:47.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:08:47.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:08:47.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:08:47.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:08:47.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:47.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:47.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:08:47.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:08:47.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:08:47.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:08:47.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:08:47.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:47.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:08:47.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:08:47.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-01-15T18:08:47.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T18:08:47.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:47.136 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:08:47.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-01-15T18:08:47.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T18:08:47.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:08:47.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:08:47.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:08:47.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:47.146+0000 7fc53f245080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:47.148 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:47.147+0000 7fc53f245080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:47.150 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:47.149+0000 7fc53f245080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:47.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T18:08:47.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:08:47.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:08:47.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:08:47.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:08:47.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:47.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:08:47.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:47.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:08:47.539 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:08:47.915 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:47.915+0000 7fc53f245080 -1 Falling back to public interface 2021-01-15T18:08:47.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:48.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:08:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:48.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:08:48.944 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:08:49.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:50.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:50.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:50.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:08:50.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:50.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:08:50.336 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:08:50.409 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:50.408+0000 7fc53f245080 -1 osd.1 31 log_to_monitors {default=true} 2021-01-15T18:08:50.744 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 40 up_thru 29 down_at 32 last_clean_interval [10,31) [v2:127.0.0.1:6810/699874,v1:127.0.0.1:6811/699874] [v2:127.0.0.1:6812/699874,v1:127.0.0.1:6813/699874] exists,up 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:08:50.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:08:50.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:08:50.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:08:50.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-01-15T18:08:50.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=699874 2021-01-15T18:08:50.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-01-15T18:08:50.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:08:50.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:08:50.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T18:08:50.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:08:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-01-15T18:08:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:08:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:08:50.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:08:50.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:08:50.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-01-15T18:08:50.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-01-15T18:08:50.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:08:50.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:08:50.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:08:50.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:08:50.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:08:50.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:08:50.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:08:50.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:08:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:08:50.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:08:50.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:08:50.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:08:50.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:08:50.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:08:50.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:50.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:08:50.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:08:50.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-01-15T18:08:50.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T18:08:50.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:50.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-01-15T18:08:50.758 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:08:50.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T18:08:50.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:08:50.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:08:50.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:08:50.768 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:50.768+0000 7f9becd0a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:50.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:50.769+0000 7f9becd0a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:50.771 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:50.770+0000 7f9becd0a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:51.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T18:08:51.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:08:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:08:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:08:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:08:51.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:08:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:51.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:08:51.149 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:08:51.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:51.292+0000 7f9becd0a080 -1 Falling back to public interface 2021-01-15T18:08:51.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:52.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:52.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:52.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:08:52.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:52.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:08:52.567 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:08:52.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:53.592 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:53.591+0000 7f9becd0a080 -1 osd.2 34 log_to_monitors {default=true} 2021-01-15T18:08:53.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:53.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:53.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:08:53.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:53.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:08:53.956 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:08:54.453 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 42 up_thru 42 down_at 35 last_clean_interval [15,34) [v2:127.0.0.1:6818/700739,v1:127.0.0.1:6819/700739] [v2:127.0.0.1:6820/700739,v1:127.0.0.1:6821/700739] exists,up d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:08:54.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:08:54.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:08:54.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:08:54.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-01-15T18:08:54.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=700739 2021-01-15T18:08:54.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-01-15T18:08:54.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:08:54.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:08:54.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2021-01-15T18:08:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:08:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-01-15T18:08:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:08:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:08:54.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:08:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:08:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-01-15T18:08:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-01-15T18:08:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:08:54.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:08:54.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:08:54.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:08:54.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:08:54.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:08:54.461 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:08:54.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:08:54.462 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:08:54.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:54.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:08:54.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:08:54.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:08:54.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:08:54.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:08:54.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:08:54.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:54.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:08:54.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:08:54.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-01-15T18:08:54.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2021-01-15T18:08:54.465 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:08:54.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:08:54.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-01-15T18:08:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2021-01-15T18:08:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:08:54.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:08:54.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:08:54.477 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:54.476+0000 7f1ebe3bd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:54.479 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:54.478+0000 7f1ebe3bd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:54.481 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:54.480+0000 7f1ebe3bd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:08:54.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2021-01-15T18:08:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:08:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:08:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:08:54.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:08:54.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:54.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:08:54.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:54.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:08:54.892 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:08:55.246 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:55.245+0000 7f1ebe3bd080 -1 Falling back to public interface 2021-01-15T18:08:55.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:08:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:08:56.288 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:08:56.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:57.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:08:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:08:57.677 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:08:58.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:08:58.269 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:08:58.268+0000 7f1ebe3bd080 -1 osd.3 36 log_to_monitors {default=true} 2021-01-15T18:08:59.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:08:59.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:08:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:08:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:08:59.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:08:59.085 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:08:59.553 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 45 up_thru 45 down_at 37 last_clean_interval [20,36) [v2:127.0.0.1:6826/701594,v1:127.0.0.1:6827/701594] [v2:127.0.0.1:6828/701594,v1:127.0.0.1:6829/701594] exists,up 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:08:59.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:08:59.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:08:59.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:08:59.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-01-15T18:08:59.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=701594 2021-01-15T18:08:59.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2021-01-15T18:08:59.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:08:59.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:08:59.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:08:59.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:08:59.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:08:59.557 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:08:59.558 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:08:59.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:08:59.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:08:59.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:08:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:08:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:08:59.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:08:59.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:08:59.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:09:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:09:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:09:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:09:00.064 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:09:00.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:09:00.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:09:00.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:09:00.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=163208757253 2021-01-15T18:09:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 163208757253 2021-01-15T18:09:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-163208757253' 2021-01-15T18:09:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:09:00.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:09:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=171798691844 2021-01-15T18:09:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 171798691844 2021-01-15T18:09:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-163208757253 1-171798691844' 2021-01-15T18:09:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:09:00.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:09:00.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=180388626436 2021-01-15T18:09:00.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 180388626436 2021-01-15T18:09:00.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-163208757253 1-171798691844 2-180388626436' 2021-01-15T18:09:00.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:09:00.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:09:00.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=193273528323 2021-01-15T18:09:00.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 193273528323 2021-01-15T18:09:00.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-163208757253 1-171798691844 2-180388626436 3-193273528323' 2021-01-15T18:09:00.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:09:00.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-163208757253 2021-01-15T18:09:00.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:09:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:09:00.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-163208757253 2021-01-15T18:09:00.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:09:00.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=163208757253 2021-01-15T18:09:00.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 163208757253' 2021-01-15T18:09:00.611 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 163208757253 2021-01-15T18:09:00.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:09:01.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757251 -lt 163208757253 2021-01-15T18:09:01.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:09:02.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:09:02.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:09:02.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 163208757254 -lt 163208757253 2021-01-15T18:09:02.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:09:02.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-171798691844 2021-01-15T18:09:02.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:09:02.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:09:02.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-171798691844 2021-01-15T18:09:02.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:09:02.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=171798691844 2021-01-15T18:09:02.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 171798691844' 2021-01-15T18:09:02.423 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 171798691844 2021-01-15T18:09:02.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:09:02.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 171798691845 -lt 171798691844 2021-01-15T18:09:02.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:09:02.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-180388626436 2021-01-15T18:09:02.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:09:02.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:09:02.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-180388626436 2021-01-15T18:09:02.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:09:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=180388626436 2021-01-15T18:09:02.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 180388626436' 2021-01-15T18:09:02.874 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 180388626436 2021-01-15T18:09:02.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:09:03.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 180388626436 -lt 180388626436 2021-01-15T18:09:03.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:09:03.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-193273528323 2021-01-15T18:09:03.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:09:03.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:09:03.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-193273528323 2021-01-15T18:09:03.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:09:03.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=193273528323 2021-01-15T18:09:03.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 193273528323' 2021-01-15T18:09:03.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:09:03.265 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 193273528323 2021-01-15T18:09:03.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 193273528323 -lt 193273528323 2021-01-15T18:09:03.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:09:03.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:09:03.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:09:04.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2021-01-15T18:09:04.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:09:04.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:09:04.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:09:04.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:09:04.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:09:04.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:09:04.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:09:04.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2021-01-15T18:09:04.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:09:04.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:09:04.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:09:05.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2021-01-15T18:09:05.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:09:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:09:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-01-15T18:09:05.129 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:09:05.129 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-01-15T18:09:05.129 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_702917 2021-01-15T18:09:05.130 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:09:05.130 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:09:06.129 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 30 14 0.0546812 0.0546875 0.12567 0.221843 2021-01-15T18:09:07.129 INFO:tasks.workunit.client.0.smithi145.stdout: 2 16 30 14 0.0273406 0 - 0.221843 2021-01-15T18:09:08.129 INFO:tasks.workunit.client.0.smithi145.stdout: 3 16 46 30 0.0390576 0.03125 0.270074 0.669678 2021-01-15T18:09:09.129 INFO:tasks.workunit.client.0.smithi145.stdout: 4 16 60 44 0.0429629 0.0546875 0.168038 0.54165 2021-01-15T18:09:10.129 INFO:tasks.workunit.client.0.smithi145.stdout: 5 16 64 48 0.0374953 0.015625 0.117118 0.506117 2021-01-15T18:09:11.130 INFO:tasks.workunit.client.0.smithi145.stdout: 6 16 80 64 0.0416613 0.0625 2.26826 1.48971 2021-01-15T18:09:12.130 INFO:tasks.workunit.client.0.smithi145.stdout: 7 16 130 114 0.0636082 0.195312 0.26816 0.958718 2021-01-15T18:09:13.130 INFO:tasks.workunit.client.0.smithi145.stdout: 8 16 192 176 0.0859268 0.242188 0.305653 0.723088 2021-01-15T18:09:14.130 INFO:tasks.workunit.client.0.smithi145.stdout: 9 16 242 226 0.0980779 0.195312 0.317271 0.625824 2021-01-15T18:09:15.130 INFO:tasks.workunit.client.0.smithi145.stdout: 10 16 304 288 0.112486 0.242188 0.249572 0.551503 2021-01-15T18:09:16.130 INFO:tasks.workunit.client.0.smithi145.stdout: 11 16 355 339 0.120369 0.199219 0.293974 0.511186 2021-01-15T18:09:17.131 INFO:tasks.workunit.client.0.smithi145.stdout: 12 16 416 400 0.130192 0.238281 0.268817 0.475907 2021-01-15T18:09:18.130 INFO:tasks.workunit.client.0.smithi145.stdout: 13 16 467 451 0.1355 0.199219 0.251703 0.45213 2021-01-15T18:09:19.131 INFO:tasks.workunit.client.0.smithi145.stdout: 14 16 528 512 0.142839 0.238281 0.277674 0.433949 2021-01-15T18:09:20.131 INFO:tasks.workunit.client.0.smithi145.stdout: 15 16 592 576 0.149981 0.25 0.286879 0.415877 2021-01-15T18:09:21.131 INFO:tasks.workunit.client.0.smithi145.stdout: 16 16 643 627 0.153057 0.199219 0.268726 0.404277 2021-01-15T18:09:22.131 INFO:tasks.workunit.client.0.smithi145.stdout: 17 16 704 688 0.158068 0.238281 0.277724 0.3925 2021-01-15T18:09:23.131 INFO:tasks.workunit.client.0.smithi145.stdout: 18 16 755 739 0.160353 0.199219 0.258799 0.384531 2021-01-15T18:09:24.131 INFO:tasks.workunit.client.0.smithi145.stdout: 19 16 816 800 0.164452 0.238281 0.336388 0.378084 2021-01-15T18:09:25.131 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:09:25.131985+0000 min lat: 0.100711 max lat: 5.97741 avg lat: 0.369483 2021-01-15T18:09:25.132 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:09:25.132 INFO:tasks.workunit.client.0.smithi145.stdout: 20 16 880 864 0.168728 0.25 0.266401 0.369483 2021-01-15T18:09:26.132 INFO:tasks.workunit.client.0.smithi145.stdout: 21 16 931 915 0.170179 0.199219 0.299136 0.364268 2021-01-15T18:09:27.132 INFO:tasks.workunit.client.0.smithi145.stdout: 22 16 992 976 0.173273 0.238281 0.273031 0.358397 2021-01-15T18:09:28.132 INFO:tasks.workunit.client.0.smithi145.stdout: 23 16 1043 1027 0.1744 0.199219 0.293396 0.354968 2021-01-15T18:09:29.132 INFO:tasks.workunit.client.0.smithi145.stdout: 24 16 1104 1088 0.17706 0.238281 0.276486 0.352176 2021-01-15T18:09:30.132 INFO:tasks.workunit.client.0.smithi145.stdout: 25 16 1155 1139 0.177945 0.199219 0.30023 0.349192 2021-01-15T18:09:31.132 INFO:tasks.workunit.client.0.smithi145.stdout: 26 16 1203 1187 0.178311 0.1875 0.281344 0.347301 2021-01-15T18:09:32.132 INFO:tasks.workunit.client.0.smithi145.stdout: 27 16 1264 1248 0.180531 0.238281 0.309328 0.343894 2021-01-15T18:09:33.133 INFO:tasks.workunit.client.0.smithi145.stdout: 28 16 1328 1312 0.183011 0.25 0.274996 0.340987 2021-01-15T18:09:34.133 INFO:tasks.workunit.client.0.smithi145.stdout: 29 16 1376 1360 0.183165 0.1875 0.302449 0.340374 2021-01-15T18:09:35.133 INFO:tasks.workunit.client.0.smithi145.stdout: 30 16 1427 1411 0.1837 0.199219 0.276813 0.338314 2021-01-15T18:09:36.133 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 30.2105 2021-01-15T18:09:36.134 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 1427 2021-01-15T18:09:36.135 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:09:36.135 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:09:36.136 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.184512 2021-01-15T18:09:36.136 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0.0779678 2021-01-15T18:09:36.137 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.25 2021-01-15T18:09:36.137 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0 2021-01-15T18:09:36.137 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 47 2021-01-15T18:09:36.138 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 19.9598 2021-01-15T18:09:36.138 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 64 2021-01-15T18:09:36.139 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 0 2021-01-15T18:09:36.139 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.337491 2021-01-15T18:09:36.140 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 0.489081 2021-01-15T18:09:36.140 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 5.97741 2021-01-15T18:09:36.141 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.100711 2021-01-15T18:09:36.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 698983 2021-01-15T18:09:36.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2021-01-15T18:09:37.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 698983 2021-01-15T18:09:37.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (698983) - No such process 2021-01-15T18:09:37.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2021-01-15T18:09:37.726 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:09:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 699874 2021-01-15T18:09:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2021-01-15T18:09:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 699874 2021-01-15T18:09:38.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (699874) - No such process 2021-01-15T18:09:38.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2021-01-15T18:09:39.148 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T18:09:39.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 700739 2021-01-15T18:09:39.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2021-01-15T18:09:40.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 700739 2021-01-15T18:09:40.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (700739) - No such process 2021-01-15T18:09:40.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2021-01-15T18:09:40.570 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T18:09:40.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 701594 2021-01-15T18:09:40.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2021-01-15T18:09:41.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 701594 2021-01-15T18:09:41.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (701594) - No such process 2021-01-15T18:09:41.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2021-01-15T18:09:42.171 INFO:tasks.workunit.client.0.smithi145.stderr:osd.3 is already down. 2021-01-15T18:09:42.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:09:43.462 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:09:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2021-01-15T18:09:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:512+0 records in 2021-01-15T18:09:45.061 INFO:tasks.workunit.client.0.smithi145.stderr:512+0 records out 2021-01-15T18:09:45.062 INFO:tasks.workunit.client.0.smithi145.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.248495 s, 2.2 GB/s 2021-01-15T18:09:45.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2021-01-15T18:09:45.062 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:09:48.249 INFO:tasks.workunit.client.0.smithi145.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2021-01-15T18:09:51.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:09:51.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:09:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2021-01-15T18:09:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:09:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:09:51.965 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.484651 s, 2.2 GB/s 2021-01-15T18:09:51.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2021-01-15T18:09:51.966 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:09:51.966 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:09:51.966 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:09:59.594 INFO:tasks.workunit.client.0.smithi145.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2021-01-15T18:09:59.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2021-01-15T18:09:59.610 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:01.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:10:03.166 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:10:03.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:10:03.950 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:10:03.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2021-01-15T18:10:03.967 INFO:tasks.workunit.client.0.smithi145.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-01-15T18:10:03.967 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:04.736 INFO:tasks.workunit.client.0.smithi145.stdout:DB device added 2021-01-15T18:10:04.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2021-01-15T18:10:04.751 INFO:tasks.workunit.client.0.smithi145.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-01-15T18:10:04.752 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:06.021 INFO:tasks.workunit.client.0.smithi145.stdout:WAL device added 2021-01-15T18:10:06.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2021-01-15T18:10:06.035 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:07.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:10:09.603 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:10:09.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:10:10.893 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:10:10.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2021-01-15T18:10:10.909 INFO:tasks.workunit.client.0.smithi145.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2021-01-15T18:10:10.910 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:12.179 INFO:tasks.workunit.client.0.smithi145.stdout:WAL device added 2021-01-15T18:10:12.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2021-01-15T18:10:12.197 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:10:13.967 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2021-01-15T18:10:13.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:10:15.254 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:10:15.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-01-15T18:10:15.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:10:15.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:10:15.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:10:15.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:10:15.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-01-15T18:10:15.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:10:15.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:10:15.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:10:15.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:10:15.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-01-15T18:10:15.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-01-15T18:10:15.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:10:15.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:10:15.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:10:15.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:10:15.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:10:15.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:10:15.264 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:10:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:10:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:10:15.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:15.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:10:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:10:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:10:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:10:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:10:15.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:10:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:10:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-01-15T18:10:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:10:15.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:15.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-01-15T18:10:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:10:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:10:15.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:10:15.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:10:15.269 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:10:15.284 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:15.283+0000 7fbd8b572080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:15.286 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:15.285+0000 7fbd8b572080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:15.288 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:15.288+0000 7fbd8b572080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:15.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:10:15.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:10:15.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:10:15.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:10:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:10:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:10:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:15.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:10:15.660 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:10:16.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:16.304+0000 7fbd8b572080 -1 Falling back to public interface 2021-01-15T18:10:17.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:17.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:17.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:10:17.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:17.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:10:17.057 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:10:17.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:18.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:10:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:18.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:10:18.459 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:10:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:19.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:19.678+0000 7fbd8b572080 -1 osd.0 47 log_to_monitors {default=true} 2021-01-15T18:10:19.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:19.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:19.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:10:19.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:19.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:10:19.861 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:10:20.275 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 57 up_thru 57 down_at 48 last_clean_interval [38,47) [v2:127.0.0.1:6802/708669,v1:127.0.0.1:6803/708669] [v2:127.0.0.1:6804/708669,v1:127.0.0.1:6805/708669] exists,up 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:10:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:10:20.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:10:20.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:10:20.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-01-15T18:10:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=708669 2021-01-15T18:10:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-01-15T18:10:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:10:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:10:20.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T18:10:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:10:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-01-15T18:10:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:10:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:10:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:10:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:10:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-01-15T18:10:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-01-15T18:10:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:10:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:10:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:10:20.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:10:20.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:10:20.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:10:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:10:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:10:20.283 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:10:20.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:10:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:10:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:10:20.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:10:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:10:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:10:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:10:20.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-01-15T18:10:20.287 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:10:20.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T18:10:20.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:20.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-01-15T18:10:20.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T18:10:20.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:10:20.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:10:20.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:10:20.299 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:20.298+0000 7f80e53fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:20.300 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:20.300+0000 7f80e53fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:20.302 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:20.301+0000 7f80e53fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:20.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T18:10:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:10:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:10:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:10:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:10:20.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:10:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:20.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:10:20.677 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:10:21.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:21.325 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:21.325+0000 7f80e53fa080 -1 Falling back to public interface 2021-01-15T18:10:22.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:22.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:22.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:10:22.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:22.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:10:22.078 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:10:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:10:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:23.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:10:23.476 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:10:23.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:24.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:24.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:24.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:10:24.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:24.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:10:24.891 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:10:24.928 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:24.927+0000 7f80e53fa080 -1 osd.1 50 log_to_monitors {default=true} 2021-01-15T18:10:25.325 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 59 up_thru 48 down_at 51 last_clean_interval [40,50) [v2:127.0.0.1:6810/709570,v1:127.0.0.1:6811/709570] [v2:127.0.0.1:6812/709570,v1:127.0.0.1:6813/709570] exists,up 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:10:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:10:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:10:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:10:25.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-01-15T18:10:25.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=709570 2021-01-15T18:10:25.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-01-15T18:10:25.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:10:25.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:10:25.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T18:10:25.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:10:25.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-01-15T18:10:25.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:10:25.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:10:25.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:10:25.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:10:25.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-01-15T18:10:25.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-01-15T18:10:25.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:10:25.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:10:25.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:10:25.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:10:25.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:10:25.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:10:25.335 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:10:25.335 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:10:25.335 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:10:25.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:25.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:25.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:10:25.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:10:25.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:10:25.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:10:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:10:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:10:25.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:10:25.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-01-15T18:10:25.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T18:10:25.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:25.338 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:10:25.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-01-15T18:10:25.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T18:10:25.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:10:25.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:10:25.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:10:25.350 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:25.349+0000 7fb7a03ba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:25.351 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:25.351+0000 7fb7a03ba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:25.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:25.353+0000 7fb7a03ba080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:25.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T18:10:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:10:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:10:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:10:25.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:10:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:10:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:25.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:10:25.895 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:10:26.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:26.371 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:26.371+0000 7fb7a03ba080 -1 Falling back to public interface 2021-01-15T18:10:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:10:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:10:27.290 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:10:27.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:10:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:28.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:10:28.693 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:10:29.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:30.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:30.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:30.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:10:30.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:30.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:10:30.092 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:10:30.341 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:30.340+0000 7fb7a03ba080 -1 osd.2 53 log_to_monitors {default=true} 2021-01-15T18:10:30.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:31.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:31.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:31.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 4 2021-01-15T18:10:31.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:31.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:10:31.492 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:10:31.910 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 62 up_thru 62 down_at 54 last_clean_interval [42,53) [v2:127.0.0.1:6818/710471,v1:127.0.0.1:6819/710471] [v2:127.0.0.1:6820/710471,v1:127.0.0.1:6821/710471] exists,up d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:10:31.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:10:31.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:10:31.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:10:31.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-01-15T18:10:31.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=710471 2021-01-15T18:10:31.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-01-15T18:10:31.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:10:31.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:10:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2021-01-15T18:10:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:10:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-01-15T18:10:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:10:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:10:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:10:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:10:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-01-15T18:10:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-01-15T18:10:31.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:10:31.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:10:31.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:10:31.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:10:31.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:10:31.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:10:31.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:10:31.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:10:31.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:10:31.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:31.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:10:31.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:10:31.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:10:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:10:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:10:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:10:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:31.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:10:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:10:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-01-15T18:10:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2021-01-15T18:10:31.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:10:31.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-01-15T18:10:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2021-01-15T18:10:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:10:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:10:31.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:10:31.924 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:10:31.935 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:31.934+0000 7f7d67678080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:31.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:31.936+0000 7f7d67678080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:31.938 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:31.937+0000 7f7d67678080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:10:32.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2021-01-15T18:10:32.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:10:32.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:10:32.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:10:32.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:10:32.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:32.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:10:32.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:32.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:10:32.333 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:10:32.704 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:32.703+0000 7f7d67678080 -1 Falling back to public interface 2021-01-15T18:10:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:10:33.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:33.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:10:33.730 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:10:34.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:10:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:10:35.131 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:10:35.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:10:36.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:10:36.230+0000 7f7d67678080 -1 osd.3 55 log_to_monitors {default=true} 2021-01-15T18:10:36.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:10:36.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:10:36.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:10:36.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:10:36.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:10:36.598 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:10:37.076 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 65 up_thru 54 down_at 56 last_clean_interval [45,55) [v2:127.0.0.1:6826/711398,v1:127.0.0.1:6827/711398] [v2:127.0.0.1:6828/711398,v1:127.0.0.1:6829/711398] exists,up 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:10:37.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:10:37.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:10:37.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:10:37.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-01-15T18:10:37.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=711398 2021-01-15T18:10:37.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-01-15T18:10:37.103 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:10:37.104 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-01-15T18:10:37.104 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_712282 2021-01-15T18:10:37.104 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:10:37.104 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:10:38.104 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 29 13 0.0507774 0.0507812 0.402342 0.550172 2021-01-15T18:10:39.104 INFO:tasks.workunit.client.0.smithi145.stdout: 2 16 41 25 0.0488214 0.046875 0.13397 0.371458 2021-01-15T18:10:40.104 INFO:tasks.workunit.client.0.smithi145.stdout: 3 16 41 25 0.0325476 0 - 0.371458 2021-01-15T18:10:41.104 INFO:tasks.workunit.client.0.smithi145.stdout: 4 16 41 25 0.0244109 0 - 0.371458 2021-01-15T18:10:42.104 INFO:tasks.workunit.client.0.smithi145.stdout: 5 16 41 25 0.0195287 0 - 0.371458 2021-01-15T18:10:43.105 INFO:tasks.workunit.client.0.smithi145.stdout: 6 16 41 25 0.0162739 0 - 0.371458 2021-01-15T18:10:44.105 INFO:tasks.workunit.client.0.smithi145.stdout: 7 16 41 25 0.013949 0 - 0.371458 2021-01-15T18:10:45.107 INFO:tasks.workunit.client.0.smithi145.stdout: 8 16 41 25 0.0122054 0 - 0.371458 2021-01-15T18:10:46.105 INFO:tasks.workunit.client.0.smithi145.stdout: 9 16 41 25 0.0108492 0 - 0.371458 2021-01-15T18:10:47.105 INFO:tasks.workunit.client.0.smithi145.stdout: 10 16 41 25 0.00976431 0 - 0.371458 2021-01-15T18:10:48.105 INFO:tasks.workunit.client.0.smithi145.stdout: 11 16 60 44 0.0156229 0.00824653 0.329322 2.53195 2021-01-15T18:10:49.105 INFO:tasks.workunit.client.0.smithi145.stdout: 12 16 87 71 0.023109 0.105469 0.287839 1.67133 2021-01-15T18:10:50.105 INFO:tasks.workunit.client.0.smithi145.stdout: 13 16 116 100 0.0300442 0.113281 1.43558 2.0678 2021-01-15T18:10:51.105 INFO:tasks.workunit.client.0.smithi145.stdout: 14 16 159 143 0.0398944 0.167969 0.336836 1.54267 2021-01-15T18:10:52.105 INFO:tasks.workunit.client.0.smithi145.stdout: 15 16 217 201 0.052337 0.226562 0.341428 1.1867 2021-01-15T18:10:53.106 INFO:tasks.workunit.client.0.smithi145.stdout: 16 16 269 253 0.0617596 0.203125 0.325845 1.00556 2021-01-15T18:10:54.106 INFO:tasks.workunit.client.0.smithi145.stdout: 17 16 318 302 0.0693846 0.191406 0.325665 0.891264 2021-01-15T18:10:55.106 INFO:tasks.workunit.client.0.smithi145.stdout: 18 16 373 357 0.077464 0.214844 0.308843 0.799681 2021-01-15T18:10:56.106 INFO:tasks.workunit.client.0.smithi145.stdout: 19 16 425 409 0.0840764 0.203125 0.410647 0.736616 2021-01-15T18:10:57.106 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:10:57.106641+0000 min lat: 0.108641 max lat: 12.9693 avg lat: 0.685741 2021-01-15T18:10:57.107 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:10:57.107 INFO:tasks.workunit.client.0.smithi145.stdout: 20 16 479 463 0.0904182 0.210938 0.327241 0.685741 2021-01-15T18:10:58.106 INFO:tasks.workunit.client.0.smithi145.stdout: 21 16 528 512 0.0952258 0.191406 0.299759 0.650616 2021-01-15T18:10:59.107 INFO:tasks.workunit.client.0.smithi145.stdout: 22 16 587 571 0.101372 0.230469 0.30825 0.613726 2021-01-15T18:11:00.106 INFO:tasks.workunit.client.0.smithi145.stdout: 23 16 638 622 0.105625 0.199219 0.300695 0.586918 2021-01-15T18:11:01.107 INFO:tasks.workunit.client.0.smithi145.stdout: 24 16 695 679 0.1105 0.222656 0.293735 0.562909 2021-01-15T18:11:02.107 INFO:tasks.workunit.client.0.smithi145.stdout: 25 16 749 733 0.114517 0.210938 0.292096 0.543504 2021-01-15T18:11:03.107 INFO:tasks.workunit.client.0.smithi145.stdout: 26 16 798 782 0.117473 0.191406 0.3287 0.528104 2021-01-15T18:11:04.107 INFO:tasks.workunit.client.0.smithi145.stdout: 27 16 852 836 0.120934 0.210938 0.355971 0.514248 2021-01-15T18:11:05.107 INFO:tasks.workunit.client.0.smithi145.stdout: 28 16 906 890 0.124147 0.210938 0.342854 0.501985 2021-01-15T18:11:06.107 INFO:tasks.workunit.client.0.smithi145.stdout: 29 16 950 934 0.125792 0.171875 0.360036 0.492926 2021-01-15T18:11:07.107 INFO:tasks.workunit.client.0.smithi145.stdout: 30 16 989 973 0.126677 0.152344 0.317047 0.490316 2021-01-15T18:11:08.108 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 30.1834 2021-01-15T18:11:08.108 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 989 2021-01-15T18:11:08.108 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:11:08.108 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:11:08.109 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.127994 2021-01-15T18:11:08.109 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0.093786 2021-01-15T18:11:08.109 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.230469 2021-01-15T18:11:08.109 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0 2021-01-15T18:11:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 32 2021-01-15T18:11:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 24.014 2021-01-15T18:11:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 59 2021-01-15T18:11:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 0 2021-01-15T18:11:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.487238 2021-01-15T18:11:08.111 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 1.34103 2021-01-15T18:11:08.111 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 12.9693 2021-01-15T18:11:08.111 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.108641 2021-01-15T18:11:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 708669 2021-01-15T18:11:08.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2021-01-15T18:11:09.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 708669 2021-01-15T18:11:09.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (708669) - No such process 2021-01-15T18:11:09.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2021-01-15T18:11:09.676 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:11:09.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 709570 2021-01-15T18:11:09.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2021-01-15T18:11:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 709570 2021-01-15T18:11:10.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (709570) - No such process 2021-01-15T18:11:10.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2021-01-15T18:11:11.095 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T18:11:11.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 710471 2021-01-15T18:11:11.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2021-01-15T18:11:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 710471 2021-01-15T18:11:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (710471) - No such process 2021-01-15T18:11:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2021-01-15T18:11:12.513 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already down. 2021-01-15T18:11:12.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 711398 2021-01-15T18:11:12.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2021-01-15T18:11:13.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 711398 2021-01-15T18:11:13.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (711398) - No such process 2021-01-15T18:11:13.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2021-01-15T18:11:14.066 INFO:tasks.workunit.client.0.smithi145.stderr:osd.3 is already down. 2021-01-15T18:11:14.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:11:15.861 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:15.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2021-01-15T18:11:16.345 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:11:16.345 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:11:16.345 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.480252 s, 2.2 GB/s 2021-01-15T18:11:16.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2021-01-15T18:11:16.356 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:11:25.127 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2021-01-15T18:11:25.128 INFO:tasks.workunit.client.0.smithi145.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2021-01-15T18:11:25.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:11:27.165 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2021-01-15T18:11:27.424 INFO:tasks.workunit.client.0.smithi145.stderr:512+0 records in 2021-01-15T18:11:27.425 INFO:tasks.workunit.client.0.smithi145.stderr:512+0 records out 2021-01-15T18:11:27.425 INFO:tasks.workunit.client.0.smithi145.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.254844 s, 2.1 GB/s 2021-01-15T18:11:27.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2021-01-15T18:11:27.435 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:11:32.455 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2021-01-15T18:11:32.456 INFO:tasks.workunit.client.0.smithi145.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2021-01-15T18:11:32.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2021-01-15T18:11:32.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2021-01-15T18:11:34.295 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:34.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:11:36.086 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:36.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2021-01-15T18:11:36.570 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:11:36.571 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:11:36.571 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.47968 s, 2.2 GB/s 2021-01-15T18:11:36.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2021-01-15T18:11:36.581 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:11:45.355 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-01-15T18:11:45.355 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2021-01-15T18:11:45.355 INFO:tasks.workunit.client.0.smithi145.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2021-01-15T18:11:45.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2021-01-15T18:11:45.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:11:46.735 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:46.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2021-01-15T18:11:46.751 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:11:48.024 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2021-01-15T18:11:48.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2021-01-15T18:11:48.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2021-01-15T18:11:48.904 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:48.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2021-01-15T18:11:50.695 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:50.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-01-15T18:11:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:11:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:11:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.480745 s, 2.2 GB/s 2021-01-15T18:11:51.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-01-15T18:11:51.191 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:11:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2021-01-15T18:11:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2021-01-15T18:11:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:11:52.979 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:11:52.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2021-01-15T18:12:00.377 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:12:00.378 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:12:00.378 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.25882 s, 148 MB/s 2021-01-15T18:12:00.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2021-01-15T18:12:00.392 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:12:02.414 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2021-01-15T18:12:02.415 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2021-01-15T18:12:02.415 INFO:tasks.workunit.client.0.smithi145.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2021-01-15T18:12:02.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2021-01-15T18:12:03.951 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:12:03.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2021-01-15T18:12:04.444 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records in 2021-01-15T18:12:04.445 INFO:tasks.workunit.client.0.smithi145.stderr:1024+0 records out 2021-01-15T18:12:04.445 INFO:tasks.workunit.client.0.smithi145.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.489521 s, 2.2 GB/s 2021-01-15T18:12:04.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2021-01-15T18:12:04.455 INFO:tasks.workunit.client.0.smithi145.stdout:inferring bluefs devices from bluestore path 2021-01-15T18:12:12.476 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2021-01-15T18:12:12.477 INFO:tasks.workunit.client.0.smithi145.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2021-01-15T18:12:12.477 INFO:tasks.workunit.client.0.smithi145.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2021-01-15T18:12:12.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2021-01-15T18:12:13.769 INFO:tasks.workunit.client.0.smithi145.stdout:fsck success 2021-01-15T18:12:13.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2021-01-15T18:12:13.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:12:13.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:12:13.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:12:13.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:12:13.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2021-01-15T18:12:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:12:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:12:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:12:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:12:13.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2021-01-15T18:12:13.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2021-01-15T18:12:13.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:12:13.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:12:13.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:12:13.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:12:13.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:12:13.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:12:13.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:12:13.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:12:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:12:13.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:12:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:12:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:12:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:12:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:12:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:12:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:12:13.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2021-01-15T18:12:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:12:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:13.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2021-01-15T18:12:13.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:12:13.782 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:12:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:12:13.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:12:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:12:13.799 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:13.798+0000 7f0dd16b6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:13.801 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:13.800+0000 7f0dd16b6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:13.803 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:13.802+0000 7f0dd16b6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:14.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:12:14.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:12:14.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:12:14.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:12:14.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:12:14.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:14.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:12:14.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:14.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:12:14.180 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:12:14.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:14.818 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:14.817+0000 7f0dd16b6080 -1 Falling back to public interface 2021-01-15T18:12:15.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:15.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:15.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:12:15.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:15.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:12:15.565 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:12:15.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:16.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:16.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:16.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:12:16.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:16.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:12:16.969 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:12:17.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:12:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:18.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:12:18.367 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:12:18.521 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:18.514+0000 7f0dd16b6080 -1 osd.0 67 log_to_monitors {default=true} 2021-01-15T18:12:18.906 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 77 up_thru 65 down_at 68 last_clean_interval [57,67) [v2:127.0.0.1:6802/718107,v1:127.0.0.1:6803/718107] [v2:127.0.0.1:6804/718107,v1:127.0.0.1:6805/718107] exists,up 5791b99b-226a-4b6b-adf7-3cb9dae2f5b3 2021-01-15T18:12:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:12:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:12:18.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:12:18.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2021-01-15T18:12:18.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=718107 2021-01-15T18:12:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2021-01-15T18:12:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:12:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:12:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T18:12:18.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:12:18.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2021-01-15T18:12:18.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:12:18.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:12:18.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:12:18.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:12:18.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2021-01-15T18:12:18.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2021-01-15T18:12:18.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:12:18.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:12:18.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:12:18.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:12:18.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:12:18.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:12:18.914 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:12:18.914 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:12:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:12:18.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:18.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:12:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:12:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:12:18.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:12:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:12:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:12:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:12:18.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2021-01-15T18:12:18.918 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:12:18.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T18:12:18.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:18.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2021-01-15T18:12:18.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T18:12:18.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:12:18.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:12:18.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:12:18.932 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:18.932+0000 7fdf1f7ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:18.934 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:18.933+0000 7fdf1f7ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:18.937 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:18.936+0000 7fdf1f7ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:19.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T18:12:19.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:12:19.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:12:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:12:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:12:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:12:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:19.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:12:19.407 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:12:19.454 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:19.454+0000 7fdf1f7ca080 -1 Falling back to public interface 2021-01-15T18:12:19.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:20.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:20.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:20.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:12:20.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:20.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:12:20.800 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:12:21.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:22.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:22.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:22.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:12:22.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:22.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:12:22.194 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:12:22.435 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:22.434+0000 7fdf1f7ca080 -1 osd.1 70 log_to_monitors {default=true} 2021-01-15T18:12:22.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 3 2021-01-15T18:12:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:12:23.608 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:12:23.996 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 71 last_clean_interval [59,70) [v2:127.0.0.1:6810/719009,v1:127.0.0.1:6811/719009] [v2:127.0.0.1:6812/719009,v1:127.0.0.1:6813/719009] exists,up 93be79a4-4607-4171-8cc5-1d56128fbaec 2021-01-15T18:12:23.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:12:23.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:12:23.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:12:24.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2021-01-15T18:12:24.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=719009 2021-01-15T18:12:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2021-01-15T18:12:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:12:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:12:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T18:12:24.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:12:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2021-01-15T18:12:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:12:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:12:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:12:24.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:12:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2021-01-15T18:12:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2021-01-15T18:12:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:12:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:12:24.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:12:24.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:12:24.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:12:24.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:12:24.004 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:12:24.005 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:12:24.005 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:12:24.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:24.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:24.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:12:24.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:12:24.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:12:24.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:12:24.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:12:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:12:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:12:24.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2021-01-15T18:12:24.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T18:12:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:24.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2021-01-15T18:12:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T18:12:24.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:12:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:12:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:12:24.010 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:12:24.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:24.020+0000 7f419fd2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:24.022 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:24.022+0000 7f419fd2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:24.024 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:24.023+0000 7f419fd2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:24.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T18:12:24.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:12:24.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:12:24.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:12:24.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:12:24.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:24.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:12:24.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:24.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:12:24.399 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:12:24.796 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:24.796+0000 7f419fd2e080 -1 Falling back to public interface 2021-01-15T18:12:24.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:25.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:25.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:25.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:12:25.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:25.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:12:25.806 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:12:26.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:27.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:27.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:27.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:12:27.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:27.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:12:27.202 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:12:27.396 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:27.395+0000 7f419fd2e080 -1 osd.2 73 log_to_monitors {default=true} 2021-01-15T18:12:27.672 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 82 up_thru 71 down_at 74 last_clean_interval [62,73) [v2:127.0.0.1:6818/719890,v1:127.0.0.1:6819/719890] [v2:127.0.0.1:6820/719890,v1:127.0.0.1:6821/719890] exists,up d46bab4e-10e9-487f-868e-560dd20ffd2b 2021-01-15T18:12:27.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:12:27.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:12:27.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:12:27.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2021-01-15T18:12:27.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=719890 2021-01-15T18:12:27.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2021-01-15T18:12:27.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:12:27.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:12:27.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=3 2021-01-15T18:12:27.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:12:27.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2021-01-15T18:12:27.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2021-01-15T18:12:27.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:12:27.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:12:27.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:12:27.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2021-01-15T18:12:27.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2021-01-15T18:12:27.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:12:27.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:12:27.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2021-01-15T18:12:27.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:12:27.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:12:27.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:12:27.691 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:12:27.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:12:27.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:12:27.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:27.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' 2021-01-15T18:12:27.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:12:27.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2021-01-15T18:12:27.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2021-01-15T18:12:27.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:12:27.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:12:27.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:27.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:12:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:12:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2021-01-15T18:12:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.3 2021-01-15T18:12:27.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 3 --fsid=17129678-1a3b-427e-84b9-41e5764f39a5 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.679134/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:12:27.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2021-01-15T18:12:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 3 = 3 ']' 2021-01-15T18:12:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:12:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:12:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:12:27.697 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:12:27.709 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:27.708+0000 7f39fbf5b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:27.711 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:27.711+0000 7f39fbf5b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:27.712 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:27.712+0000 7f39fbf5b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:12:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 3 2021-01-15T18:12:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:12:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:12:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:12:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:12:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:12:28.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:28.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:12:28.143 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:12:28.486 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:28.486+0000 7f39fbf5b080 -1 Falling back to public interface 2021-01-15T18:12:28.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:29.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:29.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:12:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:12:29.720 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:12:30.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:12:31.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:12:31.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:12:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:12:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:12:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:12:31.133 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:12:31.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:12:31.359+0000 7f39fbf5b080 -1 osd.3 75 log_to_monitors {default=true} 2021-01-15T18:12:31.567 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 85 up_thru 74 down_at 76 last_clean_interval [65,75) [v2:127.0.0.1:6826/720755,v1:127.0.0.1:6827/720755] [v2:127.0.0.1:6828/720755,v1:127.0.0.1:6829/720755] exists,up 2c65a15a-25a8-495b-ac17-67309590a354 2021-01-15T18:12:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:12:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:12:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:12:31.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2021-01-15T18:12:31.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=720755 2021-01-15T18:12:31.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2021-01-15T18:12:31.665 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:12:31.666 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-01-15T18:12:31.666 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_721605 2021-01-15T18:12:31.666 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:12:31.666 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:12:32.665 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 20 4 0.0156237 0.015625 0.171617 0.260902 2021-01-15T18:12:33.665 INFO:tasks.workunit.client.0.smithi145.stdout: 2 16 47 31 0.0605402 0.105469 0.265001 0.622462 2021-01-15T18:12:34.666 INFO:tasks.workunit.client.0.smithi145.stdout: 3 16 60 44 0.057286 0.0507812 0.175521 0.502381 2021-01-15T18:12:35.666 INFO:tasks.workunit.client.0.smithi145.stdout: 4 16 66 50 0.0488231 0.0234375 0.0422413 0.454993 2021-01-15T18:12:36.666 INFO:tasks.workunit.client.0.smithi145.stdout: 5 16 66 50 0.0390585 0 - 0.454993 2021-01-15T18:12:37.666 INFO:tasks.workunit.client.0.smithi145.stdout: 6 16 66 50 0.0325487 0 - 0.454993 2021-01-15T18:12:38.666 INFO:tasks.workunit.client.0.smithi145.stdout: 7 16 66 50 0.0278988 0 - 0.454993 2021-01-15T18:12:39.666 INFO:tasks.workunit.client.0.smithi145.stdout: 8 16 66 50 0.0244115 0 - 0.454993 2021-01-15T18:12:40.666 INFO:tasks.workunit.client.0.smithi145.stdout: 9 16 95 79 0.0342844 0.0226562 0.242853 1.12454 2021-01-15T18:12:41.667 INFO:tasks.workunit.client.0.smithi145.stdout: 10 16 109 93 0.0363241 0.0546875 0.0921911 0.979375 2021-01-15T18:12:42.667 INFO:tasks.workunit.client.0.smithi145.stdout: 11 16 109 93 0.0330218 0 - 0.979375 2021-01-15T18:12:43.667 INFO:tasks.workunit.client.0.smithi145.stdout: 12 16 109 93 0.03027 0 - 0.979375 2021-01-15T18:12:44.667 INFO:tasks.workunit.client.0.smithi145.stdout: 13 16 130 114 0.034251 0.0273438 0.13441 1.81068 2021-01-15T18:12:45.667 INFO:tasks.workunit.client.0.smithi145.stdout: 14 16 193 177 0.0493808 0.246094 0.125373 1.25262 2021-01-15T18:12:46.667 INFO:tasks.workunit.client.0.smithi145.stdout: 15 16 256 240 0.0624932 0.246094 0.277029 0.99243 2021-01-15T18:12:47.667 INFO:tasks.workunit.client.0.smithi145.stdout: 16 16 318 302 0.0737222 0.242188 0.254052 0.840069 2021-01-15T18:12:48.667 INFO:tasks.workunit.client.0.smithi145.stdout: 17 16 377 361 0.082941 0.230469 0.152756 0.746041 2021-01-15T18:12:49.667 INFO:tasks.workunit.client.0.smithi145.stdout: 18 16 433 417 0.0904844 0.21875 0.241085 0.684931 2021-01-15T18:12:50.667 INFO:tasks.workunit.client.0.smithi145.stdout: 19 16 495 479 0.0984672 0.242188 0.287746 0.631844 2021-01-15T18:12:51.668 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:12:51.668221+0000 min lat: 0.0422413 max lat: 12.859 avg lat: 0.578426 2021-01-15T18:12:51.668 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:12:51.668 INFO:tasks.workunit.client.0.smithi145.stdout: 20 16 566 550 0.107409 0.277344 0.194627 0.578426 2021-01-15T18:12:52.668 INFO:tasks.workunit.client.0.smithi145.stdout: 21 16 624 608 0.113082 0.226562 0.275013 0.548322 2021-01-15T18:12:53.668 INFO:tasks.workunit.client.0.smithi145.stdout: 22 16 688 672 0.119304 0.25 0.29847 0.520701 2021-01-15T18:12:54.668 INFO:tasks.workunit.client.0.smithi145.stdout: 23 16 744 728 0.123626 0.21875 0.267275 0.503129 2021-01-15T18:12:55.668 INFO:tasks.workunit.client.0.smithi145.stdout: 24 16 817 801 0.130355 0.285156 0.163033 0.478271 2021-01-15T18:12:56.668 INFO:tasks.workunit.client.0.smithi145.stdout: 25 16 871 855 0.133577 0.210938 0.1656 0.465666 2021-01-15T18:12:57.668 INFO:tasks.workunit.client.0.smithi145.stdout: 26 16 935 919 0.138054 0.25 0.158102 0.450496 2021-01-15T18:12:58.669 INFO:tasks.workunit.client.0.smithi145.stdout: 27 16 993 977 0.141331 0.226562 0.125848 0.439885 2021-01-15T18:12:59.669 INFO:tasks.workunit.client.0.smithi145.stdout: 28 16 1052 1036 0.144513 0.230469 0.149566 0.430282 2021-01-15T18:13:00.669 INFO:tasks.workunit.client.0.smithi145.stdout: 29 16 1123 1107 0.149092 0.277344 0.132353 0.417954 2021-01-15T18:13:01.669 INFO:tasks.workunit.client.0.smithi145.stdout: 30 16 1180 1164 0.151543 0.222656 0.100372 0.410039 2021-01-15T18:13:02.669 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 30.1031 2021-01-15T18:13:02.670 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 1180 2021-01-15T18:13:02.670 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:13:02.670 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:13:02.670 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.153119 2021-01-15T18:13:02.670 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0.112986 2021-01-15T18:13:02.671 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.285156 2021-01-15T18:13:02.671 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0 2021-01-15T18:13:02.671 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 39 2021-01-15T18:13:02.671 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 28.9551 2021-01-15T18:13:02.671 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 73 2021-01-15T18:13:02.672 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 0 2021-01-15T18:13:02.672 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.407683 2021-01-15T18:13:02.672 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 1.0873 2021-01-15T18:13:02.672 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 12.859 2021-01-15T18:13:02.672 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.0422413 2021-01-15T18:13:02.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2021-01-15T18:13:02.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:13:02.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:13:02.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:13:02.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:13:02.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:13:02.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:13:02.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:13:02.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:13:02.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:13:02.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:13:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:13:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:13:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:13:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:13:02.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:13:03.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:13:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:13:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:13:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:13:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:13:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:13:03.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:13:03.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=330712481803 2021-01-15T18:13:03.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 330712481803 2021-01-15T18:13:03.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-330712481803' 2021-01-15T18:13:03.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:13:03.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:13:03.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=339302416395 2021-01-15T18:13:03.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 339302416395 2021-01-15T18:13:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-330712481803 1-339302416395' 2021-01-15T18:13:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:13:03.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:13:03.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=352187318282 2021-01-15T18:13:03.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 352187318282 2021-01-15T18:13:03.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-330712481803 1-339302416395 2-352187318282' 2021-01-15T18:13:03.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:13:03.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:13:03.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=365072220169 2021-01-15T18:13:03.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 365072220169 2021-01-15T18:13:03.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-330712481803 1-339302416395 2-352187318282 3-365072220169' 2021-01-15T18:13:03.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:13:03.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-330712481803 2021-01-15T18:13:03.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:13:03.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:13:03.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-330712481803 2021-01-15T18:13:03.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:13:03.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=330712481803 2021-01-15T18:13:03.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 330712481803' 2021-01-15T18:13:03.822 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 330712481803 2021-01-15T18:13:03.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:13:04.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 330712481804 -lt 330712481803 2021-01-15T18:13:04.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:13:04.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-339302416395 2021-01-15T18:13:04.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:13:04.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:13:04.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-339302416395 2021-01-15T18:13:04.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:13:04.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=339302416395 2021-01-15T18:13:04.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 339302416395' 2021-01-15T18:13:04.214 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 339302416395 2021-01-15T18:13:04.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:13:04.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 339302416395 -lt 339302416395 2021-01-15T18:13:04.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:13:04.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-352187318282 2021-01-15T18:13:04.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:13:04.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:13:04.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-352187318282 2021-01-15T18:13:04.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:13:04.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=352187318282 2021-01-15T18:13:04.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 352187318282' 2021-01-15T18:13:04.606 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 352187318282 2021-01-15T18:13:04.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:13:05.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 352187318282 -lt 352187318282 2021-01-15T18:13:05.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:13:05.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-365072220169 2021-01-15T18:13:05.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:13:05.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:13:05.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-365072220169 2021-01-15T18:13:05.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:13:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=365072220169 2021-01-15T18:13:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 365072220169' 2021-01-15T18:13:05.012 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 365072220169 2021-01-15T18:13:05.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:13:05.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 365072220167 -lt 365072220169 2021-01-15T18:13:05.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:13:06.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:13:06.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:13:06.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 365072220169 -lt 365072220169 2021-01-15T18:13:06.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:13:06.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:13:06.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:13:07.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2021-01-15T18:13:07.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:13:07.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:13:07.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:13:07.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:13:07.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:13:07.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:13:07.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:13:07.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2021-01-15T18:13:07.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:13:07.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:13:07.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:13:08.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2021-01-15T18:13:08.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:13:08.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:13:08.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:25: run: teardown td/osd-bluefs-volume-ops 2021-01-15T18:13:08.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:13:08.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:08.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-01-15T18:13:08.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:08.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:08.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:08.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:08.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:08.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:08.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:08.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:08.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:08.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:08.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:08.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-01-15T18:13:08.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:08.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:08.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:13:08.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679134 2021-01-15T18:13:08.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:13:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-bluefs-volume-ops 0 2021-01-15T18:13:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2021-01-15T18:13:08.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:13:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2021-01-15T18:13:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:08.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:08.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:08.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:08.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:08.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:08.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:08.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:08.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:08.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:08.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:08.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:08.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:08.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:08.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:13:08.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2021-01-15T18:13:08.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:08.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:08.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.679134 2021-01-15T18:13:08.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.679134 2021-01-15T18:13:08.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:08.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:08.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:13:08.695 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:13:08.695 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:13:08.767 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2021-01-15T18:13:08.767 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2021-01-15T18:13:08.827 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:13:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:13:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:08.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:13:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:13:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:13:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:13:08.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:13:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:13:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-config 2021-01-15T18:13:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2021-01-15T18:13:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2021-01-15T18:13:08.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2021-01-15T18:13:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2021-01-15T18:13:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2021-01-15T18:13:08.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2021-01-15T18:13:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none ' 2021-01-15T18:13:08.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2021-01-15T18:13:08.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2021-01-15T18:13:08.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:13:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2021-01-15T18:13:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_config_track 2021-01-15T18:13:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_default_adjustment' 2021-01-15T18:13:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-01-15T18:13:08.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-01-15T18:13:08.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-01-15T18:13:08.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-01-15T18:13:08.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:08.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:08.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:08.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:08.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:08.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:08.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:08.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:08.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:08.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:08.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:08.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:08.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:08.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:08.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:08.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:08.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-01-15T18:13:08.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:13:08.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:08.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:08.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.722181 2021-01-15T18:13:08.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:13:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:13:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:13:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:13:08.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:13:08.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:13:08.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:13:08.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-01-15T18:13:08.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-01-15T18:13:09.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:13:09.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:09.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:09.140 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:09.140 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:09.141 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:09.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:09.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:13:09.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:13:09.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:13:09.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:09.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:09.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:13:09.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:13:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:09.318 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:09.319 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:09.319 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-mon.a.asok 2021-01-15T18:13:09.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:09.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.722181/ceph-mon.a.asok config get fsid 2021-01-15T18:13:09.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:13:09.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:09.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:09.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:13:09.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:13:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:09.419 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:09.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-mon.a.asok 2021-01-15T18:13:09.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:09.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.722181/ceph-mon.a.asok config get mon_host 2021-01-15T18:13:09.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2021-01-15T18:13:09.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-config 2021-01-15T18:13:09.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:13:09.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:13:09.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:13:09.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-config/x 2021-01-15T18:13:09.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:13:10.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:13:10.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:10.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:10.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:10.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:10.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:10.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:10.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:10.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:10.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2021-01-15T18:13:10.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2021-01-15T18:13:10.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-01-15T18:13:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-config 2021-01-15T18:13:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:13:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:13:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:13:10.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-config/0 2021-01-15T18:13:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-01-15T18:13:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:10.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-01-15T18:13:10.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-01-15T18:13:10.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:13:10.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:13:10.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-01-15T18:13:10.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:13:10.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:10.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:10.046 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:10.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:10.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-01-15T18:13:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-01-15T18:13:10.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:13:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2021-01-15T18:13:10.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-config/0 2021-01-15T18:13:10.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:13:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0ee883a9-914d-4b31-a443-e400ffd76533 2021-01-15T18:13:10.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 0ee883a9-914d-4b31-a443-e400ffd76533' 2021-01-15T18:13:10.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:13:10.051 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 0ee883a9-914d-4b31-a443-e400ffd76533 2021-01-15T18:13:10.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA22wFgkMkzAxAAWlOaahKjtt8gwVRj2UCUJA== 2021-01-15T18:13:10.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA22wFgkMkzAxAAWlOaahKjtt8gwVRj2UCUJA=="}' 2021-01-15T18:13:10.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0ee883a9-914d-4b31-a443-e400ffd76533 -i td/osd-config/0/new.json 2021-01-15T18:13:10.390 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:10.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-config/0/new.json 2021-01-15T18:13:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQA22wFgkMkzAxAAWlOaahKjtt8gwVRj2UCUJA== --osd-uuid 0ee883a9-914d-4b31-a443-e400ffd76533 2021-01-15T18:13:10.424 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.423+0000 7fd1d7c74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:10.426 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.425+0000 7fd1d7c74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:10.428 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.427+0000 7fd1d7c74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.427+0000 7fd1d7c74080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.427+0000 7fd1d7c74080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.427+0000 7fd1d7c74080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:10.472 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:10.471+0000 7fd1d7c74080 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-01-15T18:13:15.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-config/0/keyring 2021-01-15T18:13:15.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:13:15.006 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:13:15.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:13:15.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:15.522 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:13:15.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:13:15.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2021-01-15T18:13:15.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:13:15.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:13:15.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:13:15.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:15.541+0000 7f7b5218b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:15.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:15.570+0000 7f7b5218b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:15.573 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:15.572+0000 7f7b5218b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:15.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:13:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:13:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:15.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:15.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:15.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:15.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:15.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:15.913 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:16.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:16.086+0000 7f7b5218b080 -1 Falling back to public interface 2021-01-15T18:13:16.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:16.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:16.871+0000 7f7b5218b080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:13:17.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:17.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:17.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:17.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:17.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:17.314 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:17.739 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/723220,v1:127.0.0.1:6803/723220] [v2:127.0.0.1:6804/723220,v1:127.0.0.1:6805/723220] exists,up 0ee883a9-914d-4b31-a443-e400ffd76533 2021-01-15T18:13:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:17.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:17.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2021-01-15T18:13:17.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T18:13:17.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T18:13:17.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:17.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:17.741 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:17.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-osd.0.asok 2021-01-15T18:13:17.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2021-01-15T18:13:17.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.722181/ceph-osd.0.asok log flush 2021-01-15T18:13:17.839 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:13:17.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-01-15T18:13:17.849 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:13:17.157+0000 7f7b5218b080 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2021-01-15T18:13:17.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-01-15T18:13:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:17.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:17.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:17.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:17.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:17.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:17.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:17.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:17.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:17.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:17.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:17.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:17.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:17.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:17.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:17.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:17.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:17.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:17.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:17.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-01-15T18:13:17.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-01-15T18:13:17.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-01-15T18:13:17.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-01-15T18:13:17.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:17.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:17.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:17.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:17.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:17.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:17.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:17.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:17.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:17.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:17.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:17.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:17.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:17.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:17.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:17.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:17.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:17.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:17.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:17.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-01-15T18:13:17.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:13:17.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:17.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:17.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.722181 2021-01-15T18:13:17.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:13:17.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:13:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:13:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-01-15T18:13:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2021-01-15T18:13:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2021-01-15T18:13:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2021-01-15T18:13:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2021-01-15T18:13:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:13:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:13:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:13:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2021-01-15T18:13:17.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2021-01-15T18:13:18.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:13:18.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:18.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:18.266 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:18.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:18.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:13:18.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:13:18.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:13:18.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:18.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:18.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:13:18.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:13:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:18.445 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:18.445 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:18.445 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-mon.a.asok 2021-01-15T18:13:18.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:18.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.722181/ceph-mon.a.asok config get fsid 2021-01-15T18:13:18.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:13:18.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:18.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:18.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:13:18.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:13:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:18.546 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:18.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-mon.a.asok 2021-01-15T18:13:18.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:18.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.722181/ceph-mon.a.asok config get mon_host 2021-01-15T18:13:18.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2021-01-15T18:13:18.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-config 2021-01-15T18:13:18.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:13:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:13:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:13:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-config/x 2021-01-15T18:13:18.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:13:19.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:13:19.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:19.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:19.137 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:19.137 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:19.137 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:19.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:19.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:19.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:19.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2021-01-15T18:13:19.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-config 2021-01-15T18:13:19.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:13:19.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:13:19.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:13:19.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-config/0 2021-01-15T18:13:19.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2021-01-15T18:13:19.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:19.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:19.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:19.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2021-01-15T18:13:19.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2021-01-15T18:13:19.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-config' 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:19.163 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:19.164 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:19.164 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:19.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:19.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' 2021-01-15T18:13:19.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:19.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:19.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:19.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2021-01-15T18:13:19.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2021-01-15T18:13:19.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:19.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:19.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:19.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:13:19.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:13:19.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-config/0 2021-01-15T18:13:19.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:13:19.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=263da4bf-c702-46b0-a40e-1ca02a5a112a 2021-01-15T18:13:19.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 263da4bf-c702-46b0-a40e-1ca02a5a112a' 2021-01-15T18:13:19.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:13:19.168 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 263da4bf-c702-46b0-a40e-1ca02a5a112a 2021-01-15T18:13:19.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA/2wFg96NXChAAcbW242WB+X12I7QYwnuFcQ== 2021-01-15T18:13:19.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA/2wFg96NXChAAcbW242WB+X12I7QYwnuFcQ=="}' 2021-01-15T18:13:19.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 263da4bf-c702-46b0-a40e-1ca02a5a112a -i td/osd-config/0/new.json 2021-01-15T18:13:19.526 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:19.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-config/0/new.json 2021-01-15T18:13:19.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA/2wFg96NXChAAcbW242WB+X12I7QYwnuFcQ== --osd-uuid 263da4bf-c702-46b0-a40e-1ca02a5a112a 2021-01-15T18:13:19.559 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.559+0000 7f7597ca6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:19.561 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.561+0000 7f7597ca6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:19.562 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.562+0000 7f7597ca6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:19.563 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.563+0000 7f7597ca6080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:19.564 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.563+0000 7f7597ca6080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:19.564 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.563+0000 7f7597ca6080 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2021-01-15T18:13:19.608 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:19.607+0000 7f7597ca6080 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2021-01-15T18:13:24.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-config/0/keyring 2021-01-15T18:13:24.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:13:24.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:13:24.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:24.130 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:13:24.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:13:24.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8638eb55-3cd2-4a02-935c-1bcd57d45f08 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.722181/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:24.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:13:24.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:13:24.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:13:24.646 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:13:24.663 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:24.662+0000 7fe6125fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:24.690 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:24.689+0000 7fe6125fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:24.691 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:24.691+0000 7fe6125fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:25.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:13:25.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:25.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:13:25.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:25.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:25.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:25.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:25.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:25.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:25.067 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:25.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:25.207+0000 7fe6125fa080 -1 Falling back to public interface 2021-01-15T18:13:25.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:25.998+0000 7fe6125fa080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:13:26.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:26.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:26.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:26.463 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:26.881 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/724754,v1:127.0.0.1:6803/724754] [v2:127.0.0.1:6804/724754,v1:127.0.0.1:6805/724754] exists,up 263da4bf-c702-46b0-a40e-1ca02a5a112a 2021-01-15T18:13:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:26.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:26.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:26.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2021-01-15T18:13:26.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2021-01-15T18:13:26.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2021-01-15T18:13:26.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2021-01-15T18:13:26.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2021-01-15T18:13:26.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2021-01-15T18:13:26.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-01-15T18:13:26.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2021-01-15T18:13:26.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2021-01-15T18:13:27.053 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:13:27.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2021-01-15T18:13:27.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T18:13:27.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T18:13:27.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:27.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.065 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-osd.0.asok 2021-01-15T18:13:27.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2021-01-15T18:13:27.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.722181/ceph-osd.0.asok log flush 2021-01-15T18:13:27.166 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:13:27.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2021-01-15T18:13:27.175 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T18:13:27.038+0000 7fe60bd92700 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2021-01-15T18:13:27.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2021-01-15T18:13:27.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2021-01-15T18:13:27.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T18:13:27.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T18:13:27.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:27.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.178 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.722181/ceph-osd.0.asok 2021-01-15T18:13:27.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2021-01-15T18:13:27.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.722181/ceph-osd.0.asok log reopen 2021-01-15T18:13:27.279 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:13:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-01-15T18:13:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:27.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:27.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:27.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:27.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:27.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:27.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:27.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:27.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:27.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:27.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:27.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:27.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:27.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:27.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:27.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:27.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:27.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:27.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:27.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:27.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2021-01-15T18:13:27.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2021-01-15T18:13:27.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2021-01-15T18:13:27.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2021-01-15T18:13:27.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:27.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:27.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:27.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:27.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:27.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:27.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:27.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:27.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:27.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:27.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:27.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:27.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:27.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:27.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:27.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:27.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:27.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:27.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:27.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:27.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:27.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:27.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:27.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:27.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2021-01-15T18:13:27.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:13:27.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.722181 2021-01-15T18:13:27.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:13:27.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:13:27.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:13:27.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2021-01-15T18:13:27.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2021-01-15T18:13:27.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2021-01-15T18:13:27.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2021-01-15T18:13:27.428 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2021-01-15T18:13:27.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2021-01-15T18:13:27.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2021-01-15T18:13:27.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2021-01-15T18:13:27.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2021-01-15T18:13:27.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2021-01-15T18:13:27.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2021-01-15T18:13:27.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2021-01-15T18:13:27.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2021-01-15T18:13:27.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2021-01-15T18:13:27.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2021-01-15T18:13:27.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2021-01-15T18:13:27.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2021-01-15T18:13:27.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2021-01-15T18:13:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2021-01-15T18:13:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2021-01-15T18:13:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2021-01-15T18:13:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:27.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:27.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:27.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:27.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:27.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:27.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:27.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:27.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:27.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:27.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:27.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:27.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:27.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:27.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:27.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:27.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:27.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:27.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:27.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:27.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:13:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-config 0 2021-01-15T18:13:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2021-01-15T18:13:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:13:27.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2021-01-15T18:13:27.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:27.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:27.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:27.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:27.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:27.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:27.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:27.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:27.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:27.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:27.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:27.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:27.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:27.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:27.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:27.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:13:27.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2021-01-15T18:13:27.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:27.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.722181 2021-01-15T18:13:27.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.722181 2021-01-15T18:13:27.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:27.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:27.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:13:27.519 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:13:27.520 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:13:27.581 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2021-01-15T18:13:27.581 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2021-01-15T18:13:27.641 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:13:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:13:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:13:27.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:13:27.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:13:27.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:13:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:13:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:13:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-copy-from 2021-01-15T18:13:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2021-01-15T18:13:27.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2021-01-15T18:13:27.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2021-01-15T18:13:27.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2021-01-15T18:13:27.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2021-01-15T18:13:27.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2021-01-15T18:13:27.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none ' 2021-01-15T18:13:27.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2021-01-15T18:13:27.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2021-01-15T18:13:27.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:13:27.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2021-01-15T18:13:27.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2021-01-15T18:13:27.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2021-01-15T18:13:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2021-01-15T18:13:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2021-01-15T18:13:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-01-15T18:13:27.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-01-15T18:13:27.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:27.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:27.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:27.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:27.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:27.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:27.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:27.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:27.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:27.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:27.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:27.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:27.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:27.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:27.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:27.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:27.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-01-15T18:13:27.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:27.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:27.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.725383 2021-01-15T18:13:27.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:27.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:27.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2021-01-15T18:13:27.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:13:27.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:27.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.725383 2021-01-15T18:13:27.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:13:27.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:13:27.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:13:27.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:13:27.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2021-01-15T18:13:27.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2021-01-15T18:13:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2021-01-15T18:13:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2021-01-15T18:13:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2021-01-15T18:13:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:13:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:13:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:13:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2021-01-15T18:13:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2021-01-15T18:13:27.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:13:27.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:27.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:27.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:27.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:13:28.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:13:28.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:13:28.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:28.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:28.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:13:28.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:13:28.133 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:28.133 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:28.134 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:28.134 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:28.135 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.725383/ceph-mon.a.asok 2021-01-15T18:13:28.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:28.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.725383/ceph-mon.a.asok config get fsid 2021-01-15T18:13:28.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:13:28.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:28.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:28.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:13:28.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:13:28.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:28.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:28.234 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:28.234 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:28.234 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:28.235 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:28.235 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.725383/ceph-mon.a.asok 2021-01-15T18:13:28.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:28.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.725383/ceph-mon.a.asok config get mon_host 2021-01-15T18:13:28.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2021-01-15T18:13:28.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-copy-from 2021-01-15T18:13:28.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:13:28.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:13:28.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:13:28.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-copy-from/x 2021-01-15T18:13:28.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:13:28.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:13:28.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:28.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:28.831 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:28.831 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:28.831 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:28.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:28.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:28.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:28.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2021-01-15T18:13:28.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-copy-from 2021-01-15T18:13:28.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:13:28.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:13:28.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:13:28.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-copy-from/0 2021-01-15T18:13:28.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-01-15T18:13:28.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:28.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:28.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:28.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2021-01-15T18:13:28.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2021-01-15T18:13:28.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:13:28.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:13:28.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-01-15T18:13:28.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:13:28.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:28.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:28.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:28.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:28.859 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:28.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:28.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:28.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:28.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:28.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:28.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-01-15T18:13:28.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-01-15T18:13:28.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:28.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:13:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:13:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-copy-from/0 2021-01-15T18:13:28.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:13:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5608cfcb-b687-4b07-9359-20b2506e516d 2021-01-15T18:13:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 5608cfcb-b687-4b07-9359-20b2506e516d' 2021-01-15T18:13:28.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:13:28.864 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 5608cfcb-b687-4b07-9359-20b2506e516d 2021-01-15T18:13:28.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBI2wFgAfGnMxAArYee326njYG0SS7327+p4Q== 2021-01-15T18:13:28.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBI2wFgAfGnMxAArYee326njYG0SS7327+p4Q=="}' 2021-01-15T18:13:28.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5608cfcb-b687-4b07-9359-20b2506e516d -i td/osd-copy-from/0/new.json 2021-01-15T18:13:29.203 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:29.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-copy-from/0/new.json 2021-01-15T18:13:29.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBI2wFgAfGnMxAArYee326njYG0SS7327+p4Q== --osd-uuid 5608cfcb-b687-4b07-9359-20b2506e516d 2021-01-15T18:13:29.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.236+0000 7f3a801c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:29.239 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.238+0000 7f3a801c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:29.240 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.239+0000 7f3a801c4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:29.240 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.239+0000 7f3a801c4080 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-01-15T18:13:29.240 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.239+0000 7f3a801c4080 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-01-15T18:13:29.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.239+0000 7f3a801c4080 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2021-01-15T18:13:29.285 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:29.285+0000 7f3a801c4080 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2021-01-15T18:13:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-copy-from/0/keyring 2021-01-15T18:13:33.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:13:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:13:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:33.813 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:13:34.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:13:34.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:34.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:13:34.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:13:34.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:13:34.341 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:13:34.359 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:34.358+0000 7f9c9f8fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:34.375+0000 7f9c9f8fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:34.377 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:34.376+0000 7f9c9f8fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:34.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:13:34.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:13:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:34.753 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:34.899 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:34.898+0000 7f9c9f8fa080 -1 Falling back to public interface 2021-01-15T18:13:35.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:35.683+0000 7f9c9f8fa080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:13:36.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:36.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:36.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:36.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:36.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:36.154 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:36.577 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/726430,v1:127.0.0.1:6803/726430] [v2:127.0.0.1:6804/726430,v1:127.0.0.1:6805/726430] exists,up 5608cfcb-b687-4b07-9359-20b2506e516d 2021-01-15T18:13:36.577 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 74954c90-6d30-4571-86d1-c9a9b16db018 2021-01-15T18:13:36.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:36.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:36.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:36.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2021-01-15T18:13:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-copy-from 2021-01-15T18:13:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:13:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:13:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:13:36.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-copy-from/1 2021-01-15T18:13:36.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 ' 2021-01-15T18:13:36.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:36.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:36.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:36.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2021-01-15T18:13:36.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2021-01-15T18:13:36.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:13:36.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:13:36.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2021-01-15T18:13:36.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:13:36.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:36.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:36.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:36.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:36.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:36.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:36.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' 2021-01-15T18:13:36.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:36.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:36.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:36.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2021-01-15T18:13:36.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2021-01-15T18:13:36.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:36.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:36.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:36.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:13:36.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:13:36.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-copy-from/1 2021-01-15T18:13:36.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:13:36.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=74954c90-6d30-4571-86d1-c9a9b16db018 2021-01-15T18:13:36.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 74954c90-6d30-4571-86d1-c9a9b16db018' 2021-01-15T18:13:36.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:13:36.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBQ2wFgMX0fIxAA6jE3JwFspygcssKLHTOs8A== 2021-01-15T18:13:36.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBQ2wFgMX0fIxAA6jE3JwFspygcssKLHTOs8A=="}' 2021-01-15T18:13:36.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 74954c90-6d30-4571-86d1-c9a9b16db018 -i td/osd-copy-from/1/new.json 2021-01-15T18:13:37.111 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:37.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-copy-from/1/new.json 2021-01-15T18:13:37.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBQ2wFgMX0fIxAA6jE3JwFspygcssKLHTOs8A== --osd-uuid 74954c90-6d30-4571-86d1-c9a9b16db018 2021-01-15T18:13:37.144 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.143+0000 7f7decf10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:37.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.145+0000 7f7decf10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:37.146 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.146+0000 7f7decf10080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:37.147 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.146+0000 7f7decf10080 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-01-15T18:13:37.147 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.146+0000 7f7decf10080 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-01-15T18:13:37.148 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.146+0000 7f7decf10080 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2021-01-15T18:13:37.195 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:37.194+0000 7f7decf10080 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2021-01-15T18:13:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-copy-from/1/keyring 2021-01-15T18:13:41.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:13:41.695 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:13:41.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:13:41.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:42.215 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:13:42.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:13:42.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=dc04a588-2479-4c37-ad20-694b265ee3df --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.725383/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:42.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:13:42.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:13:42.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:13:42.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:42.234+0000 7fde555f6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:42.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:42.236+0000 7fde555f6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:42.239 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:42.238+0000 7fde555f6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:42.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:13:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:13:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:42.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:42.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:42.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:42.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:13:42.607 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:42.758+0000 7fde555f6080 -1 Falling back to public interface 2021-01-15T18:13:43.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:43.531 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:43.531+0000 7fde555f6080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:13:44.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:44.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:13:44.033 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:44.433 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/727694,v1:127.0.0.1:6811/727694] [v2:127.0.0.1:6812/727694,v1:127.0.0.1:6813/727694] exists,up 74954c90-6d30-4571-86d1-c9a9b16db018 2021-01-15T18:13:44.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:44.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:44.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:44.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2021-01-15T18:13:44.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T18:13:44.819 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T18:13:44.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T18:13:44.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T18:13:45.377 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T18:13:45.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:13:46.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T18:13:47.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2021-01-15T18:13:47.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2021-01-15T18:13:47.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2021-01-15T18:13:47.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2021-01-15T18:13:47.851 INFO:tasks.workunit.client.0.smithi145.stdout:rbd/foo2 mtime 2021-01-15T18:13:47.000000+0000, size 790360 2021-01-15T18:13:47.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2021-01-15T18:13:47.987 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0: {} 2021-01-15T18:13:48.019 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1: {} 2021-01-15T18:13:48.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2021-01-15T18:13:48.051 INFO:tasks.workunit.client.0.smithi145.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2021-01-15T18:13:48.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2021-01-15T18:13:48.073 INFO:tasks.workunit.client.0.smithi145.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2021-01-15T18:13:48.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2021-01-15T18:13:48.201 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0: {} 2021-01-15T18:13:48.234 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1: {} 2021-01-15T18:13:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2021-01-15T18:13:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2021-01-15T18:13:48.406 INFO:tasks.workunit.client.0.smithi145.stdout:rbd/foo3 mtime 2021-01-15T18:13:47.000000+0000, size 790360 2021-01-15T18:13:48.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2021-01-15T18:13:48.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-01-15T18:13:48.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:48.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-01-15T18:13:48.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:48.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:48.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:48.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:48.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:48.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:48.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:48.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:48.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:48.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:48.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:48.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:48.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:48.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:48.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:48.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:48.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-01-15T18:13:48.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:48.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:48.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:48.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.725383 2021-01-15T18:13:48.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:48.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:48.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:13:48.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-copy-from 0 2021-01-15T18:13:48.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2021-01-15T18:13:48.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:13:48.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2021-01-15T18:13:48.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:48.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:48.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:48.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:48.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:48.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:48.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:48.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:48.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:48.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:13:48.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2021-01-15T18:13:48.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:48.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:48.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.725383 2021-01-15T18:13:48.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.725383 2021-01-15T18:13:48.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:48.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:48.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:13:48.538 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:13:48.538 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:13:48.601 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2021-01-15T18:13:48.601 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2021-01-15T18:13:48.661 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:13:48.669 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:13:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:13:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:13:48.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:13:48.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:13:48.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:13:48.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:13:48.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:13:48.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-dup 2021-01-15T18:13:48.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2021-01-15T18:13:48.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2021-01-15T18:13:48.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2021-01-15T18:13:48.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2021-01-15T18:13:48.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2021-01-15T18:13:48.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2021-01-15T18:13:48.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none ' 2021-01-15T18:13:48.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2021-01-15T18:13:48.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2021-01-15T18:13:48.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:13:48.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2021-01-15T18:13:48.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:13:48.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2021-01-15T18:13:48.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2021-01-15T18:13:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2021-01-15T18:13:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2021-01-15T18:13:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2021-01-15T18:13:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-01-15T18:13:48.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:13:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-01-15T18:13:48.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:13:48.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:13:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:13:48.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:13:48.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:13:48.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:13:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:13:48.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:13:48.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:13:48.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:13:48.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:13:48.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:13:48.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:13:48.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:13:48.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:13:48.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:13:48.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:13:48.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-01-15T18:13:48.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:13:48.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:48.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:48.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.728511 2021-01-15T18:13:48.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:13:48.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:13:48.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2021-01-15T18:13:48.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:13:48.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:48.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:48.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.728511 2021-01-15T18:13:48.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:13:48.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:13:48.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:13:48.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:13:48.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2021-01-15T18:13:48.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2021-01-15T18:13:48.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2021-01-15T18:13:48.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:13:48.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:13:48.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:13:48.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2021-01-15T18:13:48.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2021-01-15T18:13:48.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:13:48.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:48.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:48.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:48.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:48.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:48.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:48.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:13:49.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:13:49.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:13:49.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:49.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:49.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:13:49.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:13:49.119 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:49.120 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:49.121 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.728511/ceph-mon.a.asok 2021-01-15T18:13:49.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:49.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.728511/ceph-mon.a.asok config get fsid 2021-01-15T18:13:49.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:13:49.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:13:49.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:13:49.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:13:49.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:13:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:13:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.728511/ceph-mon.a.asok 2021-01-15T18:13:49.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:13:49.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.728511/ceph-mon.a.asok config get mon_host 2021-01-15T18:13:49.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2021-01-15T18:13:49.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-dup 2021-01-15T18:13:49.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:13:49.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:13:49.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:13:49.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-dup/x 2021-01-15T18:13:49.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:13:49.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:13:49.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:49.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:49.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:49.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:49.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:49.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:13:49.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2021-01-15T18:13:49.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2021-01-15T18:13:49.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T18:13:49.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=0 2021-01-15T18:13:49.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T18:13:49.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/0 2021-01-15T18:13:49.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:13:49.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:49.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:49.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:49.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2021-01-15T18:13:49.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-01-15T18:13:49.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T18:13:49.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T18:13:49.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-01-15T18:13:49.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T18:13:49.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:49.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:49.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:49.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:49.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-01-15T18:13:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-01-15T18:13:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T18:13:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T18:13:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/0 2021-01-15T18:13:49.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T18:13:49.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:13:49.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd 864cb94b-04c0-43dc-9c4d-36c360d10237' 2021-01-15T18:13:49.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T18:13:49.851 INFO:tasks.workunit.client.0.smithi145.stdout:add osd 864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:13:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQBd2wFgMqsSMxAAPH+y0Djx3nQTgqjAAqHmew== 2021-01-15T18:13:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQBd2wFgMqsSMxAAPH+y0Djx3nQTgqjAAqHmew=="}' 2021-01-15T18:13:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 864cb94b-04c0-43dc-9c4d-36c360d10237 -i td/osd-dup/0/new.json 2021-01-15T18:13:50.192 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:50.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/0/new.json 2021-01-15T18:13:50.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 0 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBd2wFgMqsSMxAAPH+y0Djx3nQTgqjAAqHmew== --osd-uuid 864cb94b-04c0-43dc-9c4d-36c360d10237 --osd-objectstore=filestore 2021-01-15T18:13:50.226 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:50.226+0000 7f938c177080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:50.228 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:50.228+0000 7f938c177080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:50.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:50.228+0000 7f938c177080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:50.651 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:50.650+0000 7f938c177080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:51.020 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.019+0000 7f938c177080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:51.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.020+0000 7f938c177080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:51.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.020+0000 7f938c177080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:51.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2021-01-15T18:13:51.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T18:13:51.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd0 key to auth repository 2021-01-15T18:13:51.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:51.497 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:13:51.680 INFO:tasks.workunit.client.0.smithi145.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-01-15T18:13:51.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.0 2021-01-15T18:13:51.682 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:13:51.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 0 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T18:13:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T18:13:51.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T18:13:51.704 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.703+0000 7effeacf1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:51.706 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.705+0000 7effeacf1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:51.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.707+0000 7effeacf1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:51.709 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.708+0000 7effeacf1080 -1 Falling back to public interface 2021-01-15T18:13:51.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 0 2021-01-15T18:13:51.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:51.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:13:51.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:51.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:51.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:51.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:51.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:51.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:51.996 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:52.000 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.999+0000 7effeacf1080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:52.000 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.999+0000 7effeacf1080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:13:52.001 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:51.999+0000 7effeacf1080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:13:52.006 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:52.006+0000 7effeacf1080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:13:52.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:53.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:53.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:53.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:13:53.274 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:53.539 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/728832,v1:127.0.0.1:6801/728832] [v2:127.0.0.1:6802/728832,v1:127.0.0.1:6803/728832] exists,up 864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:13:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:53.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:53.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2021-01-15T18:13:53.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=728832 2021-01-15T18:13:53.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2021-01-15T18:13:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2021-01-15T18:13:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T18:13:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=1 2021-01-15T18:13:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T18:13:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/1 2021-01-15T18:13:53.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:13:53.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:53.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:53.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2021-01-15T18:13:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2021-01-15T18:13:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T18:13:53.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T18:13:53.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-01-15T18:13:53.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T18:13:53.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:53.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:53.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:53.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:53.548 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:53.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:53.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:53.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:53.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:53.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:53.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-01-15T18:13:53.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-01-15T18:13:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:53.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T18:13:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T18:13:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/1 2021-01-15T18:13:53.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T18:13:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=a58be70b-aded-4240-a391-6adc7b207b3c 2021-01-15T18:13:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd a58be70b-aded-4240-a391-6adc7b207b3c' 2021-01-15T18:13:53.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T18:13:53.552 INFO:tasks.workunit.client.0.smithi145.stdout:add osd a58be70b-aded-4240-a391-6adc7b207b3c 2021-01-15T18:13:53.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQBh2wFgpCwhIRAAYfkngjoSSoEkJjesHN9X+w== 2021-01-15T18:13:53.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQBh2wFgpCwhIRAAYfkngjoSSoEkJjesHN9X+w=="}' 2021-01-15T18:13:53.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new a58be70b-aded-4240-a391-6adc7b207b3c -i td/osd-dup/1/new.json 2021-01-15T18:13:53.931 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:53.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/1/new.json 2021-01-15T18:13:53.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 1 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBh2wFgpCwhIRAAYfkngjoSSoEkJjesHN9X+w== --osd-uuid a58be70b-aded-4240-a391-6adc7b207b3c --osd-objectstore=filestore 2021-01-15T18:13:53.966 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:53.965+0000 7fdd564ea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:53.968 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:53.967+0000 7fdd564ea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:53.970 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:53.969+0000 7fdd564ea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:54.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:54.340+0000 7fdd564ea080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:54.717+0000 7fdd564ea080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:54.718 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:54.717+0000 7fdd564ea080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:54.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:54.717+0000 7fdd564ea080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:55.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2021-01-15T18:13:55.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T18:13:55.218 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:13:55.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd1 key to auth repository 2021-01-15T18:13:55.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:55.558 INFO:tasks.workunit.client.0.smithi145.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-01-15T18:13:55.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.1 2021-01-15T18:13:55.560 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:13:55.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 1 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:55.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T18:13:55.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T18:13:55.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T18:13:55.581 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.581+0000 7f4f1d74e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:55.583 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.583+0000 7f4f1d74e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:55.585 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.584+0000 7f4f1d74e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:55.586 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.586+0000 7f4f1d74e080 -1 Falling back to public interface 2021-01-15T18:13:55.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.955+0000 7f4f1d74e080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.955+0000 7f4f1d74e080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:13:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.955+0000 7f4f1d74e080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:13:55.962 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:55.961+0000 7f4f1d74e080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:13:56.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 1 2021-01-15T18:13:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:13:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:13:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:13:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:13:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:13:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:13:56.048 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:13:56.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:13:57.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:13:57.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:13:57.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:13:57.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:13:57.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:13:57.447 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:13:57.850 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/729191,v1:127.0.0.1:6811/729191] [v2:127.0.0.1:6812/729191,v1:127.0.0.1:6813/729191] exists,up a58be70b-aded-4240-a391-6adc7b207b3c 2021-01-15T18:13:57.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:13:57.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:13:57.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:13:57.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2021-01-15T18:13:57.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-dup 2021-01-15T18:13:57.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T18:13:57.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=2 2021-01-15T18:13:57.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T18:13:57.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-dup/2 2021-01-15T18:13:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:13:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:13:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:13:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:13:57.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2021-01-15T18:13:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2021-01-15T18:13:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T18:13:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T18:13:57.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2021-01-15T18:13:57.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T18:13:57.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:13:57.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:13:57.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:13:57.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:13:57.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:13:57.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:57.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:13:57.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T18:13:57.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T18:13:57.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:13:57.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T18:13:57.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T18:13:57.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-dup/2 2021-01-15T18:13:57.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T18:13:57.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=d0c320f3-9f8d-4860-9e7c-b7358511e6a1 2021-01-15T18:13:57.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd d0c320f3-9f8d-4860-9e7c-b7358511e6a1' 2021-01-15T18:13:57.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T18:13:57.860 INFO:tasks.workunit.client.0.smithi145.stdout:add osd d0c320f3-9f8d-4860-9e7c-b7358511e6a1 2021-01-15T18:13:57.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQBl2wFglMSbMxAA2ZEE+t1L4n1eTr1rvlvZcQ== 2021-01-15T18:13:57.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQBl2wFglMSbMxAA2ZEE+t1L4n1eTr1rvlvZcQ=="}' 2021-01-15T18:13:57.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new d0c320f3-9f8d-4860-9e7c-b7358511e6a1 -i td/osd-dup/2/new.json 2021-01-15T18:13:58.374 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:13:58.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-dup/2/new.json 2021-01-15T18:13:58.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 2 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBl2wFglMSbMxAA2ZEE+t1L4n1eTr1rvlvZcQ== --osd-uuid d0c320f3-9f8d-4860-9e7c-b7358511e6a1 --osd-objectstore=filestore 2021-01-15T18:13:58.409 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:58.409+0000 7f0d01ce4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:58.411 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:58.410+0000 7f0d01ce4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:58.412 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:58.412+0000 7f0d01ce4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:13:58.706 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:58.705+0000 7f0d01ce4080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:59.117 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:59.117+0000 7f0d01ce4080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:13:59.118 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:59.117+0000 7f0d01ce4080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:59.118 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:13:59.117+0000 7f0d01ce4080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T18:13:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2021-01-15T18:13:59.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T18:13:59.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd2 key to auth repository 2021-01-15T18:13:59.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:13:59.645 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:14:00.007 INFO:tasks.workunit.client.0.smithi145.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2021-01-15T18:14:00.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.2 2021-01-15T18:14:00.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 2 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:14:00.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T18:14:00.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T18:14:00.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T18:14:00.012 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:14:00.030 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.029+0000 7fe23d6f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:00.031 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.031+0000 7fe23d6f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:00.033 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.032+0000 7fe23d6f5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:00.034 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.034+0000 7fe23d6f5080 -1 Falling back to public interface 2021-01-15T18:14:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.347+0000 7fe23d6f5080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T18:14:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.347+0000 7fe23d6f5080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:14:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.347+0000 7fe23d6f5080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T18:14:00.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:00.353+0000 7fe23d6f5080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:14:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 2 2021-01-15T18:14:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:14:00.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:14:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:14:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:14:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:14:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:14:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:14:00.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:14:00.547 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:14:00.947 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/729484,v1:127.0.0.1:6819/729484] [v2:127.0.0.1:6820/729484,v1:127.0.0.1:6821/729484] exists,up d0c320f3-9f8d-4860-9e7c-b7358511e6a1 2021-01-15T18:14:00.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:14:00.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:14:00.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:14:00.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2021-01-15T18:14:05.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2021-01-15T18:14:05.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create foo 16 2021-01-15T18:14:06.729 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T18:14:06.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:14:07.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2021-01-15T18:14:07.771 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:14:07.771 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2021-01-15T18:14:07.772 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_729644 2021-01-15T18:14:07.772 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:14:07.772 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:14:08.776 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 16 0 0 0 - 0 2021-01-15T18:14:09.771 INFO:tasks.workunit.client.0.smithi145.stdout: 2 16 16 0 0 0 - 0 2021-01-15T18:14:10.771 INFO:tasks.workunit.client.0.smithi145.stdout: 3 16 18 2 0.00260388 0.00260417 2.74281 2.52836 2021-01-15T18:14:11.771 INFO:tasks.workunit.client.0.smithi145.stdout: 4 16 22 6 0.00585867 0.015625 3.40475 3.10282 2021-01-15T18:14:12.771 INFO:tasks.workunit.client.0.smithi145.stdout: 5 16 22 6 0.00468692 0 - 3.10282 2021-01-15T18:14:13.771 INFO:tasks.workunit.client.0.smithi145.stdout: 6 16 89 73 0.0475201 0.130859 0.162776 1.29413 2021-01-15T18:14:14.771 INFO:tasks.workunit.client.0.smithi145.stdout: 7 16 230 214 0.119405 0.550781 0.122701 0.522003 2021-01-15T18:14:15.771 INFO:tasks.workunit.client.0.smithi145.stdout: 8 16 361 345 0.168437 0.511719 0.128681 0.366565 2021-01-15T18:14:16.772 INFO:tasks.workunit.client.0.smithi145.stdout: 9 16 441 425 0.18444 0.3125 0.130739 0.335562 2021-01-15T18:14:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: 10 16 537 521 0.203492 0.375 0.129542 0.30345 2021-01-15T18:14:18.772 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 10.0602 2021-01-15T18:14:18.772 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 537 2021-01-15T18:14:18.773 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:14:18.773 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:14:18.774 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.20851 2021-01-15T18:14:18.774 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0.226131 2021-01-15T18:14:18.774 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.550781 2021-01-15T18:14:18.775 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0 2021-01-15T18:14:18.775 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 53 2021-01-15T18:14:18.776 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 57.966 2021-01-15T18:14:18.776 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 141 2021-01-15T18:14:18.776 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 0 2021-01-15T18:14:18.777 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.298672 2021-01-15T18:14:18.777 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 0.783918 2021-01-15T18:14:18.777 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 5.27149 2021-01-15T18:14:18.778 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.10244 2021-01-15T18:14:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 728832 2021-01-15T18:14:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2021-01-15T18:14:19.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 728832 2021-01-15T18:14:19.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (728832) - No such process 2021-01-15T18:14:19.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2021-01-15T18:14:20.277 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:14:20.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2021-01-15T18:14:20.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2021-01-15T18:14:20.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2021-01-15T18:14:20.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:14:20.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid 864cb94b-04c0-43dc-9c4d-36c360d10237' 2021-01-15T18:14:20.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:14:20.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2021-01-15T18:14:20.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid 864cb94b-04c0-43dc-9c4d-36c360d10237 --op mkfs --no-mon-config 2021-01-15T18:14:20.291 INFO:tasks.workunit.client.0.smithi145.stdout:osd fsid 864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:14:23.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2021-01-15T18:14:23.749 INFO:tasks.workunit.client.0.smithi145.stdout:dup from filestore: td/osd-dup/0.old 2021-01-15T18:14:23.750 INFO:tasks.workunit.client.0.smithi145.stdout: to bluestore: td/osd-dup/0 2021-01-15T18:14:24.589 INFO:tasks.workunit.client.0.smithi145.stdout:fsid 864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:14:24.589 INFO:tasks.workunit.client.0.smithi145.stdout:17 collections 2021-01-15T18:14:24.589 INFO:tasks.workunit.client.0.smithi145.stdout:1/17 1.1_head 2021-01-15T18:14:25.298 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 43 objects, 172032 bytes, 48 keys 2021-01-15T18:14:25.316 INFO:tasks.workunit.client.0.smithi145.stdout:2/17 1.0_head 2021-01-15T18:14:25.634 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2021-01-15T18:14:25.634 INFO:tasks.workunit.client.0.smithi145.stdout:3/17 1.b_head 2021-01-15T18:14:25.961 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 37 objects, 147456 bytes, 42 keys 2021-01-15T18:14:25.962 INFO:tasks.workunit.client.0.smithi145.stdout:4/17 1.d_head 2021-01-15T18:14:26.247 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 36 objects, 139288 bytes, 41 keys 2021-01-15T18:14:26.247 INFO:tasks.workunit.client.0.smithi145.stdout:5/17 1.6_head 2021-01-15T18:14:26.470 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2021-01-15T18:14:26.471 INFO:tasks.workunit.client.0.smithi145.stdout:6/17 1.7_head 2021-01-15T18:14:26.746 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2021-01-15T18:14:26.746 INFO:tasks.workunit.client.0.smithi145.stdout:7/17 1.c_head 2021-01-15T18:14:26.969 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 31 objects, 122880 bytes, 36 keys 2021-01-15T18:14:27.158 INFO:tasks.workunit.client.0.smithi145.stdout:8/17 1.2_head 2021-01-15T18:14:27.159 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 52 objects, 208896 bytes, 57 keys 2021-01-15T18:14:27.159 INFO:tasks.workunit.client.0.smithi145.stdout:9/17 1.9_head 2021-01-15T18:14:27.205 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 36 objects, 143360 bytes, 41 keys 2021-01-15T18:14:27.205 INFO:tasks.workunit.client.0.smithi145.stdout:10/17 1.3_head 2021-01-15T18:14:27.391 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2021-01-15T18:14:27.391 INFO:tasks.workunit.client.0.smithi145.stdout:11/17 1.e_head 2021-01-15T18:14:27.616 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 27 objects, 106496 bytes, 32 keys 2021-01-15T18:14:27.617 INFO:tasks.workunit.client.0.smithi145.stdout:12/17 1.4_head 2021-01-15T18:14:27.918 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2021-01-15T18:14:27.919 INFO:tasks.workunit.client.0.smithi145.stdout:13/17 1.f_head 2021-01-15T18:14:28.301 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 40 objects, 159744 bytes, 45 keys 2021-01-15T18:14:28.301 INFO:tasks.workunit.client.0.smithi145.stdout:14/17 1.a_head 2021-01-15T18:14:28.527 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2021-01-15T18:14:28.527 INFO:tasks.workunit.client.0.smithi145.stdout:15/17 1.5_head 2021-01-15T18:14:28.833 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2021-01-15T18:14:28.834 INFO:tasks.workunit.client.0.smithi145.stdout:16/17 meta 2021-01-15T18:14:28.837 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47104 bytes, 0 keys 2021-01-15T18:14:28.837 INFO:tasks.workunit.client.0.smithi145.stdout:17/17 1.8_head 2021-01-15T18:14:29.029 INFO:tasks.workunit.client.0.smithi145.stdout: 0 objects, 0 bytes, 0 keys 23 objects, 90112 bytes, 28 keys 2021-01-15T18:14:29.029 INFO:tasks.workunit.client.0.smithi145.stdout:keyring 2021-01-15T18:14:29.030 INFO:tasks.workunit.client.0.smithi145.stdout:duping osd metadata 2021-01-15T18:14:29.687 INFO:tasks.workunit.client.0.smithi145.stdout:done. 2021-01-15T18:14:30.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:14:30.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2021-01-15T18:14:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-dup 2021-01-15T18:14:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:14:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:14:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:14:30.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-dup/0 2021-01-15T18:14:30.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2021-01-15T18:14:30.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:14:30.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:14:30.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:14:30.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2021-01-15T18:14:30.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2021-01-15T18:14:30.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:14:30.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:14:30.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2021-01-15T18:14:30.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:14:30.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:14:30.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:14:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:14:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:14:30.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:14:30.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' 2021-01-15T18:14:30.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:14:30.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2021-01-15T18:14:30.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2021-01-15T18:14:30.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:14:30.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:14:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:14:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:14:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:14:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-dup/0 2021-01-15T18:14:30.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:14:30.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=627b733e-b366-4ac2-bf89-44ee3fb52822 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.728511/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:14:30.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-dup/0/whoami 2021-01-15T18:14:30.233 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:14:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:14:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:14:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:14:30.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:14:30.250 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:30.249+0000 7fc8e3cd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:30.252 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:30.251+0000 7fc8e3cd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:30.254 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:30.253+0000 7fc8e3cd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:14:30.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:14:30.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:14:30.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:14:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:14:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:14:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:14:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:14:30.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:14:30.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:14:30.636 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:14:30.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:30.769+0000 7fc8e3cd1080 -1 Falling back to public interface 2021-01-15T18:14:31.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:14:32.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:14:32.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:14:32.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:14:32.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:14:32.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:14:32.025 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:14:32.286 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:32.285+0000 7fc8e3cd1080 -1 osd.0 21 log_to_monitors {default=true} 2021-01-15T18:14:32.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:14:33.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:14:33.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:14:33.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:14:33.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:14:33.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:14:33.443 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:14:33.856 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/730467,v1:127.0.0.1:6801/730467] [v2:127.0.0.1:6802/730467,v1:127.0.0.1:6803/730467] exists,up 864cb94b-04c0-43dc-9c4d-36c360d10237 2021-01-15T18:14:33.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:14:33.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:14:33.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:14:33.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2021-01-15T18:14:33.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2021-01-15T18:14:34.245 INFO:tasks.workunit.client.0.smithi145.stdout:3 osds: 3 up (since 1.8561s), 3 in (since 33s); epoch: e27 2021-01-15T18:14:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2021-01-15T18:14:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2021-01-15T18:14:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T18:14:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T18:14:34.644 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T18:14:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T18:14:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2021-01-15T18:14:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T18:14:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T18:14:34.645 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T18:14:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T18:14:34.646 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T18:14:34.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2021-01-15T18:14:35.029 INFO:tasks.workunit.client.0.smithi145.stderr:instructed osd(s) 0 to scrub 2021-01-15T18:14:35.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2021-01-15T18:14:50.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2021-01-15T18:14:50.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:14:50.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:14:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:14:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:14:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:14:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:14:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:50.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:14:50.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T18:14:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T18:14:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406' 2021-01-15T18:14:50.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:50.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:14:50.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672973 2021-01-15T18:14:50.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672973 2021-01-15T18:14:50.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-42949672973' 2021-01-15T18:14:50.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:50.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:14:50.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509453 2021-01-15T18:14:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509453 2021-01-15T18:14:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-42949672973 2-64424509453' 2021-01-15T18:14:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:50.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-107374182406 2021-01-15T18:14:50.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:14:50.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182406 2021-01-15T18:14:50.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T18:14:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2021-01-15T18:14:50.852 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 107374182406 2021-01-15T18:14:50.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:14:51.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182404 -lt 107374182406 2021-01-15T18:14:51.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:14:52.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:14:52.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:14:52.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2021-01-15T18:14:52.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:52.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672973 2021-01-15T18:14:52.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:52.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:14:52.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672973 2021-01-15T18:14:52.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:52.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672973 2021-01-15T18:14:52.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2021-01-15T18:14:52.637 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672973 2021-01-15T18:14:52.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:14:53.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672973 -lt 42949672973 2021-01-15T18:14:53.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:53.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2021-01-15T18:14:53.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:53.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:14:53.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509453 2021-01-15T18:14:53.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:53.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509453 2021-01-15T18:14:53.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2021-01-15T18:14:53.033 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509453 2021-01-15T18:14:53.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:14:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509453 -lt 64424509453 2021-01-15T18:14:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2021-01-15T18:14:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:14:53.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:14:53.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:14:53.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:14:53.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:14:53.431 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:14:53.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:14:53.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:14:53.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:14:53.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:14:53.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:14:53.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:14:53.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:14:53.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:14:53.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:14:53.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:14:55.875 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:14:55.875 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:14:55.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:14:55.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:55.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:14:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182409 2021-01-15T18:14:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182409 2021-01-15T18:14:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409' 2021-01-15T18:14:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:55.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:14:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672975 2021-01-15T18:14:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672975 2021-01-15T18:14:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409 1-42949672975' 2021-01-15T18:14:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:14:55.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:14:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509456 2021-01-15T18:14:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509456 2021-01-15T18:14:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182409 1-42949672975 2-64424509456' 2021-01-15T18:14:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:55.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-107374182409 2021-01-15T18:14:55.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:14:55.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182409 2021-01-15T18:14:55.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182409 2021-01-15T18:14:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 107374182409' 2021-01-15T18:14:55.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:14:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182407 -lt 107374182409 2021-01-15T18:14:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:14:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:14:55.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:14:55.884 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 107374182409 2021-01-15T18:14:56.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182409 -lt 107374182409 2021-01-15T18:14:56.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:56.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2021-01-15T18:14:56.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:56.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:14:56.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672975 2021-01-15T18:14:56.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:56.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672975 2021-01-15T18:14:56.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-01-15T18:14:56.102 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672975 2021-01-15T18:14:56.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:14:56.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672976 -lt 42949672975 2021-01-15T18:14:56.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:14:56.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509456 2021-01-15T18:14:56.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:14:56.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:14:56.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509456 2021-01-15T18:14:56.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:14:56.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509456 2021-01-15T18:14:56.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2021-01-15T18:14:56.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:14:56.509 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509456 2021-01-15T18:14:56.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509456 -lt 64424509456 2021-01-15T18:14:56.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:14:56.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:14:56.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:14:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 16 == 0 2021-01-15T18:14:57.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:14:57.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:14:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:14:57.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:14:57.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:14:57.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:14:57.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:14:57.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=16 2021-01-15T18:14:57.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:14:57.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:14:57.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:14:58.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 16 = 16 2021-01-15T18:14:58.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:14:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:14:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2021-01-15T18:14:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-01-15T18:14:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:14:58.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-01-15T18:14:58.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:14:58.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:14:58.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:14:58.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:14:58.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:14:58.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:14:58.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:14:58.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:14:58.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:14:58.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:14:58.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:14:58.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:14:58.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:14:58.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:14:58.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:14:58.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:14:58.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-01-15T18:14:58.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:14:58.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:58.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:14:58.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.728511 2021-01-15T18:14:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:14:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:14:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:14:58.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-dup 0 2021-01-15T18:14:58.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2021-01-15T18:14:58.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:14:58.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2021-01-15T18:14:58.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:14:58.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:14:58.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:14:58.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:14:58.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:14:58.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:14:58.501 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:14:58.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:14:58.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:14:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:14:58.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:14:58.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:14:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:14:58.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:14:58.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:14:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:14:58.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2021-01-15T18:14:58.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:14:58.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:58.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.728511 2021-01-15T18:14:58.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.728511 2021-01-15T18:14:58.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:14:58.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:14:58.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:14:58.509 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:14:58.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:14:58.572 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2021-01-15T18:14:58.572 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2021-01-15T18:14:58.632 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:14:58.639 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:14:58.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:14:58.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:14:58.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:14:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:14:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:14:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:14:58.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:14:58.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:14:58.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-fast-mark-down 2021-01-15T18:14:58.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2021-01-15T18:14:58.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2021-01-15T18:14:58.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2021-01-15T18:14:58.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2021-01-15T18:14:58.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2021-01-15T18:14:58.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2021-01-15T18:14:58.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2021-01-15T18:14:58.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none ' 2021-01-15T18:14:58.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none ' 2021-01-15T18:14:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2021-01-15T18:14:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2021-01-15T18:14:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2021-01-15T18:14:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-01-15T18:14:58.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-01-15T18:14:58.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-01-15T18:14:58.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-01-15T18:14:58.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:14:58.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-01-15T18:14:58.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:14:58.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:14:58.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:14:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:14:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:14:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:14:58.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:14:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:14:58.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:14:58.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:14:58.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:14:58.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:14:58.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:14:58.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:14:58.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:58.651 INFO:tasks.workunit.client.0.smithi145.stdout:Ensuring old behavior is there... 2021-01-15T18:14:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:14:58.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:14:58.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-01-15T18:14:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:14:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:58.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:58.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.731738 2021-01-15T18:14:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:14:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:14:58.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-01-15T18:14:58.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:14:58.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:58.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:58.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.731738 2021-01-15T18:14:58.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:14:58.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:14:58.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:14:58.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:14:58.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-01-15T18:14:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-01-15T18:14:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-01-15T18:14:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:14:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:14:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:14:58.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-01-15T18:14:58.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-01-15T18:14:58.670 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:58.669+0000 7f9363f6d700 -1 failed for service _ceph-mon._tcp 2021-01-15T18:14:58.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:58.669+0000 7f9363f6d700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-01-15T18:14:58.671 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T18:14:58.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:58.669+0000 7f9363f6d700 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2021-01-15T18:14:58.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:14:58.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:14:58.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:14:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:14:58.956 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:58.957 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:58.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:14:58.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=3 2021-01-15T18:14:59.137 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:59.136+0000 7f2b0f099700 -1 failed for service _ceph-mon._tcp 2021-01-15T18:14:59.138 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:59.136+0000 7f2b0f099700 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2021-01-15T18:14:59.138 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T18:14:59.138 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:14:59.136+0000 7f2b0f099700 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2021-01-15T18:14:59.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:484: run_mon: return 1 2021-01-15T18:14:59.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2021-01-15T18:14:59.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-01-15T18:14:59.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2021-01-15T18:14:59.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-01-15T18:14:59.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-01-15T18:14:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:14:59.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-01-15T18:14:59.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:14:59.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:14:59.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:14:59.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:14:59.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:14:59.145 INFO:tasks.workunit.client.0.smithi145.stdout:Testing async msgr... 2021-01-15T18:14:59.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:14:59.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:14:59.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:59.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:14:59.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:14:59.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:14:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:14:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:14:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-01-15T18:14:59.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:14:59.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:59.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:59.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.731738 2021-01-15T18:14:59.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:14:59.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:14:59.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2021-01-15T18:14:59.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:14:59.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:59.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:59.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.731738 2021-01-15T18:14:59.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:14:59.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:14:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:14:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2021-01-15T18:14:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2021-01-15T18:14:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2021-01-15T18:14:59.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:14:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:14:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:14:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2021-01-15T18:14:59.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2021-01-15T18:14:59.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:14:59.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:14:59.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:14:59.442 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:14:59.443 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:59.443 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:59.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:14:59.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=3 2021-01-15T18:14:59.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:14:59.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:14:59.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:14:59.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:14:59.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:14:59.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:14:59.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:14:59.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:14:59.620 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:14:59.621 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:14:59.621 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:59.622 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:59.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.731738/ceph-mon.a.asok 2021-01-15T18:14:59.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:14:59.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.731738/ceph-mon.a.asok config get fsid 2021-01-15T18:14:59.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:14:59.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:14:59.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:14:59.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:14:59.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:14:59.722 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:14:59.722 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:14:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:14:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:14:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:14:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:14:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.731738/ceph-mon.a.asok 2021-01-15T18:14:59.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:14:59.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.731738/ceph-mon.a.asok config get mon_host 2021-01-15T18:14:59.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2021-01-15T18:14:59.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-fast-mark-down 2021-01-15T18:14:59.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:14:59.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:14:59.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:14:59.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-fast-mark-down/x 2021-01-15T18:14:59.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:15:00.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:15:00.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:00.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:00.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:00.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:15:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:15:00.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-01-15T18:15:00.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2021-01-15T18:15:00.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2021-01-15T18:15:00.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:00.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:15:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/0 2021-01-15T18:15:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-01-15T18:15:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:00.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2021-01-15T18:15:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2021-01-15T18:15:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:00.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-01-15T18:15:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:00.341 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:00.342 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:00.342 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:00.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:00.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-01-15T18:15:00.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-01-15T18:15:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:00.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/0 2021-01-15T18:15:00.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:00.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3dfd5a70-e085-45b0-96ad-c37c3481488f 2021-01-15T18:15:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 3dfd5a70-e085-45b0-96ad-c37c3481488f' 2021-01-15T18:15:00.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:00.346 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 3dfd5a70-e085-45b0-96ad-c37c3481488f 2021-01-15T18:15:00.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCk2wFgqHINFRAATMxjlMNucL+E6RbWzBeJqA== 2021-01-15T18:15:00.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCk2wFgqHINFRAATMxjlMNucL+E6RbWzBeJqA=="}' 2021-01-15T18:15:00.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3dfd5a70-e085-45b0-96ad-c37c3481488f -i td/osd-fast-mark-down/0/new.json 2021-01-15T18:15:00.710 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/0/new.json 2021-01-15T18:15:00.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCk2wFgqHINFRAATMxjlMNucL+E6RbWzBeJqA== --osd-uuid 3dfd5a70-e085-45b0-96ad-c37c3481488f 2021-01-15T18:15:00.747 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.746+0000 7f43ae721080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:00.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.748+0000 7f43ae721080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:00.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.750+0000 7f43ae721080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:00.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.751+0000 7f43ae721080 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-01-15T18:15:00.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.751+0000 7f43ae721080 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-01-15T18:15:00.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.751+0000 7f43ae721080 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2021-01-15T18:15:00.792 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:00.791+0000 7f43ae721080 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2021-01-15T18:15:05.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2021-01-15T18:15:05.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:05.405 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:15:05.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:15:05.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:05.930 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:15:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:15:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:05.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:05.953 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:05.952+0000 7facd6fbd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:05.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:05.954+0000 7facd6fbd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:05.955+0000 7facd6fbd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:06.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:15:06.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:06.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:15:06.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:06.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:06.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:06.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:06.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:06.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:15:06.322 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:06.472 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:06.472+0000 7facd6fbd080 -1 Falling back to public interface 2021-01-15T18:15:06.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:07.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:07.244+0000 7facd6fbd080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:15:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:07.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:07.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:15:07.713 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:08.134 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/732846,v1:127.0.0.1:6803/732846] [v2:127.0.0.1:6804/732846,v1:127.0.0.1:6805/732846] exists,up 3dfd5a70-e085-45b0-96ad-c37c3481488f 2021-01-15T18:15:08.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:08.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:08.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:08.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2021-01-15T18:15:08.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=732846 2021-01-15T18:15:08.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-01-15T18:15:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2021-01-15T18:15:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2021-01-15T18:15:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:15:08.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/1 2021-01-15T18:15:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-01-15T18:15:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:08.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2021-01-15T18:15:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2021-01-15T18:15:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-01-15T18:15:08.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:08.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:08.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:08.142 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:08.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-01-15T18:15:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-01-15T18:15:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/1 2021-01-15T18:15:08.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:08.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5a6ba200-05a3-4e88-89a7-aca17f35eb6a 2021-01-15T18:15:08.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 5a6ba200-05a3-4e88-89a7-aca17f35eb6a' 2021-01-15T18:15:08.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:08.146 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 5a6ba200-05a3-4e88-89a7-aca17f35eb6a 2021-01-15T18:15:08.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCs2wFgyoAVCRAA7CGampKnM0c8cCgpx6QLfw== 2021-01-15T18:15:08.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCs2wFgyoAVCRAA7CGampKnM0c8cCgpx6QLfw=="}' 2021-01-15T18:15:08.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5a6ba200-05a3-4e88-89a7-aca17f35eb6a -i td/osd-fast-mark-down/1/new.json 2021-01-15T18:15:08.641 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:08.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/1/new.json 2021-01-15T18:15:08.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCs2wFgyoAVCRAA7CGampKnM0c8cCgpx6QLfw== --osd-uuid 5a6ba200-05a3-4e88-89a7-aca17f35eb6a 2021-01-15T18:15:08.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.678+0000 7f1acd35c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:08.680 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.680+0000 7f1acd35c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:08.681 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.681+0000 7f1acd35c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:08.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.681+0000 7f1acd35c080 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-01-15T18:15:08.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.681+0000 7f1acd35c080 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-01-15T18:15:08.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.681+0000 7f1acd35c080 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2021-01-15T18:15:08.733 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:08.733+0000 7f1acd35c080 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2021-01-15T18:15:13.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2021-01-15T18:15:13.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:13.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:15:13.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:13.293 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:15:13.819 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:15:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:15:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:13.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:13.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:13.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:13.841+0000 7f1b6590c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:13.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:13.843+0000 7f1b6590c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:13.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:13.844+0000 7f1b6590c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:14.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:15:14.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:14.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:15:14.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:14.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:15:14.216 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:14.362 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:14.361+0000 7f1b6590c080 -1 Falling back to public interface 2021-01-15T18:15:14.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:15.166 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:15.166+0000 7f1b6590c080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:15:15.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:15.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:15.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:15.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:15.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:15:15.633 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:16.040 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/734111,v1:127.0.0.1:6811/734111] [v2:127.0.0.1:6812/734111,v1:127.0.0.1:6813/734111] exists,up 5a6ba200-05a3-4e88-89a7-aca17f35eb6a 2021-01-15T18:15:16.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:16.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:16.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:16.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2021-01-15T18:15:16.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=734111 2021-01-15T18:15:16.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2021-01-15T18:15:16.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2021-01-15T18:15:16.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-fast-mark-down 2021-01-15T18:15:16.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:16.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:15:16.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:16.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-fast-mark-down/2 2021-01-15T18:15:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2021-01-15T18:15:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:16.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2021-01-15T18:15:16.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2021-01-15T18:15:16.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:16.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:16.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2021-01-15T18:15:16.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:16.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:16.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:16.048 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:16.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:16.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:16.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:16.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' 2021-01-15T18:15:16.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2021-01-15T18:15:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2021-01-15T18:15:16.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:16.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-fast-mark-down/2 2021-01-15T18:15:16.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:16.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=ed9d5649-97df-4be6-877d-b70918c56252 2021-01-15T18:15:16.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 ed9d5649-97df-4be6-877d-b70918c56252' 2021-01-15T18:15:16.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:16.053 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 ed9d5649-97df-4be6-877d-b70918c56252 2021-01-15T18:15:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC02wFgcPudAxAAP7KRJeVvKxhQ7LQ3qDJOUg== 2021-01-15T18:15:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC02wFgcPudAxAAP7KRJeVvKxhQ7LQ3qDJOUg=="}' 2021-01-15T18:15:16.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new ed9d5649-97df-4be6-877d-b70918c56252 -i td/osd-fast-mark-down/2/new.json 2021-01-15T18:15:16.585 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:15:16.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-fast-mark-down/2/new.json 2021-01-15T18:15:16.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC02wFgcPudAxAAP7KRJeVvKxhQ7LQ3qDJOUg== --osd-uuid ed9d5649-97df-4be6-877d-b70918c56252 2021-01-15T18:15:16.622 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.621+0000 7f07d1389080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:16.624 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.623+0000 7f07d1389080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:16.625 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.625+0000 7f07d1389080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.625+0000 7f07d1389080 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-01-15T18:15:16.626 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.625+0000 7f07d1389080 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-01-15T18:15:16.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.625+0000 7f07d1389080 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2021-01-15T18:15:16.672 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:16.671+0000 7f07d1389080 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2021-01-15T18:15:21.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2021-01-15T18:15:21.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:21.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:15:21.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:21.291 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:15:21.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:15:21.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:21.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=c7fb0b2a-0e4c-4646-841d-8fa5cb68e474 --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.731738/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:21.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:21.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:21.818 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:15:21.839 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:21.839+0000 7f3b3d5cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:21.871 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:21.870+0000 7f3b3d5cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:21.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:21.871+0000 7f3b3d5cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:15:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:22.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:15:22.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:22.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:22.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:22.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:22.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:22.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:15:22.212 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:22.387 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:22.386+0000 7f3b3d5cd080 -1 Falling back to public interface 2021-01-15T18:15:22.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:23.181 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:23.180+0000 7f3b3d5cd080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:15:23.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:23.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:15:23.605 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:24.038 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/735376,v1:127.0.0.1:6819/735376] [v2:127.0.0.1:6820/735376,v1:127.0.0.1:6821/735376] exists,up ed9d5649-97df-4be6-877d-b70918c56252 2021-01-15T18:15:24.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:24.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:24.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:24.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2021-01-15T18:15:24.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=735376 2021-01-15T18:15:24.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2021-01-15T18:15:24.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T18:15:24.492 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T18:15:24.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T18:15:24.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T18:15:25.043 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T18:15:25.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:15:26.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T18:15:27.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2021-01-15T18:15:27.672 INFO:tasks.workunit.client.0.smithi145.stdout:hints = 1 2021-01-15T18:15:27.672 INFO:tasks.workunit.client.0.smithi145.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2021-01-15T18:15:27.673 INFO:tasks.workunit.client.0.smithi145.stdout:Object prefix: benchmark_data_smithi145_735918 2021-01-15T18:15:27.673 INFO:tasks.workunit.client.0.smithi145.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-01-15T18:15:27.673 INFO:tasks.workunit.client.0.smithi145.stdout: 0 0 0 0 0 0 - 0 2021-01-15T18:15:28.671 INFO:tasks.workunit.client.0.smithi145.stdout: 1 16 80 64 0.249967 0.25 0.259169 0.238553 2021-01-15T18:15:29.671 INFO:tasks.workunit.client.0.smithi145.stdout:Total time run: 1.92726 2021-01-15T18:15:29.672 INFO:tasks.workunit.client.0.smithi145.stdout:Total writes made: 128 2021-01-15T18:15:29.672 INFO:tasks.workunit.client.0.smithi145.stdout:Write size: 4096 2021-01-15T18:15:29.672 INFO:tasks.workunit.client.0.smithi145.stdout:Object size: 4096 2021-01-15T18:15:29.672 INFO:tasks.workunit.client.0.smithi145.stdout:Bandwidth (MB/sec): 0.259436 2021-01-15T18:15:29.673 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Bandwidth: 0 2021-01-15T18:15:29.673 INFO:tasks.workunit.client.0.smithi145.stdout:Max bandwidth (MB/sec): 0.25 2021-01-15T18:15:29.673 INFO:tasks.workunit.client.0.smithi145.stdout:Min bandwidth (MB/sec): 0.25 2021-01-15T18:15:29.673 INFO:tasks.workunit.client.0.smithi145.stdout:Average IOPS: 66 2021-01-15T18:15:29.673 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev IOPS: 0 2021-01-15T18:15:29.674 INFO:tasks.workunit.client.0.smithi145.stdout:Max IOPS: 64 2021-01-15T18:15:29.674 INFO:tasks.workunit.client.0.smithi145.stdout:Min IOPS: 64 2021-01-15T18:15:29.674 INFO:tasks.workunit.client.0.smithi145.stdout:Average Latency(s): 0.239826 2021-01-15T18:15:29.674 INFO:tasks.workunit.client.0.smithi145.stdout:Stddev Latency(s): 0.0239917 2021-01-15T18:15:29.674 INFO:tasks.workunit.client.0.smithi145.stdout:Max latency(s): 0.270155 2021-01-15T18:15:29.675 INFO:tasks.workunit.client.0.smithi145.stdout:Min latency(s): 0.15161 2021-01-15T18:15:29.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2021-01-15T18:15:30.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2021-01-15T18:15:30.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2021-01-15T18:15:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-01-15T18:15:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2021-01-15T18:15:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=735376 2021-01-15T18:15:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 735376 -eq 0 ']' 2021-01-15T18:15:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=735376 2021-01-15T18:15:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 735376 2021-01-15T18:15:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-01-15T18:15:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-01-15T18:15:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-01-15T18:15:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-01-15T18:15:31.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-01-15T18:15:31.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-01-15T18:15:31.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-01-15T18:15:31.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-01-15T18:15:31.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-01-15T18:15:32.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2021-01-15T18:15:32.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2021-01-15T18:15:32.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2021-01-15T18:15:32.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-01-15T18:15:32.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2021-01-15T18:15:32.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2021-01-15T18:15:32.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 735376 -eq 735376 ']' 2021-01-15T18:15:32.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=732846 2021-01-15T18:15:32.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 732846 -eq 735376 ']' 2021-01-15T18:15:32.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=732846 2021-01-15T18:15:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 732846 2021-01-15T18:15:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2021-01-15T18:15:32.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2021-01-15T18:15:32.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2021-01-15T18:15:32.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2021-01-15T18:15:33.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2021-01-15T18:15:33.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2021-01-15T18:15:33.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2021-01-15T18:15:33.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2021-01-15T18:15:33.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2021-01-15T18:15:33.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2021-01-15T18:15:33.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2021-01-15T18:15:33.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2021-01-15T18:15:33.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2021-01-15T18:15:33.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2021-01-15T18:15:33.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2021-01-15T18:15:33.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2021-01-15T18:15:33.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-01-15T18:15:33.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:15:33.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-01-15T18:15:33.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:15:33.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:15:33.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:15:33.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:15:33.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:15:33.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:15:33.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:15:33.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:15:33.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:15:33.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:15:33.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:15:33.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:15:33.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:15:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:15:33.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:15:33.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:15:33.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-01-15T18:15:33.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:15:33.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:33.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:33.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.731738 2021-01-15T18:15:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:15:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:15:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2021-01-15T18:15:33.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:15:33.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-fast-mark-down 0 2021-01-15T18:15:33.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2021-01-15T18:15:33.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:15:33.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2021-01-15T18:15:33.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:15:33.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:15:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:15:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:15:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:15:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:15:33.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:15:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:15:33.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:15:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:15:33.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:15:33.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:15:33.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:15:33.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:15:33.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:15:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:15:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2021-01-15T18:15:33.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:15:33.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:33.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.731738 2021-01-15T18:15:33.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.731738 2021-01-15T18:15:33.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:15:33.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:15:33.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:15:33.755 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:15:33.756 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:15:33.815 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2021-01-15T18:15:33.816 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2021-01-15T18:15:33.875 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:15:33.883 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:15:33.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:15:33.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:15:33.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:15:33.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:15:33.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:15:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:15:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:15:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:15:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-force-create-pg 2021-01-15T18:15:33.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2021-01-15T18:15:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2021-01-15T18:15:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2021-01-15T18:15:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2021-01-15T18:15:33.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2021-01-15T18:15:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2021-01-15T18:15:33.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none ' 2021-01-15T18:15:33.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2021-01-15T18:15:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2021-01-15T18:15:33.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:15:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2021-01-15T18:15:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2021-01-15T18:15:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: TEST_reuse_id td/osd-force-create-pg 2021-01-15T18:15:33.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:20: TEST_reuse_id: local dir=td/osd-force-create-pg 2021-01-15T18:15:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: setup td/osd-force-create-pg 2021-01-15T18:15:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2021-01-15T18:15:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2021-01-15T18:15:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-01-15T18:15:33.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:15:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-01-15T18:15:33.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:15:33.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:15:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:15:33.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:15:33.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:15:33.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:15:33.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:15:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:15:33.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:15:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:15:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:15:33.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:15:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:15:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:15:33.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:15:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:15:33.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:15:33.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-01-15T18:15:33.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:15:33.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:33.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:33.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.736080 2021-01-15T18:15:33.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:15:33.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2021-01-15T18:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:33.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.736080 2021-01-15T18:15:33.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:15:33.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:15:33.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:15:33.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:15:33.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2021-01-15T18:15:33.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:23: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T18:15:33.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2021-01-15T18:15:33.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:15:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:15:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:15:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2021-01-15T18:15:33.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T18:15:34.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:15:34.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:34.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:34.189 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:34.189 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:34.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:34.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:34.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T18:15:34.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:15:34.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:15:34.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:15:34.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:15:34.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:15:34.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:15:34.375 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:15:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:15:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:15:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:15:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:34.376 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:34.377 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.736080/ceph-mon.a.asok 2021-01-15T18:15:34.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:15:34.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.736080/ceph-mon.a.asok config get fsid 2021-01-15T18:15:34.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:15:34.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:15:34.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:15:34.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:15:34.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:15:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:15:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:15:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:15:34.476 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:15:34.477 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:34.477 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:34.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.736080/ceph-mon.a.asok 2021-01-15T18:15:34.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:15:34.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.736080/ceph-mon.a.asok config get mon_host 2021-01-15T18:15:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2021-01-15T18:15:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-force-create-pg 2021-01-15T18:15:34.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:15:34.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:15:34.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:15:34.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-force-create-pg/x 2021-01-15T18:15:34.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:15:35.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:15:35.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:35.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:35.465 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:35.465 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:35.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:35.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:35.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:15:35.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:15:35.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2021-01-15T18:15:35.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2021-01-15T18:15:35.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:35.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:15:35.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:35.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/0 2021-01-15T18:15:35.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:15:35.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:35.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:35.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:35.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-01-15T18:15:35.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-01-15T18:15:35.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:35.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:35.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:15:35.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:35.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:35.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:35.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:15:35.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:15:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/0 2021-01-15T18:15:35.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=146bf154-6c9f-42e4-95db-cfc1e11fe571 2021-01-15T18:15:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 146bf154-6c9f-42e4-95db-cfc1e11fe571' 2021-01-15T18:15:35.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:35.494 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 146bf154-6c9f-42e4-95db-cfc1e11fe571 2021-01-15T18:15:35.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDH2wFgv7yWHRAAgPf+UAaV3Y8kbw6xKP+Xrg== 2021-01-15T18:15:35.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDH2wFgv7yWHRAAgPf+UAaV3Y8kbw6xKP+Xrg=="}' 2021-01-15T18:15:35.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 146bf154-6c9f-42e4-95db-cfc1e11fe571 -i td/osd-force-create-pg/0/new.json 2021-01-15T18:15:35.834 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:35.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/0/new.json 2021-01-15T18:15:35.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDH2wFgv7yWHRAAgPf+UAaV3Y8kbw6xKP+Xrg== --osd-uuid 146bf154-6c9f-42e4-95db-cfc1e11fe571 2021-01-15T18:15:35.867 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.867+0000 7fe245778080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:35.869 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.869+0000 7fe245778080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:35.870 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.870+0000 7fe245778080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:35.871 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.871+0000 7fe245778080 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-01-15T18:15:35.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.871+0000 7fe245778080 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-01-15T18:15:35.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.871+0000 7fe245778080 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2021-01-15T18:15:35.914 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:35.913+0000 7fe245778080 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2021-01-15T18:15:40.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2021-01-15T18:15:40.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:40.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:15:40.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:40.454 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:15:40.980 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:15:40.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:15:40.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:40.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:40.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:40.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:40.999 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:40.999+0000 7f919c0a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:41.001 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:41.000+0000 7f919c0a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:41.002+0000 7f919c0a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:41.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:15:41.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:41.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:15:41.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:41.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:15:41.374 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:41.519 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:41.518+0000 7f919c0a7080 -1 Falling back to public interface 2021-01-15T18:15:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:42.305 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:42.304+0000 7f919c0a7080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:15:42.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:42.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:42.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:42.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:42.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:15:42.801 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:43.193 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/737127,v1:127.0.0.1:6803/737127] [v2:127.0.0.1:6804/737127,v1:127.0.0.1:6805/737127] exists,up 146bf154-6c9f-42e4-95db-cfc1e11fe571 2021-01-15T18:15:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2021-01-15T18:15:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2021-01-15T18:15:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:15:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/1 2021-01-15T18:15:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:15:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-01-15T18:15:43.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-01-15T18:15:43.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:43.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:43.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:15:43.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:43.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:43.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:43.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:43.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:43.199 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:43.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:43.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:43.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:43.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:43.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:43.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:15:43.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:15:43.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:43.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:43.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:43.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:43.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:43.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/1 2021-01-15T18:15:43.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:43.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=db479f1e-6b81-418a-a780-70ebd0a51b27 2021-01-15T18:15:43.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 db479f1e-6b81-418a-a780-70ebd0a51b27' 2021-01-15T18:15:43.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:43.203 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 db479f1e-6b81-418a-a780-70ebd0a51b27 2021-01-15T18:15:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDP2wFg5SRNDBAAAWMMgZNLFgHvpEQ4HdZMow== 2021-01-15T18:15:43.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDP2wFg5SRNDBAAAWMMgZNLFgHvpEQ4HdZMow=="}' 2021-01-15T18:15:43.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new db479f1e-6b81-418a-a780-70ebd0a51b27 -i td/osd-force-create-pg/1/new.json 2021-01-15T18:15:43.774 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:43.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/1/new.json 2021-01-15T18:15:43.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDP2wFg5SRNDBAAAWMMgZNLFgHvpEQ4HdZMow== --osd-uuid db479f1e-6b81-418a-a780-70ebd0a51b27 2021-01-15T18:15:43.808 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.807+0000 7f6296e93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:43.810 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.809+0000 7f6296e93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:43.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.810+0000 7f6296e93080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:43.811 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.810+0000 7f6296e93080 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-01-15T18:15:43.812 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.810+0000 7f6296e93080 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-01-15T18:15:43.812 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.810+0000 7f6296e93080 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2021-01-15T18:15:43.853 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:43.853+0000 7f6296e93080 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2021-01-15T18:15:48.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2021-01-15T18:15:48.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:48.606 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:15:48.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:15:48.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:49.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:15:49.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:49.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:49.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:49.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:49.134 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:15:49.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:49.151+0000 7fec39646080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:49.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:49.153+0000 7fec39646080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:49.155 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:49.154+0000 7fec39646080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:15:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:15:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:49.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:49.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:15:49.533 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:49.675 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:49.675+0000 7fec39646080 -1 Falling back to public interface 2021-01-15T18:15:49.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:50.486 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:50.485+0000 7fec39646080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:15:50.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:50.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:50.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:50.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:50.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:15:50.921 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:51.343 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/738391,v1:127.0.0.1:6811/738391] [v2:127.0.0.1:6812/738391,v1:127.0.0.1:6813/738391] exists,up db479f1e-6b81-418a-a780-70ebd0a51b27 2021-01-15T18:15:51.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:51.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:51.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:51.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2021-01-15T18:15:51.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-force-create-pg 2021-01-15T18:15:51.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:15:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:15:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:15:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-force-create-pg/2 2021-01-15T18:15:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:15:51.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:15:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:15:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:15:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-01-15T18:15:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-01-15T18:15:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:15:51.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:15:51.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:15:51.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:15:51.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:15:51.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:15:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:15:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:15:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:15:51.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:15:51.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:15:51.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:15:51.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:15:51.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:15:51.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:15:51.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:15:51.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:15:51.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:51.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:15:51.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:15:51.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-force-create-pg/2 2021-01-15T18:15:51.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:15:51.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d1836c20-9d48-41e8-9cf5-b964383c1a9f 2021-01-15T18:15:51.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 d1836c20-9d48-41e8-9cf5-b964383c1a9f' 2021-01-15T18:15:51.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:15:51.352 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 d1836c20-9d48-41e8-9cf5-b964383c1a9f 2021-01-15T18:15:51.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDX2wFgH05PFRAAyd5SD2TR2csfSRO1AGCeNA== 2021-01-15T18:15:51.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDX2wFgH05PFRAAyd5SD2TR2csfSRO1AGCeNA=="}' 2021-01-15T18:15:51.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d1836c20-9d48-41e8-9cf5-b964383c1a9f -i td/osd-force-create-pg/2/new.json 2021-01-15T18:15:51.921 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:15:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-force-create-pg/2/new.json 2021-01-15T18:15:51.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDX2wFgH05PFRAAyd5SD2TR2csfSRO1AGCeNA== --osd-uuid d1836c20-9d48-41e8-9cf5-b964383c1a9f 2021-01-15T18:15:51.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.953+0000 7f20e2352080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:51.955 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.955+0000 7f20e2352080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.956+0000 7f20e2352080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:51.958 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.957+0000 7f20e2352080 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-01-15T18:15:51.958 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.957+0000 7f20e2352080 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-01-15T18:15:51.958 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:51.957+0000 7f20e2352080 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2021-01-15T18:15:52.042 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:52.042+0000 7f20e2352080 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2021-01-15T18:15:56.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2021-01-15T18:15:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:15:56.539 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:15:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:15:56.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:15:57.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:15:57.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:15:57.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:15:57.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:15:57.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:15:57.063 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:15:57.081 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:57.080+0000 7f5560257080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:57.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:57.081+0000 7f5560257080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:57.084 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:57.082+0000 7f5560257080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:15:57.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:15:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:15:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:15:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:15:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:15:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:15:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:15:57.463 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:15:57.599 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:57.599+0000 7f5560257080 -1 Falling back to public interface 2021-01-15T18:15:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:15:58.400 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:15:58.399+0000 7f5560257080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:15:58.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:15:58.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:15:58.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:15:58.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:15:58.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:15:58.852 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:15:59.253 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/739655,v1:127.0.0.1:6819/739655] [v2:127.0.0.1:6820/739655,v1:127.0.0.1:6821/739655] exists,up d1836c20-9d48-41e8-9cf5-b964383c1a9f 2021-01-15T18:15:59.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:15:59.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:15:59.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:15:59.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:29: TEST_reuse_id: ceph osd pool create foo 50 2021-01-15T18:16:00.011 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T18:16:00.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: wait_for_clean 2021-01-15T18:16:00.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:16:00.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:16:00.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:16:00.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:16:00.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:16:00.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:16:00.024 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:16:00.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:16:00.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:16:00.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:16:00.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:16:00.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:16:00.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:16:00.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:16:00.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:16:00.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:16:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:16:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:16:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:16:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:00.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:16:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2021-01-15T18:16:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2021-01-15T18:16:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2021-01-15T18:16:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:00.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:16:00.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672964 2021-01-15T18:16:00.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672964 2021-01-15T18:16:00.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672964' 2021-01-15T18:16:00.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:00.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:16:00.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509444 2021-01-15T18:16:00.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509444 2021-01-15T18:16:00.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672964 2-64424509444' 2021-01-15T18:16:00.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:00.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2021-01-15T18:16:00.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:00.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:16:00.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2021-01-15T18:16:00.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:00.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2021-01-15T18:16:00.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-01-15T18:16:00.966 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2021-01-15T18:16:00.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:16:01.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836486 2021-01-15T18:16:01.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:01.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672964 2021-01-15T18:16:01.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:01.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:16:01.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672964 2021-01-15T18:16:01.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672964 2021-01-15T18:16:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-01-15T18:16:01.380 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672964 2021-01-15T18:16:01.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:16:01.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672964 -lt 42949672964 2021-01-15T18:16:01.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:01.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2021-01-15T18:16:01.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:01.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:16:01.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509444 2021-01-15T18:16:01.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:01.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509444 2021-01-15T18:16:01.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2021-01-15T18:16:01.774 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509444 2021-01-15T18:16:01.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:16:02.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509444 -lt 64424509444 2021-01-15T18:16:02.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:16:02.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:02.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:02.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 50 == 0 2021-01-15T18:16:02.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:16:02.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:16:02.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:16:02.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:16:02.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:16:02.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:16:02.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:16:03.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=36 2021-01-15T18:16:03.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:16:03.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:03.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:03.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 36 = 50 2021-01-15T18:16:03.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 36 '!=' -1 2021-01-15T18:16:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:16:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=36 2021-01-15T18:16:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:16:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:16:03.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:16:03.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:16:03.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:16:03.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:16:03.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:16:03.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:16:03.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:16:03.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:16:04.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=36 2021-01-15T18:16:04.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:16:04.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:04.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:04.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 36 = 50 2021-01-15T18:16:04.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 36 '!=' 36 2021-01-15T18:16:04.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:16:04.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:16:04.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:16:04.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:16:04.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:16:04.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:16:04.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:16:05.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T18:16:05.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T18:16:05.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2021-01-15T18:16:05.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:16:05.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T18:16:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:16:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:16:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:16:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:16:05.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:16:05.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:16:05.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:16:05.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:16:05.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=50 2021-01-15T18:16:05.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:16:05.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:05.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:06.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 50 = 50 2021-01-15T18:16:06.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:16:06.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:16:06.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:32: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2021-01-15T18:16:06.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:16:06.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:16:06.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:16:06.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:16:06.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:16:06.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:16:06.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2021-01-15T18:16:06.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:16:06.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:16:06.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:16:06.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:16:06.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:16:06.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:16:06.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2021-01-15T18:16:06.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:16:06.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:16:06.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:16:06.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:16:06.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:16:06.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:16:06.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2021-01-15T18:16:07.473 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2021-01-15T18:16:07.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2021-01-15T18:16:08.610 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2021-01-15T18:16:08.610 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2021-01-15T18:16:08.611 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.0_head removing 1.0 2021-01-15T18:16:08.710 INFO:tasks.workunit.client.0.smithi145.stdout:Remove successful 2021-01-15T18:16:09.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2021-01-15T18:16:10.130 INFO:tasks.workunit.client.0.smithi145.stderr:PG '1.0' not found 2021-01-15T18:16:10.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2021-01-15T18:16:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2021-01-15T18:16:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:16:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:16:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:16:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/0 2021-01-15T18:16:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:16:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2021-01-15T18:16:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2021-01-15T18:16:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:16:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:16:10.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:16:10.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:10.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:16:10.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:10.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:10.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:16:10.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:16:10.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:10.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:10.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:10.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:16:10.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:16:10.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/0 2021-01-15T18:16:10.409 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:16:10.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:16:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:10.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/0/whoami 2021-01-15T18:16:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:16:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:16:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:16:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:16:10.425 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:10.424+0000 7f4764241080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:10.427 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:10.426+0000 7f4764241080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:10.430 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:10.429+0000 7f4764241080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:10.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:16:10.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:10.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:16:10.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:10.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:10.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:10.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:10.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:10.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:16:10.804 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:10.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:10.944+0000 7f4764241080 -1 Falling back to public interface 2021-01-15T18:16:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:11.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:11.801+0000 7f4764241080 -1 osd.0 22 log_to_monitors {default=true} 2021-01-15T18:16:12.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:12.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:12.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:12.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:12.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:16:12.195 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:12.603 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/741768,v1:127.0.0.1:6803/741768] [v2:127.0.0.1:6804/741768,v1:127.0.0.1:6805/741768] exists,up 146bf154-6c9f-42e4-95db-cfc1e11fe571 2021-01-15T18:16:12.603 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:16:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:12.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2021-01-15T18:16:12.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2021-01-15T18:16:12.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:16:12.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T18:16:12.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:16:12.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/1 2021-01-15T18:16:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:16:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2021-01-15T18:16:12.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2021-01-15T18:16:12.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:12.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:16:12.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:16:12.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:16:12.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:12.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:12.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:12.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:12.608 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:16:12.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:12.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:12.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:12.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:16:12.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:16:12.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:12.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:12.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:12.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:16:12.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:16:12.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/1 2021-01-15T18:16:12.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T18:16:12.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:12.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/1/whoami 2021-01-15T18:16:12.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T18:16:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:16:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:16:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:16:12.621 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:12.621+0000 7f4aaa122080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:12.623 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:12.622+0000 7f4aaa122080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:12.624 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:12.623+0000 7f4aaa122080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T18:16:12.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:16:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:12.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:12.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:12.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:12.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:16:12.995 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:13.141 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:13.140+0000 7f4aaa122080 -1 Falling back to public interface 2021-01-15T18:16:13.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:14.041 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:14.040+0000 7f4aaa122080 -1 osd.1 22 log_to_monitors {default=true} 2021-01-15T18:16:14.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:14.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:14.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:14.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:14.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:16:14.393 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:14.785 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 27 up_thru 27 down_at 24 last_clean_interval [10,22) [v2:127.0.0.1:6810/742257,v1:127.0.0.1:6811/742257] [v2:127.0.0.1:6812/742257,v1:127.0.0.1:6813/742257] exists,up db479f1e-6b81-418a-a780-70ebd0a51b27 2021-01-15T18:16:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:14.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2021-01-15T18:16:14.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-force-create-pg 2021-01-15T18:16:14.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:16:14.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T18:16:14.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:16:14.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-force-create-pg/2 2021-01-15T18:16:14.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2021-01-15T18:16:14.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:14.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:14.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:14.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2021-01-15T18:16:14.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2021-01-15T18:16:14.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:14.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:16:14.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2021-01-15T18:16:14.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:16:14.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:14.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:14.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:14.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:14.794 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:16:14.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:14.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' 2021-01-15T18:16:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2021-01-15T18:16:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2021-01-15T18:16:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:14.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:14.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:14.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:16:14.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:16:14.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-force-create-pg/2 2021-01-15T18:16:14.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T18:16:14.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=89d32bb7-6e83-4ee3-a5fe-ed83e9c0f703 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.736080/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:14.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-force-create-pg/2/whoami 2021-01-15T18:16:14.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T18:16:14.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:16:14.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:16:14.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:16:14.799 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:16:14.807 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:14.806+0000 7f873f07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:14.809 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:14.808+0000 7f873f07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:14.810 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:14.809+0000 7f873f07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:15.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T18:16:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:16:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:15.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:15.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:15.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:15.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:16:15.179 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:15.325 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:15.324+0000 7f873f07e080 -1 Falling back to public interface 2021-01-15T18:16:15.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:16.199 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:16.198+0000 7f873f07e080 -1 osd.2 23 log_to_monitors {default=true} 2021-01-15T18:16:16.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:16.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:16.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:16.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:16.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:16:16.572 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:17.067 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [15,23) [v2:127.0.0.1:6818/742746,v1:127.0.0.1:6819/742746] [v2:127.0.0.1:6820/742746,v1:127.0.0.1:6821/742746] exists,up d1836c20-9d48-41e8-9cf5-b964383c1a9f 2021-01-15T18:16:17.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:17.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: sleep 10 2021-01-15T18:16:27.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: ceph pg ls 2021-01-15T18:16:27.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep 1.0 2021-01-15T18:16:27.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: grep stale 2021-01-15T18:16:27.459 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 stale+active+clean 26s 0'0 20:9 [1]p1 [1]p1 2021-01-15T18:15:59.672328+0000 2021-01-15T18:15:59.672328+0000 2021-01-15T18:16:27.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:44: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2021-01-15T18:16:28.006 INFO:tasks.workunit.client.0.smithi145.stderr:pg 1.0 already creating 2021-01-15T18:16:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: wait_for_clean 2021-01-15T18:16:28.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:16:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:16:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:16:28.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:16:28.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:16:28.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:16:28.017 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:16:28.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:16:28.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:16:28.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:16:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:16:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:16:28.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:16:28.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:16:28.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:16:28.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:16:28.573 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:16:28.574 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:16:28.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:16:28.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:28.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:16:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182406 2021-01-15T18:16:28.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182406 2021-01-15T18:16:28.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406' 2021-01-15T18:16:28.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:28.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:16:28.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116997 2021-01-15T18:16:28.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116997 2021-01-15T18:16:28.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-115964116997' 2021-01-15T18:16:28.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:16:28.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:16:28.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051589 2021-01-15T18:16:28.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051589 2021-01-15T18:16:28.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-107374182406 1-115964116997 2-124554051589' 2021-01-15T18:16:28.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:28.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-107374182406 2021-01-15T18:16:28.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:28.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:16:28.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-107374182406 2021-01-15T18:16:28.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:28.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182406 2021-01-15T18:16:28.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 107374182406' 2021-01-15T18:16:28.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:16:28.980 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 107374182406 2021-01-15T18:16:29.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182404 -lt 107374182406 2021-01-15T18:16:29.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:16:30.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:16:30.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:16:30.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182406 -lt 107374182406 2021-01-15T18:16:30.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:30.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-115964116997 2021-01-15T18:16:30.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:30.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:16:30.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-115964116997 2021-01-15T18:16:30.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:30.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116997 2021-01-15T18:16:30.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 115964116997' 2021-01-15T18:16:30.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:16:30.775 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 115964116997 2021-01-15T18:16:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116998 -lt 115964116997 2021-01-15T18:16:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:16:31.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-124554051589 2021-01-15T18:16:31.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:16:31.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:16:31.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-124554051589 2021-01-15T18:16:31.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:16:31.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051589 2021-01-15T18:16:31.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 124554051589' 2021-01-15T18:16:31.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:16:31.170 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 124554051589 2021-01-15T18:16:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051590 -lt 124554051589 2021-01-15T18:16:31.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:16:31.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:31.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:32.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 50 == 0 2021-01-15T18:16:32.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:16:32.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:16:32.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:16:32.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:16:32.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:16:32.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:16:32.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:16:32.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=50 2021-01-15T18:16:32.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:16:32.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:16:32.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:16:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 50 = 50 2021-01-15T18:16:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:16:32.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:16:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:16:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-force-create-pg 0 2021-01-15T18:16:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2021-01-15T18:16:32.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:16:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2021-01-15T18:16:32.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:16:32.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:16:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:16:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:16:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:16:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:16:33.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:16:33.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:16:33.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:16:33.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:16:33.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:16:33.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:16:33.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:16:33.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:16:33.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:16:33.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:16:33.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:16:33.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:16:33.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2021-01-15T18:16:33.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:16:33.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.736080 2021-01-15T18:16:33.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.736080 2021-01-15T18:16:33.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:16:33.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:16:33.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:16:33.103 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:16:33.103 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:16:33.166 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2021-01-15T18:16:33.167 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2021-01-15T18:16:33.228 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:16:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:16:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:16:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:16:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:16:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:16:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:16:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:16:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:16:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:16:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-markdown 2021-01-15T18:16:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2021-01-15T18:16:33.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2021-01-15T18:16:33.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2021-01-15T18:16:33.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2021-01-15T18:16:33.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2021-01-15T18:16:33.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2021-01-15T18:16:33.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none ' 2021-01-15T18:16:33.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2021-01-15T18:16:33.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2021-01-15T18:16:33.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:16:33.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2021-01-15T18:16:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_markdown_boot_exceed_time 2021-01-15T18:16:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_markdown_exceed_maxdown_count 2021-01-15T18:16:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_osd_stop' 2021-01-15T18:16:33.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-01-15T18:16:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-01-15T18:16:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-01-15T18:16:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-01-15T18:16:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:16:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:16:33.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:16:33.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:16:33.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:16:33.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:16:33.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:16:33.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:16:33.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:16:33.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:16:33.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:16:33.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:16:33.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:16:33.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:16:33.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:16:33.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:16:33.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:16:33.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:16:33.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:16:33.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:16:33.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:16:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:16:33.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:16:33.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:33.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:16:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:16:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:16:33.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-01-15T18:16:33.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:16:33.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.743735 2021-01-15T18:16:33.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:16:33.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:16:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:16:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:16:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-01-15T18:16:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2021-01-15T18:16:33.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2021-01-15T18:16:33.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2021-01-15T18:16:33.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-01-15T18:16:33.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:16:33.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:16:33.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:16:33.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-01-15T18:16:33.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-01-15T18:16:33.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:16:33.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:33.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:33.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:33.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:16:33.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:16:33.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:16:33.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:16:33.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:16:33.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:16:33.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:16:33.719 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:16:33.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:16:33.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:16:33.720 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:16:33.720 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.720 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:33.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:16:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:16:33.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get fsid 2021-01-15T18:16:33.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:16:33.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:16:33.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:16:33.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:16:33.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:16:33.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:16:33.821 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:16:33.821 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:16:33.821 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:16:33.821 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:33.821 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:33.822 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:16:33.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:16:33.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get mon_host 2021-01-15T18:16:33.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2021-01-15T18:16:33.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2021-01-15T18:16:33.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:16:33.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:16:33.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:16:33.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2021-01-15T18:16:33.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:16:34.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:16:34.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:34.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:34.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:34.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:34.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:34.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:34.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:16:34.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:16:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2021-01-15T18:16:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:16:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:16:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:16:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:16:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2021-01-15T18:16:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:16:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:34.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:34.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-01-15T18:16:34.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-01-15T18:16:34.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:34.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:16:34.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:16:34.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:16:34.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:34.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:34.850 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:34.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:34.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:34.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:34.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:34.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:34.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:16:34.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:16:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:16:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:16:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:34.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:34.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:34.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:16:34.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:16:34.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2021-01-15T18:16:34.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:16:34.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=954c23cb-9962-48fd-8012-e597509fe690 2021-01-15T18:16:34.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 954c23cb-9962-48fd-8012-e597509fe690' 2021-01-15T18:16:34.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:16:34.856 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 954c23cb-9962-48fd-8012-e597509fe690 2021-01-15T18:16:34.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAC3AFg40MAMxAAMFHQS8GU2aPnMi2p7Wfcuw== 2021-01-15T18:16:34.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAC3AFg40MAMxAAMFHQS8GU2aPnMi2p7Wfcuw=="}' 2021-01-15T18:16:34.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 954c23cb-9962-48fd-8012-e597509fe690 -i td/osd-markdown/0/new.json 2021-01-15T18:16:35.193 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:35.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2021-01-15T18:16:35.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAC3AFg40MAMxAAMFHQS8GU2aPnMi2p7Wfcuw== --osd-uuid 954c23cb-9962-48fd-8012-e597509fe690 2021-01-15T18:16:35.226 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.225+0000 7ff49dd7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:35.228 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.227+0000 7ff49dd7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:35.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.229+0000 7ff49dd7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:35.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.229+0000 7ff49dd7f080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:16:35.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.229+0000 7ff49dd7f080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:16:35.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.229+0000 7ff49dd7f080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:16:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:35.273+0000 7ff49dd7f080 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-01-15T18:16:39.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-01-15T18:16:39.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:16:39.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:16:39.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:16:39.757 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:16:40.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:16:40.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:40.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:16:40.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:16:40.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:16:40.280 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:16:40.297 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:40.297+0000 7f61bbd74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:40.299 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:40.298+0000 7f61bbd74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:40.300 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:40.300+0000 7f61bbd74080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:40.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:16:40.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:40.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:16:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:40.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:40.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:16:40.670 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:40.817 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:40.815+0000 7f61bbd74080 -1 Falling back to public interface 2021-01-15T18:16:41.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:41.593 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:41.593+0000 7f61bbd74080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:16:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:42.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:42.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:16:42.057 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:42.475 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/744782,v1:127.0.0.1:6803/744782] [v2:127.0.0.1:6804/744782,v1:127.0.0.1:6805/744782] exists,up 954c23cb-9962-48fd-8012-e597509fe690 2021-01-15T18:16:42.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2021-01-15T18:16:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:16:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:16:42.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:16:42.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:16:42.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2021-01-15T18:16:42.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:16:42.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:42.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-01-15T18:16:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-01-15T18:16:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:16:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:16:42.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:16:42.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:42.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:42.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:16:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:16:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:16:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:16:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:16:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:16:42.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2021-01-15T18:16:42.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:16:42.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d33e7298-b8bf-4f80-8df3-85e7b12f4f3a 2021-01-15T18:16:42.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 d33e7298-b8bf-4f80-8df3-85e7b12f4f3a' 2021-01-15T18:16:42.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:16:42.487 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 d33e7298-b8bf-4f80-8df3-85e7b12f4f3a 2021-01-15T18:16:42.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAK3AFg5Bc1HRAA7xjnI+dLwjBbkSN11/L1Fg== 2021-01-15T18:16:42.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAK3AFg5Bc1HRAA7xjnI+dLwjBbkSN11/L1Fg=="}' 2021-01-15T18:16:42.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d33e7298-b8bf-4f80-8df3-85e7b12f4f3a -i td/osd-markdown/1/new.json 2021-01-15T18:16:42.994 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:43.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2021-01-15T18:16:43.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAK3AFg5Bc1HRAA7xjnI+dLwjBbkSN11/L1Fg== --osd-uuid d33e7298-b8bf-4f80-8df3-85e7b12f4f3a 2021-01-15T18:16:43.027 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.025+0000 7fe94bb62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:43.028 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.027+0000 7fe94bb62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:43.029 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.027+0000 7fe94bb62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:43.030 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.028+0000 7fe94bb62080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:16:43.030 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.028+0000 7fe94bb62080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:16:43.030 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.028+0000 7fe94bb62080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:16:43.081 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:43.080+0000 7fe94bb62080 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-01-15T18:16:47.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-01-15T18:16:47.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:16:47.620 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:16:47.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:16:47.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:16:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:16:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:48.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:16:48.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:16:48.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:16:48.145 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:16:48.163 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:48.162+0000 7fdfa1d26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:48.188 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:48.187+0000 7fdfa1d26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:48.189 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:48.188+0000 7fdfa1d26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:16:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:48.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:16:48.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:48.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:48.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:48.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:48.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:48.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:16:48.533 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:48.707 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:48.704+0000 7fdfa1d26080 -1 Falling back to public interface 2021-01-15T18:16:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:49.514 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:49.513+0000 7fdfa1d26080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:16:49.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:49.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:49.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:49.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:49.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:16:49.935 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:50.352 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/746046,v1:127.0.0.1:6811/746046] [v2:127.0.0.1:6812/746046,v1:127.0.0.1:6813/746046] exists,up d33e7298-b8bf-4f80-8df3-85e7b12f4f3a 2021-01-15T18:16:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2021-01-15T18:16:50.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:16:50.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:16:50.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:16:50.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:16:50.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2021-01-15T18:16:50.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-01-15T18:16:50.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:16:50.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:16:50.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:16:50.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:16:50.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:16:50.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:16:50.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:16:50.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:16:50.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:16:50.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:50.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:16:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:16:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:16:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:16:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:16:50.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2021-01-15T18:16:50.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:16:50.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3b228ddc-2b59-44d6-bb95-d9968bfee287 2021-01-15T18:16:50.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 3b228ddc-2b59-44d6-bb95-d9968bfee287' 2021-01-15T18:16:50.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:16:50.361 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 3b228ddc-2b59-44d6-bb95-d9968bfee287 2021-01-15T18:16:50.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAS3AFgE1riFRAAI5d4Ka4Rj0HkCHunhCJSOw== 2021-01-15T18:16:50.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAS3AFgE1riFRAAI5d4Ka4Rj0HkCHunhCJSOw=="}' 2021-01-15T18:16:50.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3b228ddc-2b59-44d6-bb95-d9968bfee287 -i td/osd-markdown/2/new.json 2021-01-15T18:16:50.989 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:16:51.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2021-01-15T18:16:51.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAS3AFgE1riFRAAI5d4Ka4Rj0HkCHunhCJSOw== --osd-uuid 3b228ddc-2b59-44d6-bb95-d9968bfee287 2021-01-15T18:16:51.024 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.023+0000 7f04f87e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:51.025 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.025+0000 7f04f87e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:51.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.026+0000 7f04f87e6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:51.027 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.026+0000 7f04f87e6080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:16:51.027 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.026+0000 7f04f87e6080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:16:51.027 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.026+0000 7f04f87e6080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:16:51.081 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:51.081+0000 7f04f87e6080 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-01-15T18:16:55.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-01-15T18:16:55.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:16:55.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:16:55.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:16:55.684 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:16:56.208 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:16:56.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:16:56.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:16:56.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:16:56.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:16:56.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:16:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:56.226+0000 7fb63c3ed080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:56.228+0000 7fb63c3ed080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:56.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:56.229+0000 7fb63c3ed080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:16:56.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:16:56.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:16:56.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:16:56.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:16:56.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:16:56.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:56.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:16:56.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:56.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:16:56.598 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:16:56.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:56.749+0000 7fb63c3ed080 -1 Falling back to public interface 2021-01-15T18:16:57.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:16:57.530 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:16:57.529+0000 7fb63c3ed080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:16:58.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:16:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:16:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:16:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:16:58.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:16:58.013 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:16:58.415 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/747310,v1:127.0.0.1:6819/747310] [v2:127.0.0.1:6820/747310,v1:127.0.0.1:6821/747310] exists,up 3b228ddc-2b59-44d6-bb95-d9968bfee287 2021-01-15T18:16:58.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:16:58.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:16:58.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:16:58.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2021-01-15T18:16:58.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T18:16:58.903 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T18:16:58.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T18:16:58.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T18:16:59.442 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T18:16:59.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:17:00.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T18:17:01.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2021-01-15T18:17:01.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2021-01-15T18:17:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2021-01-15T18:17:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-01-15T18:17:01.876 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:17:01.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2021-01-15T18:17:02.012 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:17:02.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2021-01-15T18:17:02.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2021-01-15T18:17:02.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2021-01-15T18:17:02.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-01-15T18:17:02.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2021-01-15T18:17:02.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:17:02.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:17:02.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:17:02.538 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:17:02.538 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:17:02.539 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:17:02.539 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:02.539 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:17:02.539 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:17:02.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:17:02.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:17:02.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:17:02.973 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:02.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:17:02.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:17:03.379 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:17:03.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:17:13.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:17:13.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:17:13.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:17:13.921 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:17:13.922 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:17:13.922 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:17:13.922 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:13.922 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:17:13.923 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:17:13.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:17:13.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:17:13.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:17:14.318 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:14.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:17:14.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:17:14.734 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:17:14.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:17:24.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:17:24.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:17:24.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:17:25.263 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:17:25.264 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:17:25.264 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:17:25.264 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:25.264 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:17:25.264 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:17:25.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:17:25.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:17:25.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:17:25.670 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:17:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:17:26.077 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:17:26.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:17:36.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2021-01-15T18:17:51.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:17:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2021-01-15T18:17:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2021-01-15T18:17:51.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2021-01-15T18:17:51.623 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:17:51.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-01-15T18:17:51.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:17:51.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:17:51.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:17:51.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:17:51.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:17:51.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:17:51.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:17:51.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:17:51.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:17:51.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:17:51.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:17:51.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:17:51.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:17:51.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:17:51.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:17:51.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:17:51.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:17:51.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:17:51.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:17:51.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:17:51.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:17:51.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:17:51.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:17:51.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:51.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:51.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:17:51.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:17:51.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:17:51.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-01-15T18:17:51.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-01-15T18:17:51.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-01-15T18:17:51.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-01-15T18:17:51.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:17:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:17:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:17:51.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:17:51.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:17:51.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:17:51.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:17:51.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:17:51.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:17:51.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:17:51.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:17:51.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:17:51.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:17:51.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:17:51.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:17:51.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:17:51.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:17:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:17:51.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:17:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:17:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:17:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:17:51.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:17:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:51.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:51.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:17:51.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:17:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:17:51.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-01-15T18:17:51.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:17:51.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:51.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:51.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.743735 2021-01-15T18:17:51.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:17:51.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:17:51.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:17:51.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-01-15T18:17:51.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2021-01-15T18:17:51.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2021-01-15T18:17:51.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2021-01-15T18:17:51.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-01-15T18:17:51.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:17:51.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:17:51.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:17:51.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-01-15T18:17:51.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-01-15T18:17:52.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:17:52.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:17:52.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:17:52.048 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:17:52.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:52.049 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:52.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:17:52.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:17:52.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:17:52.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:17:52.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:17:52.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:17:52.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:17:52.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:17:52.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:17:52.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:17:52.226 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:17:52.227 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:17:52.227 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:52.227 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:52.227 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:17:52.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:17:52.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get fsid 2021-01-15T18:17:52.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:17:52.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:17:52.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:17:52.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:17:52.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:17:52.327 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:17:52.327 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:17:52.327 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:17:52.327 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:17:52.328 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:52.328 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:52.328 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:17:52.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:17:52.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get mon_host 2021-01-15T18:17:52.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2021-01-15T18:17:52.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2021-01-15T18:17:52.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:17:52.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:17:52.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:17:52.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2021-01-15T18:17:52.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:17:53.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:17:53.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:17:53.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:17:53.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:17:53.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:53.313 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:53.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:17:53.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:17:53.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:17:53.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2021-01-15T18:17:53.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:17:53.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:17:53.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:17:53.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:17:53.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2021-01-15T18:17:53.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:17:53.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:17:53.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:17:53.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:17:53.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-01-15T18:17:53.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-01-15T18:17:53.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:17:53.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:17:53.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:17:53.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:17:53.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:17:53.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:17:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:17:53.337 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:17:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:17:53.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:17:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:17:53.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:17:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:17:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:17:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:17:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:17:53.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:17:53.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:17:53.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:17:53.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:17:53.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:17:53.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2021-01-15T18:17:53.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:17:53.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a 2021-01-15T18:17:53.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a' 2021-01-15T18:17:53.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:17:53.342 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a 2021-01-15T18:17:53.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBR3AFgnlSoFBAAUL9O/LhL19BqKY7Beywm3Q== 2021-01-15T18:17:53.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBR3AFgnlSoFBAAUL9O/LhL19BqKY7Beywm3Q=="}' 2021-01-15T18:17:53.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a -i td/osd-markdown/0/new.json 2021-01-15T18:17:53.691 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:17:53.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2021-01-15T18:17:53.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBR3AFgnlSoFBAAUL9O/LhL19BqKY7Beywm3Q== --osd-uuid 9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a 2021-01-15T18:17:53.724 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.723+0000 7f40279ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:53.726 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.725+0000 7f40279ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:53.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.726+0000 7f40279ef080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.727+0000 7f40279ef080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:17:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.727+0000 7f40279ef080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:17:53.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.727+0000 7f40279ef080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:17:53.773 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:53.772+0000 7f40279ef080 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-01-15T18:17:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-01-15T18:17:58.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:17:58.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:17:58.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:17:58.349 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:17:58.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:17:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:17:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:17:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:17:58.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:17:58.870 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:17:58.887 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:58.886+0000 7f76303e2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:58.913 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:58.913+0000 7f76303e2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:58.915 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:58.914+0000 7f76303e2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:17:59.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:17:59.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:17:59.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:17:59.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:17:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:17:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:17:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:17:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:17:59.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:17:59.262 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:17:59.431 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:17:59.430+0000 7f76303e2080 -1 Falling back to public interface 2021-01-15T18:17:59.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:18:00.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:00.231+0000 7f76303e2080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:18:00.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:18:00.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:18:00.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:18:00.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:18:00.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:18:00.660 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:18:01.051 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/749321,v1:127.0.0.1:6803/749321] [v2:127.0.0.1:6804/749321,v1:127.0.0.1:6805/749321] exists,up 9a7d0f6f-298a-4ec7-8c2f-5d91253b4b0a 2021-01-15T18:18:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:18:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:18:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:18:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2021-01-15T18:18:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:18:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:18:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:18:01.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:18:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2021-01-15T18:18:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:18:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:18:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:18:01.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:18:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-01-15T18:18:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-01-15T18:18:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:18:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:18:01.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:18:01.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:18:01.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:18:01.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:18:01.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:18:01.056 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:18:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:18:01.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:18:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:18:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:18:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:18:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:18:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:18:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:18:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:18:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:18:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:18:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:18:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:18:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2021-01-15T18:18:01.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:18:01.060 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 d2500967-6bc4-4285-b005-1f6a2defe82e 2021-01-15T18:18:01.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d2500967-6bc4-4285-b005-1f6a2defe82e 2021-01-15T18:18:01.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 d2500967-6bc4-4285-b005-1f6a2defe82e' 2021-01-15T18:18:01.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:18:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBZ3AFgTW7gAxAA124Ehmtfi2l4UNoqXrrXwA== 2021-01-15T18:18:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBZ3AFgTW7gAxAA124Ehmtfi2l4UNoqXrrXwA=="}' 2021-01-15T18:18:01.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d2500967-6bc4-4285-b005-1f6a2defe82e -i td/osd-markdown/1/new.json 2021-01-15T18:18:01.590 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:18:01.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2021-01-15T18:18:01.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZ3AFgTW7gAxAA124Ehmtfi2l4UNoqXrrXwA== --osd-uuid d2500967-6bc4-4285-b005-1f6a2defe82e 2021-01-15T18:18:01.623 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.622+0000 7f5197fdf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:01.625 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.624+0000 7f5197fdf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:01.626 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.626+0000 7f5197fdf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:01.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.626+0000 7f5197fdf080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:18:01.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.626+0000 7f5197fdf080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:18:01.628 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.626+0000 7f5197fdf080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:18:01.674 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:01.674+0000 7f5197fdf080 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-01-15T18:18:06.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-01-15T18:18:06.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:18:06.230 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:18:06.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:18:06.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:18:06.749 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:18:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:18:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:18:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:18:06.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:18:06.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:18:06.769 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:06.768+0000 7f6cdb04a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:06.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:06.770+0000 7f6cdb04a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:06.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:06.770+0000 7f6cdb04a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:07.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:18:07.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:18:07.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:18:07.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:18:07.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:18:07.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:18:07.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:18:07.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:18:07.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:18:07.144 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:18:07.289 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:07.289+0000 7f6cdb04a080 -1 Falling back to public interface 2021-01-15T18:18:07.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:18:08.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:08.060+0000 7f6cdb04a080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:18:08.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:18:08.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:18:08.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:18:08.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:18:08.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:18:08.542 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:18:08.931 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/750585,v1:127.0.0.1:6811/750585] [v2:127.0.0.1:6812/750585,v1:127.0.0.1:6813/750585] exists,up d2500967-6bc4-4285-b005-1f6a2defe82e 2021-01-15T18:18:08.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:18:08.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:18:08.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:18:08.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2021-01-15T18:18:08.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:18:08.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:18:08.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:18:08.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:18:08.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2021-01-15T18:18:08.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:18:08.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:18:08.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:18:08.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:18:08.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-01-15T18:18:08.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-01-15T18:18:08.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:18:08.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:18:08.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:18:08.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:18:08.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:18:08.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:18:08.936 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:18:08.936 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:18:08.936 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:18:08.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:18:08.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:18:08.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:18:08.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:18:08.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:18:08.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:18:08.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:18:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:18:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:18:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:18:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:18:08.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:18:08.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2021-01-15T18:18:08.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:18:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8d6bd982-90a3-4ef1-a58b-96dafd2ab226 2021-01-15T18:18:08.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8d6bd982-90a3-4ef1-a58b-96dafd2ab226' 2021-01-15T18:18:08.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:18:08.940 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 8d6bd982-90a3-4ef1-a58b-96dafd2ab226 2021-01-15T18:18:08.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBg3AFg87A/OBAAB0OIr/m9uhIsrI9+lpAEOQ== 2021-01-15T18:18:08.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBg3AFg87A/OBAAB0OIr/m9uhIsrI9+lpAEOQ=="}' 2021-01-15T18:18:08.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8d6bd982-90a3-4ef1-a58b-96dafd2ab226 -i td/osd-markdown/2/new.json 2021-01-15T18:18:09.502 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:18:09.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2021-01-15T18:18:09.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBg3AFg87A/OBAAB0OIr/m9uhIsrI9+lpAEOQ== --osd-uuid 8d6bd982-90a3-4ef1-a58b-96dafd2ab226 2021-01-15T18:18:09.535 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.534+0000 7f5017ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:09.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.536+0000 7f5017ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:09.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.536+0000 7f5017ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:09.538 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.537+0000 7f5017ed5080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:18:09.538 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.537+0000 7f5017ed5080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:18:09.539 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.537+0000 7f5017ed5080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:18:09.576 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:09.575+0000 7f5017ed5080 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-01-15T18:18:14.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-01-15T18:18:14.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:18:14.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:18:14.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:18:14.127 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:18:14.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:18:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:18:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:18:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:18:14.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:18:14.646 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:18:14.664 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:14.663+0000 7f0f3a6ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:14.691 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:14.690+0000 7f0f3a6ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:14.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:14.692+0000 7f0f3a6ca080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:18:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:18:15.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:18:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:18:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:18:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:18:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:18:15.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:18:15.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:18:15.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:18:15.042 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:18:15.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:15.207+0000 7f0f3a6ca080 -1 Falling back to public interface 2021-01-15T18:18:15.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:18:16.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:18:16.008+0000 7f0f3a6ca080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:18:16.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:18:16.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:18:16.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:18:16.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:18:16.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:18:16.444 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:18:16.855 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/751849,v1:127.0.0.1:6819/751849] [v2:127.0.0.1:6820/751849,v1:127.0.0.1:6821/751849] exists,up 8d6bd982-90a3-4ef1-a58b-96dafd2ab226 2021-01-15T18:18:16.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:18:16.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:18:16.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:18:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2021-01-15T18:18:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T18:18:17.300 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T18:18:17.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T18:18:17.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T18:18:17.944 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T18:18:17.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:18:18.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T18:18:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2021-01-15T18:18:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2021-01-15T18:18:20.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2021-01-15T18:18:20.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-01-15T18:18:20.629 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:18:20.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2021-01-15T18:18:20.765 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:18:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2021-01-15T18:18:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-01-15T18:18:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2021-01-15T18:18:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-01-15T18:18:20.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-01-15T18:18:20.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:18:20.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:18:20.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:18:21.283 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:18:21.283 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:18:21.284 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:18:21.284 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:21.284 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:18:21.284 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:18:21.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:18:21.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:18:21.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:18:21.684 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:18:21.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:18:22.119 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:18:22.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:18:32.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:18:32.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:18:32.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:18:32.650 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:18:32.650 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:18:32.650 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:18:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:32.651 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:18:32.652 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:18:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:18:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:18:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:18:33.080 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:33.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:18:33.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:18:33.491 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:18:33.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:18:43.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:18:43.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:18:43.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:18:44.022 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:18:44.023 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:18:44.023 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:18:44.023 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:44.023 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:18:44.023 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:18:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:18:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:18:44.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:18:44.429 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:44.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:18:44.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:18:44.845 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:18:44.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:18:54.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:18:54.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:18:55.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:18:55.378 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:18:55.378 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:18:55.378 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:18:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:18:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:18:55.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:18:55.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:18:55.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:18:55.797 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:18:55.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:18:55.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:18:56.198 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:18:56.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:19:06.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2021-01-15T18:19:21.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:19:21.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2021-01-15T18:19:21.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2021-01-15T18:19:21.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2021-01-15T18:19:21.744 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:19:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-01-15T18:19:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:19:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:19:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:19:21.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:19:21.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:19:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:19:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:19:21.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:19:21.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:19:21.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:19:21.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:19:21.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:19:21.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:19:21.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:19:21.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:19:21.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:19:21.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:19:21.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:19:21.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:19:21.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:19:21.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:19:21.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:19:21.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:19:21.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:21.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:21.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:19:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:19:21.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:19:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-01-15T18:19:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-01-15T18:19:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-01-15T18:19:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-01-15T18:19:21.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:19:21.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:19:21.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:19:21.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:19:21.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:19:21.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:19:21.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:19:21.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:19:21.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:19:21.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:19:21.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:19:21.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:19:21.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:19:21.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:19:21.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:19:21.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:19:21.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:19:21.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:19:21.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:21.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:21.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:19:21.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:19:21.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:19:21.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-01-15T18:19:21.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:19:21.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:21.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:21.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.743735 2021-01-15T18:19:21.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:19:21.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:19:21.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:19:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-01-15T18:19:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2021-01-15T18:19:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2021-01-15T18:19:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2021-01-15T18:19:21.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-01-15T18:19:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:19:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:19:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:19:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-01-15T18:19:21.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-01-15T18:19:22.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:19:22.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:19:22.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:19:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:19:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:22.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:19:22.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:19:22.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:19:22.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:19:22.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:19:22.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:19:22.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:19:22.347 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:19:22.347 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:19:22.347 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:19:22.348 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:19:22.348 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:22.348 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:22.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:19:22.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:19:22.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get fsid 2021-01-15T18:19:22.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:19:22.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:19:22.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:19:22.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:19:22.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:19:22.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:19:22.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:19:22.451 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:19:22.451 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:19:22.452 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:22.452 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:22.452 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:19:22.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:19:22.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get mon_host 2021-01-15T18:19:22.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2021-01-15T18:19:22.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2021-01-15T18:19:22.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:19:22.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:19:22.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:19:22.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2021-01-15T18:19:22.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:19:23.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:19:23.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:19:23.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:19:23.040 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:19:23.040 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:23.040 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:23.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:23.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:19:23.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:19:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2021-01-15T18:19:23.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:19:23.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:19:23.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:19:23.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:19:23.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2021-01-15T18:19:23.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:19:23.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:19:23.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:19:23.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:19:23.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-01-15T18:19:23.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-01-15T18:19:23.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:19:23.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:19:23.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:19:23.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:19:23.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:19:23.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:19:23.076 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:19:23.077 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:23.077 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:23.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:23.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:23.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:19:23.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:19:23.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:19:23.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:19:23.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:19:23.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:19:23.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:19:23.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:23.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:19:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:19:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2021-01-15T18:19:23.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:19:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c2def781-1111-4fb3-8bf3-9941b23c582b 2021-01-15T18:19:23.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c2def781-1111-4fb3-8bf3-9941b23c582b' 2021-01-15T18:19:23.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:19:23.081 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 c2def781-1111-4fb3-8bf3-9941b23c582b 2021-01-15T18:19:23.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCr3AFg5UmMBBAAy6p3or1c0b3hwwLXu8M/7g== 2021-01-15T18:19:23.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCr3AFg5UmMBBAAy6p3or1c0b3hwwLXu8M/7g=="}' 2021-01-15T18:19:23.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c2def781-1111-4fb3-8bf3-9941b23c582b -i td/osd-markdown/0/new.json 2021-01-15T18:19:23.431 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:19:23.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2021-01-15T18:19:23.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCr3AFg5UmMBBAAy6p3or1c0b3hwwLXu8M/7g== --osd-uuid c2def781-1111-4fb3-8bf3-9941b23c582b 2021-01-15T18:19:23.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.463+0000 7f767f9cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:23.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.465+0000 7f767f9cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:23.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.466+0000 7f767f9cd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:23.468 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.467+0000 7f767f9cd080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:19:23.468 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.467+0000 7f767f9cd080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:19:23.469 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.467+0000 7f767f9cd080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:19:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:23.510+0000 7f767f9cd080 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-01-15T18:19:28.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-01-15T18:19:28.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:19:28.087 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:19:28.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:19:28.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:19:28.606 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:19:28.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:19:28.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:28.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:19:28.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:19:28.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:19:28.625 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:28.624+0000 7f71f2b03080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:28.651 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:28.650+0000 7f71f2b03080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:28.653 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:28.652+0000 7f71f2b03080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:28.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:19:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:19:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:19:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:19:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:19:29.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:29.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:19:29.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:29.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:19:29.001 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:19:29.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:29.167+0000 7f71f2b03080 -1 Falling back to public interface 2021-01-15T18:19:29.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:19:29.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:29.953+0000 7f71f2b03080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:19:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:19:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:19:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:19:30.402 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:19:30.818 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/753959,v1:127.0.0.1:6803/753959] [v2:127.0.0.1:6804/753959,v1:127.0.0.1:6805/753959] exists,up c2def781-1111-4fb3-8bf3-9941b23c582b 2021-01-15T18:19:30.818 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 81914ea7-c9a7-4af5-8356-53db6d72f7bb 2021-01-15T18:19:30.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:19:30.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:19:30.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:19:30.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2021-01-15T18:19:30.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:19:30.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:19:30.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:19:30.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:19:30.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2021-01-15T18:19:30.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:19:30.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:19:30.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:19:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:19:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-01-15T18:19:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-01-15T18:19:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:19:30.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:19:30.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:19:30.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:19:30.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:19:30.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:19:30.823 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:19:30.824 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:30.824 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:30.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:30.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:19:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:19:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:19:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:19:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:19:30.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:19:30.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:19:30.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:30.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:19:30.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:19:30.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2021-01-15T18:19:30.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:19:30.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=81914ea7-c9a7-4af5-8356-53db6d72f7bb 2021-01-15T18:19:30.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 81914ea7-c9a7-4af5-8356-53db6d72f7bb' 2021-01-15T18:19:30.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:19:30.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCy3AFgV/56MRAA3ZxtTvn+B1I421yYEjNNAA== 2021-01-15T18:19:30.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCy3AFgV/56MRAA3ZxtTvn+B1I421yYEjNNAA=="}' 2021-01-15T18:19:30.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 81914ea7-c9a7-4af5-8356-53db6d72f7bb -i td/osd-markdown/1/new.json 2021-01-15T18:19:31.337 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:19:31.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2021-01-15T18:19:31.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCy3AFgV/56MRAA3ZxtTvn+B1I421yYEjNNAA== --osd-uuid 81914ea7-c9a7-4af5-8356-53db6d72f7bb 2021-01-15T18:19:31.370 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.369+0000 7f90c8e07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:31.371 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.371+0000 7f90c8e07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:31.372 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.372+0000 7f90c8e07080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:31.373 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.372+0000 7f90c8e07080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:19:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.372+0000 7f90c8e07080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:19:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.372+0000 7f90c8e07080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:19:31.425 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:31.424+0000 7f90c8e07080 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-01-15T18:19:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-01-15T18:19:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:19:35.918 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:19:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:19:35.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:19:36.439 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:19:36.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:19:36.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:36.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:19:36.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:19:36.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:19:36.458 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:36.457+0000 7fc803bc2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:36.481 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:36.480+0000 7fc803bc2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:36.482 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:36.481+0000 7fc803bc2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:36.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:19:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:19:36.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:19:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:19:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:19:36.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:36.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:19:36.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:36.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:19:36.830 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:19:36.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:36.997+0000 7fc803bc2080 -1 Falling back to public interface 2021-01-15T18:19:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:19:37.790 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:37.790+0000 7fc803bc2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:19:38.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:19:38.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:19:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:19:38.249 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:19:38.654 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/755231,v1:127.0.0.1:6811/755231] [v2:127.0.0.1:6812/755231,v1:127.0.0.1:6813/755231] exists,up 81914ea7-c9a7-4af5-8356-53db6d72f7bb 2021-01-15T18:19:38.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:19:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:19:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:19:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2021-01-15T18:19:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:19:38.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:19:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:19:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:19:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2021-01-15T18:19:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:19:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:19:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:19:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:19:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-01-15T18:19:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-01-15T18:19:38.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:19:38.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:19:38.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:19:38.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:19:38.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:19:38.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:19:38.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:19:38.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:19:38.659 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:19:38.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:38.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:19:38.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:19:38.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:19:38.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:19:38.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:19:38.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:19:38.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:19:38.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:19:38.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:38.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:19:38.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:19:38.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2021-01-15T18:19:38.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:19:38.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a2634f8f-aa01-4e11-b2df-15286a1be305 2021-01-15T18:19:38.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 a2634f8f-aa01-4e11-b2df-15286a1be305' 2021-01-15T18:19:38.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:19:38.664 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 a2634f8f-aa01-4e11-b2df-15286a1be305 2021-01-15T18:19:38.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC63AFgJdrAJxAAJvjf5nUMHstkeP0Z8XPBJw== 2021-01-15T18:19:38.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC63AFgJdrAJxAAJvjf5nUMHstkeP0Z8XPBJw=="}' 2021-01-15T18:19:38.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a2634f8f-aa01-4e11-b2df-15286a1be305 -i td/osd-markdown/2/new.json 2021-01-15T18:19:39.200 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:19:39.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2021-01-15T18:19:39.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC63AFgJdrAJxAAJvjf5nUMHstkeP0Z8XPBJw== --osd-uuid a2634f8f-aa01-4e11-b2df-15286a1be305 2021-01-15T18:19:39.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.233+0000 7f20da07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.235+0000 7f20da07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:39.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.236+0000 7f20da07e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.236+0000 7f20da07e080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:19:39.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.236+0000 7f20da07e080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:19:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.236+0000 7f20da07e080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:19:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:39.271+0000 7f20da07e080 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-01-15T18:19:44.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-01-15T18:19:44.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:19:44.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:19:44.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:19:44.059 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:19:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:19:44.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:19:44.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:19:44.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:19:44.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:19:44.587 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:19:44.604 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:44.602+0000 7f2efe8e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:44.638 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:44.637+0000 7f2efe8e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:44.640 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:44.639+0000 7f2efe8e0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:19:44.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:19:44.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:19:44.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:19:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:19:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:19:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:19:44.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:44.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:19:44.975 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:19:45.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:45.154+0000 7f2efe8e0080 -1 Falling back to public interface 2021-01-15T18:19:45.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:19:45.931 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:19:45.930+0000 7f2efe8e0080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:19:46.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:19:46.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:19:46.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:19:46.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:19:46.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:19:46.411 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:19:46.815 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/756495,v1:127.0.0.1:6819/756495] [v2:127.0.0.1:6820/756495,v1:127.0.0.1:6821/756495] exists,up a2634f8f-aa01-4e11-b2df-15286a1be305 2021-01-15T18:19:46.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:19:46.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:19:46.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:19:46.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2021-01-15T18:19:46.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T18:19:47.235 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T18:19:47.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T18:19:47.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T18:19:47.975 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T18:19:47.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:19:48.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T18:19:50.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2021-01-15T18:19:50.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2021-01-15T18:19:50.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2021-01-15T18:19:50.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2021-01-15T18:19:50.511 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:19:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2021-01-15T18:19:50.648 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:19:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2021-01-15T18:19:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2021-01-15T18:19:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2021-01-15T18:19:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2021-01-15T18:19:50.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2021-01-15T18:19:50.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:19:50.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:19:50.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:19:51.178 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:19:51.178 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:19:51.178 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:19:51.178 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:19:51.179 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:19:51.179 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:19:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:19:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:19:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:19:51.584 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:19:51.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:19:51.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:19:52.015 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:19:52.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:20:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:20:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:20:02.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:20:02.568 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:20:02.568 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:20:02.568 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:20:02.568 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:02.569 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:20:02.569 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:20:02.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:20:02.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:20:02.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:20:02.968 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:02.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:20:02.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:20:03.372 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:20:03.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:20:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:20:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:20:13.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:20:13.906 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:20:13.906 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:20:13.906 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:20:13.907 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:13.907 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:20:13.907 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:20:13.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:20:13.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:20:13.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:20:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:20:14.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:20:14.732 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:20:14.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:20:24.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2021-01-15T18:20:24.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2021-01-15T18:20:24.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2021-01-15T18:20:25.296 INFO:tasks.workunit.client.0.smithi145.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2021-01-15T18:20:25.296 INFO:tasks.workunit.client.0.smithi145.stdout:-1 0.29306 root default 2021-01-15T18:20:25.296 INFO:tasks.workunit.client.0.smithi145.stdout:-3 0.29306 host smithi145 2021-01-15T18:20:25.297 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:25.297 INFO:tasks.workunit.client.0.smithi145.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2021-01-15T18:20:25.297 INFO:tasks.workunit.client.0.smithi145.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2021-01-15T18:20:25.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2021-01-15T18:20:25.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2021-01-15T18:20:25.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2021-01-15T18:20:25.694 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:20:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2021-01-15T18:20:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2021-01-15T18:20:26.134 INFO:tasks.workunit.client.0.smithi145.stderr:marked down osd.0. 2021-01-15T18:20:26.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2021-01-15T18:20:36.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2021-01-15T18:20:36.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2021-01-15T18:20:36.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2021-01-15T18:20:36.541 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-01-15T18:20:36.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-01-15T18:20:36.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:20:36.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:20:36.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:20:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:20:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:20:36.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:20:36.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:20:36.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:20:36.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:20:36.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:20:36.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:20:36.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:20:36.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:20:36.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:20:36.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:20:36.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:20:36.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:20:36.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:20:36.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:20:36.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:20:36.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:20:36.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:20:36.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:20:36.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:36.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:36.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:20:36.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:20:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:20:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2021-01-15T18:20:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2021-01-15T18:20:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2021-01-15T18:20:36.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2021-01-15T18:20:36.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:20:36.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:20:36.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:20:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:20:36.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:20:36.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:20:36.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:20:36.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:20:36.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:20:36.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:20:36.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:20:36.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:20:36.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:20:36.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:20:36.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:20:36.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:20:36.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:20:36.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:20:36.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:20:36.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:20:36.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:20:36.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:20:36.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:20:36.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:36.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:20:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:20:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:20:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2021-01-15T18:20:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:20:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:36.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:36.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.743735 2021-01-15T18:20:36.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:20:36.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:20:36.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:20:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2021-01-15T18:20:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2021-01-15T18:20:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2021-01-15T18:20:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2021-01-15T18:20:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2021-01-15T18:20:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:20:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:20:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:20:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2021-01-15T18:20:36.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2021-01-15T18:20:36.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:20:36.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:20:36.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:20:36.955 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:20:36.955 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:36.956 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:36.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:36.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:20:37.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:20:37.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:20:37.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:20:37.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:20:37.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:20:37.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:20:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:20:37.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:20:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:20:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:20:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:37.132 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:20:37.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:20:37.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get fsid 2021-01-15T18:20:37.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:20:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:20:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:20:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:20:37.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:20:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:20:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:20:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:20:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:20:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:37.234 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.743735/ceph-mon.a.asok 2021-01-15T18:20:37.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:20:37.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.743735/ceph-mon.a.asok config get mon_host 2021-01-15T18:20:37.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2021-01-15T18:20:37.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-markdown 2021-01-15T18:20:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:20:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:20:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:20:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-markdown/x 2021-01-15T18:20:37.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:20:37.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:20:37.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:20:37.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:20:37.824 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:20:37.824 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:37.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:37.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:37.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:20:37.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:20:37.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2021-01-15T18:20:37.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:20:37.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:20:37.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:20:37.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:20:37.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/0 2021-01-15T18:20:37.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:20:37.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:20:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:20:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:20:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2021-01-15T18:20:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2021-01-15T18:20:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:20:37.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:20:37.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:20:37.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:20:37.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:20:37.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:20:37.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:20:37.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:37.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:37.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:37.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:37.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:20:37.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:20:37.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:20:37.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:20:37.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:20:37.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:20:37.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:20:37.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:37.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:20:37.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:20:37.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/0 2021-01-15T18:20:37.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:20:37.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=277e0362-ca04-4ae9-ad42-97e4aa7921dc 2021-01-15T18:20:37.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 277e0362-ca04-4ae9-ad42-97e4aa7921dc' 2021-01-15T18:20:37.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:20:37.855 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 277e0362-ca04-4ae9-ad42-97e4aa7921dc 2021-01-15T18:20:37.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD13AFgDAZOMxAAmB2d0T8sGuFH8VFBpcywcw== 2021-01-15T18:20:37.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD13AFgDAZOMxAAmB2d0T8sGuFH8VFBpcywcw=="}' 2021-01-15T18:20:37.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 277e0362-ca04-4ae9-ad42-97e4aa7921dc -i td/osd-markdown/0/new.json 2021-01-15T18:20:38.197 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:20:38.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/0/new.json 2021-01-15T18:20:38.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD13AFgDAZOMxAAmB2d0T8sGuFH8VFBpcywcw== --osd-uuid 277e0362-ca04-4ae9-ad42-97e4aa7921dc 2021-01-15T18:20:38.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.229+0000 7fb4f6a28080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:38.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.231+0000 7fb4f6a28080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:38.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.233+0000 7fb4f6a28080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:38.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.234+0000 7fb4f6a28080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:20:38.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.234+0000 7fb4f6a28080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:20:38.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.234+0000 7fb4f6a28080 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2021-01-15T18:20:38.272 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:38.271+0000 7fb4f6a28080 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2021-01-15T18:20:42.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/0/keyring 2021-01-15T18:20:42.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:20:42.899 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:20:42.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:20:42.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:20:43.425 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:20:43.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:20:43.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:43.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:20:43.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:20:43.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:20:43.444 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:43.443+0000 7f76924dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:43.471 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:43.469+0000 7f76924dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:43.472 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:43.472+0000 7f76924dd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:43.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:20:43.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:20:43.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:20:43.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:20:43.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:20:43.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:20:43.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:20:43.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:20:43.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:20:43.816 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:20:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:43.988+0000 7f76924dd080 -1 Falling back to public interface 2021-01-15T18:20:44.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:20:44.755 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:44.755+0000 7f76924dd080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:20:45.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:20:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:20:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:20:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:20:45.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:20:45.224 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:20:45.617 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/758586,v1:127.0.0.1:6803/758586] [v2:127.0.0.1:6804/758586,v1:127.0.0.1:6805/758586] exists,up 277e0362-ca04-4ae9-ad42-97e4aa7921dc 2021-01-15T18:20:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:20:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:20:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:20:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2021-01-15T18:20:45.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:20:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:20:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:20:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:20:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/1 2021-01-15T18:20:45.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:20:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:20:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:20:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:20:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2021-01-15T18:20:45.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2021-01-15T18:20:45.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:20:45.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:20:45.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:20:45.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:20:45.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:20:45.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:20:45.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:20:45.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:45.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:45.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:20:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:20:45.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:20:45.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:20:45.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:20:45.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:20:45.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:20:45.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:45.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:20:45.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:20:45.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/1 2021-01-15T18:20:45.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:20:45.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=048a4b14-3091-4444-82c1-142946b9aa4e 2021-01-15T18:20:45.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 048a4b14-3091-4444-82c1-142946b9aa4e' 2021-01-15T18:20:45.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:20:45.627 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 048a4b14-3091-4444-82c1-142946b9aa4e 2021-01-15T18:20:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD93AFgSfqWJRAAHGRY/2JLVPFf2wlUVeYITg== 2021-01-15T18:20:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD93AFgSfqWJRAAHGRY/2JLVPFf2wlUVeYITg=="}' 2021-01-15T18:20:45.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 048a4b14-3091-4444-82c1-142946b9aa4e -i td/osd-markdown/1/new.json 2021-01-15T18:20:46.181 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:20:46.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/1/new.json 2021-01-15T18:20:46.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD93AFgSfqWJRAAHGRY/2JLVPFf2wlUVeYITg== --osd-uuid 048a4b14-3091-4444-82c1-142946b9aa4e 2021-01-15T18:20:46.214 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.213+0000 7efe99407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:46.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.215+0000 7efe99407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:46.217 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.216+0000 7efe99407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:46.217 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.216+0000 7efe99407080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:20:46.217 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.216+0000 7efe99407080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:20:46.218 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.216+0000 7efe99407080 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2021-01-15T18:20:46.268 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:46.268+0000 7efe99407080 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2021-01-15T18:20:50.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/1/keyring 2021-01-15T18:20:50.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:20:50.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:20:50.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:20:50.736 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:20:51.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:20:51.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:51.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:20:51.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:20:51.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:20:51.260 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:20:51.279 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:51.278+0000 7feb698b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:51.300 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:51.299+0000 7feb698b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:51.301 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:51.301+0000 7feb698b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:20:51.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:20:51.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:20:51.665 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:20:51.816 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:51.815+0000 7feb698b2080 -1 Falling back to public interface 2021-01-15T18:20:52.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:20:52.610 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:52.609+0000 7feb698b2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:20:53.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:20:53.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:20:53.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:20:53.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:20:53.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:20:53.074 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:20:53.486 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/759850,v1:127.0.0.1:6811/759850] [v2:127.0.0.1:6812/759850,v1:127.0.0.1:6813/759850] exists,up 048a4b14-3091-4444-82c1-142946b9aa4e 2021-01-15T18:20:53.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:20:53.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:20:53.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:20:53.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2021-01-15T18:20:53.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-markdown 2021-01-15T18:20:53.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:20:53.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:20:53.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:20:53.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-markdown/2 2021-01-15T18:20:53.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2021-01-15T18:20:53.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:20:53.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:20:53.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:20:53.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2021-01-15T18:20:53.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2021-01-15T18:20:53.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:20:53.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:20:53.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2021-01-15T18:20:53.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:20:53.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:20:53.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:20:53.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:20:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:20:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:20:53.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' 2021-01-15T18:20:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:20:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:20:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:20:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2021-01-15T18:20:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2021-01-15T18:20:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:20:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:20:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:20:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:20:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-markdown/2 2021-01-15T18:20:53.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:20:53.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=54c4e72b-0175-45ed-88e3-52043b190525 2021-01-15T18:20:53.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 54c4e72b-0175-45ed-88e3-52043b190525' 2021-01-15T18:20:53.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:20:53.495 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 54c4e72b-0175-45ed-88e3-52043b190525 2021-01-15T18:20:53.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAF3QFgica8HRAA/XZ0KN86j5ALu+uSzStrGA== 2021-01-15T18:20:53.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAF3QFgica8HRAA/XZ0KN86j5ALu+uSzStrGA=="}' 2021-01-15T18:20:53.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 54c4e72b-0175-45ed-88e3-52043b190525 -i td/osd-markdown/2/new.json 2021-01-15T18:20:54.076 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:20:54.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-markdown/2/new.json 2021-01-15T18:20:54.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAF3QFgica8HRAA/XZ0KN86j5ALu+uSzStrGA== --osd-uuid 54c4e72b-0175-45ed-88e3-52043b190525 2021-01-15T18:20:54.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.109+0000 7fca94591080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:54.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.110+0000 7fca94591080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:54.112 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.112+0000 7fca94591080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:54.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.112+0000 7fca94591080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:20:54.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.112+0000 7fca94591080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:20:54.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.112+0000 7fca94591080 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2021-01-15T18:20:54.162 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:54.161+0000 7fca94591080 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2021-01-15T18:20:58.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-markdown/2/keyring 2021-01-15T18:20:58.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:20:58.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:20:58.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:20:58.765 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:20:59.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:20:59.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=acceac3d-a79e-430b-8401-507e67149660 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.743735/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:20:59.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:20:59.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:20:59.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:20:59.285 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:20:59.303 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:59.302+0000 7fcaf1c26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:59.304 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:59.304+0000 7fcaf1c26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:59.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:59.306+0000 7fcaf1c26080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:20:59.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:20:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:20:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:20:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:20:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:20:59.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:20:59.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:20:59.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:20:59.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:20:59.687 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:20:59.823 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:20:59.822+0000 7fcaf1c26080 -1 Falling back to public interface 2021-01-15T18:21:00.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:21:00.608 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:00.607+0000 7fcaf1c26080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:21:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:21:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:21:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:21:01.089 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:21:01.484 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/761114,v1:127.0.0.1:6819/761114] [v2:127.0.0.1:6820/761114,v1:127.0.0.1:6821/761114] exists,up 54c4e72b-0175-45ed-88e3-52043b190525 2021-01-15T18:21:01.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:21:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:21:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:21:01.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2021-01-15T18:21:01.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=758586 2021-01-15T18:21:01.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 758586 2021-01-15T18:21:01.491 INFO:tasks.workunit.client.0.smithi145.stdout: PID TTY TIME CMD 2021-01-15T18:21:01.491 INFO:tasks.workunit.client.0.smithi145.stdout: 758586 ? 00:00:00 ceph-osd 2021-01-15T18:21:01.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2021-01-15T18:21:01.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2021-01-15T18:21:01.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2021-01-15T18:21:01.880 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2021-01-15T18:21:01.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2021-01-15T18:21:02.424 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already stopped. 2021-01-15T18:21:02.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2021-01-15T18:21:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2021-01-15T18:21:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2021-01-15T18:21:17.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2021-01-15T18:21:17.831 INFO:tasks.workunit.client.0.smithi145.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2021-01-15T18:21:17.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 758586 2021-01-15T18:21:17.837 INFO:tasks.workunit.client.0.smithi145.stdout: PID TTY TIME CMD 2021-01-15T18:21:17.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2021-01-15T18:21:17.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:21:17.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:21:17.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:21:17.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:17.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:17.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:17.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:17.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:17.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:17.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:17.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:17.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:17.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:17.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:17.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:17.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:17.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:17.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:17.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:21:17.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:21:17.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:17.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:17.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:21:17.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:21:17.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:17.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:17.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:21:17.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-markdown 0 2021-01-15T18:21:17.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2021-01-15T18:21:17.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:21:17.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2021-01-15T18:21:17.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:17.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:17.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:17.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:17.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:17.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:17.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:17.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:17.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:17.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:17.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:17.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:17.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:17.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:21:17.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2021-01-15T18:21:17.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:17.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:17.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.743735 2021-01-15T18:21:17.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.743735 2021-01-15T18:21:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:17.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:17.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:21:17.966 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:21:17.966 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:21:17.999 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2021-01-15T18:21:17.999 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2021-01-15T18:21:18.060 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:21:18.067 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:21:18.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:21:18.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:21:18.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:21:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:21:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:21:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:21:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:21:18.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:21:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-reactivate 2021-01-15T18:21:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2021-01-15T18:21:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2021-01-15T18:21:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2021-01-15T18:21:18.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2021-01-15T18:21:18.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2021-01-15T18:21:18.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2021-01-15T18:21:18.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none ' 2021-01-15T18:21:18.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2021-01-15T18:21:18.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2021-01-15T18:21:18.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:21:18.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2021-01-15T18:21:18.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2021-01-15T18:21:18.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2021-01-15T18:21:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2021-01-15T18:21:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2021-01-15T18:21:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-01-15T18:21:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:21:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-01-15T18:21:18.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:18.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:18.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:18.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:18.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:18.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:18.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:18.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:18.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:18.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:21:18.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-01-15T18:21:18.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:18.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:18.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:18.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.761751 2021-01-15T18:21:18.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:18.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:18.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2021-01-15T18:21:18.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:21:18.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:18.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:18.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.761751 2021-01-15T18:21:18.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:21:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:21:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:21:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:21:18.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2021-01-15T18:21:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2021-01-15T18:21:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2021-01-15T18:21:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2021-01-15T18:21:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2021-01-15T18:21:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:21:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:21:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:21:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2021-01-15T18:21:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2021-01-15T18:21:18.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:21:18.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:18.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:18.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:18.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:18.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:18.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:18.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:21:18.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:21:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:21:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:21:18.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:21:18.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:21:18.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:21:18.527 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:21:18.527 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:21:18.528 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:21:18.528 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:21:18.528 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:18.528 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:18.528 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.761751/ceph-mon.a.asok 2021-01-15T18:21:18.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:21:18.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.761751/ceph-mon.a.asok config get fsid 2021-01-15T18:21:18.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:21:18.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:21:18.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:21:18.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:21:18.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:21:18.630 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:21:18.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:21:18.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:21:18.631 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:21:18.632 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:18.632 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:18.632 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.761751/ceph-mon.a.asok 2021-01-15T18:21:18.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:21:18.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.761751/ceph-mon.a.asok config get mon_host 2021-01-15T18:21:18.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2021-01-15T18:21:18.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-reactivate 2021-01-15T18:21:18.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:21:18.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:21:18.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:21:18.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-reactivate/x 2021-01-15T18:21:18.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:21:19.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:21:19.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:19.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:19.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:19.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:19.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:19.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:19.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:21:19.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:21:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2021-01-15T18:21:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-reactivate 2021-01-15T18:21:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:21:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:21:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:21:19.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-reactivate/0 2021-01-15T18:21:19.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-01-15T18:21:19.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:21:19.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:21:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:21:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-01-15T18:21:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-01-15T18:21:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:21:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:21:19.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-01-15T18:21:19.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:21:19.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:19.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:19.245 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:19.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:21:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:21:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:21:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-01-15T18:21:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-01-15T18:21:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:21:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:21:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:21:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:21:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-reactivate/0 2021-01-15T18:21:19.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:21:19.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=20d85371-2f4b-44ba-a763-476ca5751e43 2021-01-15T18:21:19.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 20d85371-2f4b-44ba-a763-476ca5751e43' 2021-01-15T18:21:19.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:21:19.250 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 20d85371-2f4b-44ba-a763-476ca5751e43 2021-01-15T18:21:19.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAf3QFgYqIwDxAAC4VDyswZwYsYk8uN8q5FPw== 2021-01-15T18:21:19.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAf3QFgYqIwDxAAC4VDyswZwYsYk8uN8q5FPw=="}' 2021-01-15T18:21:19.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 20d85371-2f4b-44ba-a763-476ca5751e43 -i td/osd-reactivate/0/new.json 2021-01-15T18:21:19.617 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:19.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-reactivate/0/new.json 2021-01-15T18:21:19.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAf3QFgYqIwDxAAC4VDyswZwYsYk8uN8q5FPw== --osd-uuid 20d85371-2f4b-44ba-a763-476ca5751e43 2021-01-15T18:21:19.650 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.649+0000 7f0a1b2c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:19.652 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.651+0000 7f0a1b2c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:19.653 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.652+0000 7f0a1b2c5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:19.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.653+0000 7f0a1b2c5080 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-01-15T18:21:19.654 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.653+0000 7f0a1b2c5080 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-01-15T18:21:19.655 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.653+0000 7f0a1b2c5080 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2021-01-15T18:21:19.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:19.697+0000 7f0a1b2c5080 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2021-01-15T18:21:24.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-reactivate/0/keyring 2021-01-15T18:21:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:21:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:21:24.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:21:24.292 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:21:24.812 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:21:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:21:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:21:24.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:21:24.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:21:24.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:24.831+0000 7f832feb7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:24.854+0000 7f832feb7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:24.855+0000 7f832feb7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:21:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:21:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:21:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:21:25.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:21:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:21:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:21:25.207 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:25.372 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:25.371+0000 7f832feb7080 -1 Falling back to public interface 2021-01-15T18:21:25.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:21:26.140 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:26.139+0000 7f832feb7080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:21:26.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:21:26.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:26.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:21:26.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:26.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:21:26.612 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:21:27.023 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/762790,v1:127.0.0.1:6803/762790] [v2:127.0.0.1:6804/762790,v1:127.0.0.1:6805/762790] exists,up 20d85371-2f4b-44ba-a763-476ca5751e43 2021-01-15T18:21:27.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:21:27.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:21:27.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:21:27.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2021-01-15T18:21:27.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:27.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:27.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:27.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:27.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:27.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:27.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2021-01-15T18:21:27.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2021-01-15T18:21:27.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2021-01-15T18:21:27.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2021-01-15T18:21:27.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-reactivate 2021-01-15T18:21:27.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:21:27.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:21:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:21:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-reactivate/0 2021-01-15T18:21:27.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2021-01-15T18:21:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:21:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:21:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:21:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2021-01-15T18:21:27.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2021-01-15T18:21:27.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:21:27.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:21:27.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2021-01-15T18:21:27.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:21:27.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:27.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:27.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:27.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:27.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:27.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' 2021-01-15T18:21:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:21:27.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2021-01-15T18:21:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2021-01-15T18:21:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:21:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:21:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:21:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:21:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-reactivate/0 2021-01-15T18:21:27.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:21:27.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=4b6fabae-db0f-4197-9450-3022fe5876b1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.761751/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:27.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-reactivate/0/whoami 2021-01-15T18:21:27.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:21:27.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:21:27.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:21:27.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:21:27.151 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:21:27.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:27.148+0000 7ff7ce896080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:27.168+0000 7ff7ce896080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:27.169+0000 7ff7ce896080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:27.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:21:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:21:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:21:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:21:27.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:21:27.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:27.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:21:27.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:27.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:21:27.522 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:27.688 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:27.687+0000 7ff7ce896080 -1 Falling back to public interface 2021-01-15T18:21:27.917 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/762790,v1:127.0.0.1:6803/762790] [v2:127.0.0.1:6804/762790,v1:127.0.0.1:6805/762790] exists,up 20d85371-2f4b-44ba-a763-476ca5751e43 2021-01-15T18:21:27.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:21:27.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:21:27.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:21:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2021-01-15T18:21:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-01-15T18:21:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:21:27.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-01-15T18:21:27.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:27.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:27.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:27.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:27.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:28.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:28.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:28.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:28.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:28.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:28.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:28.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:28.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:28.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:21:28.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-01-15T18:21:28.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:28.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:28.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.761751 2021-01-15T18:21:28.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:28.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:28.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:21:28.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-reactivate 0 2021-01-15T18:21:28.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2021-01-15T18:21:28.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:21:28.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2021-01-15T18:21:28.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:28.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:28.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:28.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:28.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:28.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:28.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:28.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:28.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:28.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:28.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:28.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:28.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:28.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:28.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:28.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:21:28.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2021-01-15T18:21:28.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:28.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.761751 2021-01-15T18:21:28.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.761751 2021-01-15T18:21:28.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:28.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:28.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:21:28.044 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:21:28.045 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:21:28.107 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2021-01-15T18:21:28.107 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2021-01-15T18:21:28.166 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:21:28.176 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:21:28.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:21:28.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:21:28.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:21:28.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:21:28.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:21:28.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:21:28.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:21:28.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:21:28.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-recovery-prio 2021-01-15T18:21:28.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2021-01-15T18:21:28.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-01-15T18:21:28.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-01-15T18:21:28.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2021-01-15T18:21:28.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2021-01-15T18:21:28.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none ' 2021-01-15T18:21:28.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:21:28.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: export objects=200 2021-01-15T18:21:28.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:29: run: objects=200 2021-01-15T18:21:28.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: export poolprefix=test 2021-01-15T18:21:28.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:30: run: poolprefix=test 2021-01-15T18:21:28.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: export FORCE_PRIO=255 2021-01-15T18:21:28.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: FORCE_PRIO=255 2021-01-15T18:21:28.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export NORMAL_PRIO=190 2021-01-15T18:21:28.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: NORMAL_PRIO=190 2021-01-15T18:21:28.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: set 2021-01-15T18:21:28.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:21:28.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: local 'funcs=TEST_recovery_pool_priority 2021-01-15T18:21:28.184 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_priority' 2021-01-15T18:21:28.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-01-15T18:21:28.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-01-15T18:21:28.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-01-15T18:21:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:21:28.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-01-15T18:21:28.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:21:28.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:21:28.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:21:28.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:21:28.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:21:28.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:21:28.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:21:28.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:21:28.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:21:28.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:21:28.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:21:28.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:21:28.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:21:28.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:21:28.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:21:28.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:21:28.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:21:28.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-01-15T18:21:28.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:21:28.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.763429 2021-01-15T18:21:28.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:21:28.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:21:28.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-01-15T18:21:28.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:21:28.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:28.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.763429 2021-01-15T18:21:28.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:21:28.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:21:28.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:21:28.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:21:28.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-01-15T18:21:28.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_pool_priority td/osd-recovery-prio 2021-01-15T18:21:28.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:323: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:324: TEST_recovery_pool_priority: local pools=3 2021-01-15T18:21:28.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:325: TEST_recovery_pool_priority: local OSDS=2 2021-01-15T18:21:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2021-01-15T18:21:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:21:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:21:28.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:21:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-01-15T18:21:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-01-15T18:21:28.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:21:28.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:28.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:28.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:21:28.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:21:28.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:21:28.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:21:28.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:21:28.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:21:28.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:21:28.662 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:21:28.662 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:21:28.663 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:21:28.663 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:21:28.663 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.663 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:28.663 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-mon.a.asok 2021-01-15T18:21:28.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:21:28.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.763429/ceph-mon.a.asok config get fsid 2021-01-15T18:21:28.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:21:28.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:21:28.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:21:28.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:21:28.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:21:28.762 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:21:28.762 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:28.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-mon.a.asok 2021-01-15T18:21:28.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:21:28.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.763429/ceph-mon.a.asok config get mon_host 2021-01-15T18:21:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2021-01-15T18:21:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-prio 2021-01-15T18:21:28.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:21:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:21:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:21:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-prio/x 2021-01-15T18:21:28.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:21:29.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:21:29.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:29.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:29.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:29.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:29.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:29.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:29.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:21:29.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:21:29.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:329: TEST_recovery_pool_priority: export CEPH_ARGS 2021-01-15T18:21:29.776 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: expr 2 - 1 2021-01-15T18:21:29.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: seq 0 1 2021-01-15T18:21:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:21:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2021-01-15T18:21:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:21:29.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:21:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:21:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:21:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/0 2021-01-15T18:21:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:21:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:21:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:21:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:21:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-01-15T18:21:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-01-15T18:21:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:21:29.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:21:29.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:21:29.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:21:29.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:29.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:29.785 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:29.785 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:29.785 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:29.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:29.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:21:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:21:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:21:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:21:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:21:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:21:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:21:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:21:29.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:21:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/0 2021-01-15T18:21:29.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:21:29.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a 2021-01-15T18:21:29.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a' 2021-01-15T18:21:29.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:21:29.789 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a 2021-01-15T18:21:29.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAp3QFg1hVsLxAAGXjmggc4Dx677bulgKVJ9g== 2021-01-15T18:21:29.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAp3QFg1hVsLxAAGXjmggc4Dx677bulgKVJ9g=="}' 2021-01-15T18:21:29.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a -i td/osd-recovery-prio/0/new.json 2021-01-15T18:21:30.135 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:30.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/0/new.json 2021-01-15T18:21:30.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAp3QFg1hVsLxAAGXjmggc4Dx677bulgKVJ9g== --osd-uuid 52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a 2021-01-15T18:21:30.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.170+0000 7f2015fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:30.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.172+0000 7f2015fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:30.174 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.174+0000 7f2015fd4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:30.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.175+0000 7f2015fd4080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:21:30.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.175+0000 7f2015fd4080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:21:30.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.175+0000 7f2015fd4080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:21:30.218 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:30.217+0000 7f2015fd4080 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-01-15T18:21:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-01-15T18:21:34.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:21:34.753 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:21:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:21:34.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:21:35.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:21:35.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:21:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:21:35.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:21:35.284 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:21:35.304 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:35.303+0000 7fb469c4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:35.325 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:35.324+0000 7fb469c4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:35.326 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:35.325+0000 7fb469c4b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:35.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:21:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:21:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:21:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:21:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:21:35.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:35.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:21:35.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:35.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:21:35.680 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:35.846 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:35.845+0000 7fb469c4b080 -1 Falling back to public interface 2021-01-15T18:21:36.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:21:36.675 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:36.674+0000 7fb469c4b080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:21:37.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:21:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:21:37.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:37.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:21:37.085 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:21:37.505 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/764479,v1:127.0.0.1:6803/764479] [v2:127.0.0.1:6804/764479,v1:127.0.0.1:6805/764479] exists,up 52d9ffa1-1f00-49e6-89c8-7c1b8f1c790a 2021-01-15T18:21:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:21:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:21:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:21:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:21:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:333: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2021-01-15T18:21:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:21:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:21:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:21:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:21:37.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/1 2021-01-15T18:21:37.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:21:37.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:21:37.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:21:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:21:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-01-15T18:21:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-01-15T18:21:37.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:21:37.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:21:37.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:21:37.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:21:37.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:21:37.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:21:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:21:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:21:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:21:37.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:21:37.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:21:37.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:21:37.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:21:37.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:21:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:21:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:21:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:21:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:37.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:21:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:21:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/1 2021-01-15T18:21:37.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:21:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6dd10c0b-ffcc-426d-a0b7-9472d7271391 2021-01-15T18:21:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 6dd10c0b-ffcc-426d-a0b7-9472d7271391' 2021-01-15T18:21:37.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:21:37.518 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 6dd10c0b-ffcc-426d-a0b7-9472d7271391 2021-01-15T18:21:37.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAx3QFgTg8bHxAAGZ05Khiljz55v9kOQIW3TQ== 2021-01-15T18:21:37.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAx3QFgTg8bHxAAGZ05Khiljz55v9kOQIW3TQ=="}' 2021-01-15T18:21:37.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6dd10c0b-ffcc-426d-a0b7-9472d7271391 -i td/osd-recovery-prio/1/new.json 2021-01-15T18:21:38.048 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:21:38.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/1/new.json 2021-01-15T18:21:38.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAx3QFgTg8bHxAAGZ05Khiljz55v9kOQIW3TQ== --osd-uuid 6dd10c0b-ffcc-426d-a0b7-9472d7271391 2021-01-15T18:21:38.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.082+0000 7f3fb6e2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.084+0000 7f3fb6e2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:38.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.086+0000 7f3fb6e2e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:38.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.086+0000 7f3fb6e2e080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:21:38.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.086+0000 7f3fb6e2e080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:21:38.088 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.086+0000 7f3fb6e2e080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:21:38.139 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:38.139+0000 7f3fb6e2e080 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-01-15T18:21:42.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-01-15T18:21:42.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:21:42.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:21:42.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:21:42.684 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:21:43.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:21:43.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:21:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:21:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:21:43.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:21:43.206 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:21:43.226 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:43.225+0000 7f778ff2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:43.228 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:43.227+0000 7f778ff2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:43.230 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:43.229+0000 7f778ff2d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:21:43.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:21:43.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:21:43.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:21:43.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:21:43.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:21:43.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:43.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:21:43.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:43.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:21:43.600 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:21:43.747 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:43.747+0000 7f778ff2d080 -1 Falling back to public interface 2021-01-15T18:21:44.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:21:44.538 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:21:44.537+0000 7f778ff2d080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:21:45.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:21:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:21:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:21:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:21:45.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:21:45.011 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:21:45.404 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/765743,v1:127.0.0.1:6811/765743] [v2:127.0.0.1:6812/765743,v1:127.0.0.1:6813/765743] exists,up 6dd10c0b-ffcc-426d-a0b7-9472d7271391 2021-01-15T18:21:45.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:21:45.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:21:45.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:21:45.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: seq 1 3 2021-01-15T18:21:45.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:21:45.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test1 1 1 2021-01-15T18:21:45.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T18:21:46.018 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T18:21:46.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:21:47.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-01-15T18:21:47.718 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:21:47.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:21:47.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test2 1 1 2021-01-15T18:21:47.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T18:21:48.289 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T18:21:48.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:21:49.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-01-15T18:21:50.075 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T18:21:50.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:21:50.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:338: TEST_recovery_pool_priority: create_pool test3 1 1 2021-01-15T18:21:50.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T18:21:50.703 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T18:21:50.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:21:51.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2021-01-15T18:21:52.492 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T18:21:52.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: sleep 5 2021-01-15T18:21:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:343: TEST_recovery_pool_priority: wait_for_clean 2021-01-15T18:21:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:21:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:21:57.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:21:57.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:21:57.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:21:57.505 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:21:57.505 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:21:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:21:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:21:57.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:21:57.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:21:57.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:21:57.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:21:57.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:21:57.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:21:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:21:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T18:21:58.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:21:58.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:21:58.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:21:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T18:21:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T18:21:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T18:21:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:21:58.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:21:58.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T18:21:58.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T18:21:58.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T18:21:58.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:21:58.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T18:21:58.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:21:58.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:21:58.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T18:21:58.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:21:58.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T18:21:58.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T18:21:58.354 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T18:21:58.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:21:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836484 -lt 21474836487 2021-01-15T18:21:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:21:59.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:21:59.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:22:00.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T18:22:00.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:22:00.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T18:22:00.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:22:00.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:22:00.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T18:22:00.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:22:00.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T18:22:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T18:22:00.158 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T18:22:00.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:22:00.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2021-01-15T18:22:00.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:22:00.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:22:00.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:22:01.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T18:22:01.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:22:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:22:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:22:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:22:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:22:01.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:22:01.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:22:01.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T18:22:01.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:22:01.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:22:01.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:22:01.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T18:22:01.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:22:01.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:22:01.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:345: TEST_recovery_pool_priority: ceph pg dump pgs 2021-01-15T18:22:02.338 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:22:02.338 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:21:52.706661+0000 0'0 24:16 [1,0] 1 [1,0] 1 0'0 2021-01-15T18:21:50.465278+0000 0'0 2021-01-15T18:21:50.465278+0000 0 2021-01-15T18:22:02.339 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:21:47.958167+0000 0'0 23:21 [1,0] 1 [1,0] 1 0'0 2021-01-15T18:21:45.793733+0000 0'0 2021-01-15T18:21:45.793733+0000 0 2021-01-15T18:22:02.339 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:21:50.675359+0000 0'0 23:20 [0,1] 0 [0,1] 0 0'0 2021-01-15T18:21:48.017069+0000 0'0 2021-01-15T18:21:48.017069+0000 0 2021-01-15T18:22:02.339 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:22:02.339 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:22:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:22:02.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:349: TEST_recovery_pool_priority: local PG1 2021-01-15T18:22:02.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:350: TEST_recovery_pool_priority: local POOLNUM1 2021-01-15T18:22:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:351: TEST_recovery_pool_priority: local pool1 2021-01-15T18:22:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local chk_osd1_1 2021-01-15T18:22:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local chk_osd1_2 2021-01-15T18:22:02.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local PG2 2021-01-15T18:22:02.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local POOLNUM2 2021-01-15T18:22:02.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:357: TEST_recovery_pool_priority: local pool2 2021-01-15T18:22:02.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local chk_osd2_1 2021-01-15T18:22:02.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local chk_osd2_2 2021-01-15T18:22:02.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: seq 1 3 2021-01-15T18:22:02.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:22:02.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2021-01-15T18:22:02.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-01-15T18:22:02.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:22:02.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=1 2021-01-15T18:22:02.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:22:02.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=0 2021-01-15T18:22:02.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z '' ']' 2021-01-15T18:22:02.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: PG1=1.0 2021-01-15T18:22:02.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: POOLNUM1=1 2021-01-15T18:22:02.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:370: TEST_recovery_pool_priority: pool1=test1 2021-01-15T18:22:02.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: chk_osd1_1=1 2021-01-15T18:22:02.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: chk_osd1_2=0 2021-01-15T18:22:02.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:22:02.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2021-01-15T18:22:02.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:363: TEST_recovery_pool_priority: jq '.acting[]' 2021-01-15T18:22:03.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:22:03.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: local test_osd1=0 2021-01-15T18:22:03.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:22:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:365: TEST_recovery_pool_priority: local test_osd2=1 2021-01-15T18:22:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2021-01-15T18:22:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2021-01-15T18:22:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: PG2=2.0 2021-01-15T18:22:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: POOLNUM2=2 2021-01-15T18:22:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:377: TEST_recovery_pool_priority: pool2=test2 2021-01-15T18:22:03.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: chk_osd2_1=0 2021-01-15T18:22:03.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: chk_osd2_2=1 2021-01-15T18:22:03.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: break 2021-01-15T18:22:03.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2021-01-15T18:22:03.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:385: TEST_recovery_pool_priority: '[' test2 = '' ']' 2021-01-15T18:22:03.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: seq 1 3 2021-01-15T18:22:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:22:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-01-15T18:22:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:22:03.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-01-15T18:22:03.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:391: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2021-01-15T18:22:03.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:393: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-01-15T18:22:03.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:395: TEST_recovery_pool_priority: delete_pool test3 2021-01-15T18:22:03.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T18:22:03.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T18:22:03.653 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T18:22:03.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:399: TEST_recovery_pool_priority: pool1_extra_prio=1 2021-01-15T18:22:03.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:400: TEST_recovery_pool_priority: pool2_extra_prio=2 2021-01-15T18:22:03.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: expr 190 + 1 2021-01-15T18:22:03.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:401: TEST_recovery_pool_priority: pool1_prio=191 2021-01-15T18:22:03.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: expr 190 + 2 2021-01-15T18:22:03.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool2_prio=192 2021-01-15T18:22:03.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T18:22:04.360 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T18:22:04.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-01-15T18:22:05.146 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 recovery_priority to 1 2021-01-15T18:22:05.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:406: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-01-15T18:22:05.908 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 1 2021-01-15T18:22:05.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-01-15T18:22:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 recovery_priority to 2 2021-01-15T18:22:06.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: wait_for_clean 2021-01-15T18:22:06.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:22:06.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:22:06.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:22:06.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:22:06.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:22:06.598 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:22:06.598 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:22:06.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:22:06.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:22:06.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:22:06.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:22:06.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:22:06.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:22:06.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:22:06.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:22:07.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:22:07.075 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T18:22:07.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:22:07.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:22:07.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:22:07.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T18:22:07.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T18:22:07.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T18:22:07.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:22:07.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:22:07.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672969 2021-01-15T18:22:07.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672969 2021-01-15T18:22:07.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2021-01-15T18:22:07.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:22:07.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T18:22:07.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:22:07.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:22:07.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T18:22:07.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:22:07.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T18:22:07.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T18:22:07.365 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T18:22:07.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:22:07.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836490 -lt 21474836490 2021-01-15T18:22:07.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:22:07.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2021-01-15T18:22:07.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:22:07.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:22:07.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672969 2021-01-15T18:22:07.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:22:07.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672969 2021-01-15T18:22:07.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-01-15T18:22:07.758 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672969 2021-01-15T18:22:07.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:22:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672969 -lt 42949672969 2021-01-15T18:22:08.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:22:08.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:22:08.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:22:08.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T18:22:08.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:22:08.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:22:08.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:22:08.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:22:08.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:22:08.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:22:08.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:22:09.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:22:09.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:22:09.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:22:09.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:22:09.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T18:22:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:22:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:22:09.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-01-15T18:22:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2021-01-15T18:22:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2021-01-15T18:22:09.596 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0455664 s, 230 MB/s 2021-01-15T18:22:09.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: p=1 2021-01-15T18:22:09.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-01-15T18:22:09.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-01-15T18:22:09.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:09.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-01-15T18:22:09.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:09.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-01-15T18:22:10.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:10.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-01-15T18:22:10.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:10.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-01-15T18:22:10.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:10.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-01-15T18:22:11.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:11.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-01-15T18:22:11.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:11.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-01-15T18:22:12.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:12.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-01-15T18:22:12.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:12.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-01-15T18:22:12.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:12.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-01-15T18:22:13.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:13.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-01-15T18:22:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-01-15T18:22:13.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:13.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-01-15T18:22:14.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:14.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-01-15T18:22:14.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:14.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-01-15T18:22:15.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:15.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-01-15T18:22:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-01-15T18:22:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:16.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-01-15T18:22:16.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-01-15T18:22:16.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:16.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-01-15T18:22:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-01-15T18:22:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-01-15T18:22:17.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:17.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-01-15T18:22:18.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:18.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-01-15T18:22:19.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:19.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-01-15T18:22:20.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:20.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-01-15T18:22:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:20.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-01-15T18:22:20.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:20.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-01-15T18:22:21.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:21.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-01-15T18:22:21.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:21.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-01-15T18:22:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:21.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-01-15T18:22:22.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:22.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-01-15T18:22:22.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:22.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-01-15T18:22:23.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:23.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-01-15T18:22:23.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:23.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-01-15T18:22:24.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:24.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-01-15T18:22:24.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:24.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-01-15T18:22:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-01-15T18:22:25.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:25.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-01-15T18:22:25.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:25.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-01-15T18:22:26.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:26.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-01-15T18:22:26.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:26.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-01-15T18:22:26.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:26.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-01-15T18:22:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-01-15T18:22:27.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:27.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-01-15T18:22:27.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:27.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-01-15T18:22:28.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:28.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-01-15T18:22:29.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:29.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-01-15T18:22:30.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:30.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-01-15T18:22:30.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:30.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-01-15T18:22:30.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:30.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-01-15T18:22:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:31.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-01-15T18:22:32.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:32.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-01-15T18:22:32.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:32.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-01-15T18:22:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:32.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-01-15T18:22:32.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:32.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-01-15T18:22:33.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:33.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-01-15T18:22:33.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:33.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-01-15T18:22:33.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:33.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-01-15T18:22:34.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:34.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-01-15T18:22:35.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:35.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-01-15T18:22:35.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:35.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-01-15T18:22:35.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:35.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-01-15T18:22:36.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:36.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-01-15T18:22:36.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:36.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-01-15T18:22:36.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:36.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-01-15T18:22:37.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:37.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-01-15T18:22:37.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:37.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-01-15T18:22:38.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:38.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-01-15T18:22:38.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:38.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-01-15T18:22:38.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:38.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-01-15T18:22:39.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:39.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-01-15T18:22:40.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:40.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-01-15T18:22:41.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:41.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-01-15T18:22:42.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:42.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-01-15T18:22:43.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:43.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-01-15T18:22:43.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-01-15T18:22:43.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:43.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-01-15T18:22:44.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:44.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-01-15T18:22:44.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:44.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-01-15T18:22:45.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:45.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-01-15T18:22:45.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:45.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-01-15T18:22:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:46.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-01-15T18:22:46.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:46.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-01-15T18:22:46.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:46.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-01-15T18:22:46.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:46.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-01-15T18:22:47.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:47.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-01-15T18:22:47.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:47.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-01-15T18:22:48.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:48.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-01-15T18:22:48.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:48.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-01-15T18:22:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:49.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-01-15T18:22:49.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:49.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-01-15T18:22:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-01-15T18:22:50.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:50.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-01-15T18:22:50.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:50.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-01-15T18:22:51.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:51.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-01-15T18:22:52.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:52.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-01-15T18:22:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-01-15T18:22:53.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:53.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-01-15T18:22:53.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:53.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-01-15T18:22:53.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:53.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-01-15T18:22:54.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:54.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-01-15T18:22:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:54.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-01-15T18:22:54.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:54.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-01-15T18:22:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-01-15T18:22:56.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:56.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-01-15T18:22:56.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:56.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-01-15T18:22:56.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:56.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-01-15T18:22:57.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:57.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-01-15T18:22:57.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:57.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-01-15T18:22:57.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:57.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-01-15T18:22:57.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:57.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-01-15T18:22:58.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:58.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-01-15T18:22:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:59.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-01-15T18:22:59.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:59.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-01-15T18:22:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:22:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-01-15T18:23:00.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:00.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-01-15T18:23:00.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:00.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-01-15T18:23:01.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:01.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-01-15T18:23:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-01-15T18:23:02.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:02.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-01-15T18:23:02.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:02.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-01-15T18:23:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-01-15T18:23:03.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:03.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-01-15T18:23:03.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:03.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-01-15T18:23:04.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:04.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-01-15T18:23:05.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:05.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-01-15T18:23:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-01-15T18:23:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:05.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-01-15T18:23:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-01-15T18:23:06.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:06.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-01-15T18:23:06.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:06.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-01-15T18:23:06.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:06.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-01-15T18:23:07.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-01-15T18:23:08.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:08.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-01-15T18:23:08.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:08.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-01-15T18:23:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:08.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-01-15T18:23:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:09.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-01-15T18:23:09.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:09.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-01-15T18:23:09.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:09.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-01-15T18:23:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:10.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-01-15T18:23:11.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-01-15T18:23:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-01-15T18:23:12.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:12.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-01-15T18:23:12.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:12.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-01-15T18:23:13.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:13.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-01-15T18:23:13.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:13.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-01-15T18:23:13.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:13.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-01-15T18:23:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-01-15T18:23:14.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:14.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-01-15T18:23:15.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:15.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-01-15T18:23:15.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:15.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-01-15T18:23:16.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:16.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-01-15T18:23:16.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:16.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-01-15T18:23:16.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:16.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-01-15T18:23:16.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:16.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-01-15T18:23:17.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:17.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-01-15T18:23:17.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:17.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-01-15T18:23:17.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:17.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-01-15T18:23:18.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:18.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-01-15T18:23:19.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:19.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-01-15T18:23:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-01-15T18:23:19.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:19.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-01-15T18:23:20.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:20.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-01-15T18:23:20.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:20.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-01-15T18:23:21.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:21.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-01-15T18:23:22.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:22.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-01-15T18:23:23.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:23.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-01-15T18:23:24.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:24.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-01-15T18:23:24.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:24.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-01-15T18:23:26.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:26.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-01-15T18:23:27.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:27.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-01-15T18:23:27.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:27.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-01-15T18:23:27.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:27.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-01-15T18:23:27.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:27.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-01-15T18:23:28.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:28.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-01-15T18:23:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-01-15T18:23:29.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:29.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-01-15T18:23:29.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:29.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-01-15T18:23:30.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:30.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-01-15T18:23:30.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:30.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-01-15T18:23:30.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:30.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-01-15T18:23:30.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:30.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-01-15T18:23:31.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:31.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-01-15T18:23:32.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:32.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-01-15T18:23:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-01-15T18:23:32.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:32.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-01-15T18:23:33.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:33.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-01-15T18:23:33.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:33.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-01-15T18:23:34.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:34.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-01-15T18:23:34.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:34.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-01-15T18:23:34.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:34.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-01-15T18:23:35.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:35.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-01-15T18:23:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-01-15T18:23:36.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:36.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-01-15T18:23:36.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:36.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-01-15T18:23:37.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:37.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-01-15T18:23:37.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:37.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-01-15T18:23:37.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:37.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-01-15T18:23:38.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:38.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-01-15T18:23:38.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 1 + 1 2021-01-15T18:23:38.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=2 2021-01-15T18:23:38.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:412: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2021-01-15T18:23:38.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: seq 1 200 2021-01-15T18:23:38.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:38.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2021-01-15T18:23:39.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:39.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2021-01-15T18:23:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2021-01-15T18:23:39.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:39.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2021-01-15T18:23:40.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:40.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2021-01-15T18:23:40.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:40.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2021-01-15T18:23:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:41.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2021-01-15T18:23:41.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:41.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2021-01-15T18:23:41.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:41.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2021-01-15T18:23:42.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:42.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2021-01-15T18:23:42.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:42.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2021-01-15T18:23:42.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:42.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2021-01-15T18:23:43.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:43.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2021-01-15T18:23:44.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:44.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2021-01-15T18:23:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2021-01-15T18:23:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2021-01-15T18:23:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:44.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2021-01-15T18:23:45.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:45.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2021-01-15T18:23:45.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:45.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2021-01-15T18:23:45.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:45.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2021-01-15T18:23:46.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:46.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2021-01-15T18:23:47.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:47.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2021-01-15T18:23:47.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:47.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2021-01-15T18:23:47.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:47.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2021-01-15T18:23:48.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:48.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2021-01-15T18:23:48.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:48.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2021-01-15T18:23:49.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:49.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2021-01-15T18:23:49.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:49.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2021-01-15T18:23:49.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:49.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2021-01-15T18:23:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:50.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2021-01-15T18:23:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:51.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2021-01-15T18:23:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2021-01-15T18:23:51.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:51.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2021-01-15T18:23:52.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:52.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2021-01-15T18:23:52.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2021-01-15T18:23:54.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:54.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2021-01-15T18:23:55.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:55.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2021-01-15T18:23:55.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:55.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2021-01-15T18:23:55.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:55.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2021-01-15T18:23:56.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:56.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2021-01-15T18:23:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:57.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2021-01-15T18:23:57.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:57.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2021-01-15T18:23:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2021-01-15T18:23:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2021-01-15T18:23:58.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:58.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2021-01-15T18:23:59.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:59.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2021-01-15T18:23:59.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:59.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2021-01-15T18:23:59.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:23:59.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2021-01-15T18:24:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:00.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2021-01-15T18:24:00.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:00.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2021-01-15T18:24:01.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:01.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2021-01-15T18:24:01.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2021-01-15T18:24:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2021-01-15T18:24:02.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:02.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2021-01-15T18:24:02.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:02.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2021-01-15T18:24:03.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:03.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2021-01-15T18:24:03.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:03.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2021-01-15T18:24:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2021-01-15T18:24:04.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:04.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2021-01-15T18:24:04.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:04.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2021-01-15T18:24:05.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:05.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2021-01-15T18:24:05.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:05.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2021-01-15T18:24:05.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:05.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2021-01-15T18:24:06.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:06.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2021-01-15T18:24:06.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:06.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2021-01-15T18:24:07.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:07.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2021-01-15T18:24:07.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:07.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2021-01-15T18:24:08.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:08.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2021-01-15T18:24:08.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:08.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2021-01-15T18:24:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2021-01-15T18:24:09.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:09.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2021-01-15T18:24:09.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:09.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2021-01-15T18:24:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:09.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2021-01-15T18:24:10.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:10.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2021-01-15T18:24:11.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:11.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2021-01-15T18:24:11.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:11.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2021-01-15T18:24:11.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:11.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2021-01-15T18:24:11.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:11.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2021-01-15T18:24:12.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:12.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2021-01-15T18:24:12.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:12.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2021-01-15T18:24:12.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:12.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2021-01-15T18:24:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2021-01-15T18:24:14.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:14.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2021-01-15T18:24:14.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:14.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2021-01-15T18:24:14.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:14.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2021-01-15T18:24:15.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:15.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2021-01-15T18:24:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:15.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2021-01-15T18:24:15.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:15.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2021-01-15T18:24:16.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:16.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2021-01-15T18:24:17.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:17.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2021-01-15T18:24:18.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:18.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2021-01-15T18:24:18.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:18.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2021-01-15T18:24:20.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:20.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2021-01-15T18:24:21.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:21.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2021-01-15T18:24:21.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:21.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2021-01-15T18:24:21.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:21.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2021-01-15T18:24:22.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:22.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2021-01-15T18:24:22.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:22.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2021-01-15T18:24:23.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:23.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2021-01-15T18:24:23.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:23.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2021-01-15T18:24:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2021-01-15T18:24:24.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:24.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2021-01-15T18:24:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:24.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2021-01-15T18:24:24.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:24.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2021-01-15T18:24:25.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:25.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2021-01-15T18:24:25.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:25.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2021-01-15T18:24:26.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:26.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2021-01-15T18:24:26.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:26.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2021-01-15T18:24:27.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:27.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2021-01-15T18:24:27.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:27.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2021-01-15T18:24:27.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:27.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2021-01-15T18:24:28.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:28.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2021-01-15T18:24:28.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:28.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2021-01-15T18:24:30.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:30.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2021-01-15T18:24:30.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:30.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2021-01-15T18:24:30.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:30.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2021-01-15T18:24:30.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:30.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2021-01-15T18:24:31.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:31.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2021-01-15T18:24:31.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:31.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2021-01-15T18:24:31.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:31.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2021-01-15T18:24:32.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:32.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2021-01-15T18:24:33.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:33.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2021-01-15T18:24:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2021-01-15T18:24:33.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:33.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2021-01-15T18:24:34.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:34.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2021-01-15T18:24:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:34.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2021-01-15T18:24:34.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:34.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2021-01-15T18:24:35.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:35.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2021-01-15T18:24:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2021-01-15T18:24:36.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:36.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2021-01-15T18:24:36.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:36.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2021-01-15T18:24:36.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:36.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2021-01-15T18:24:37.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:37.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2021-01-15T18:24:37.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:37.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2021-01-15T18:24:37.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:37.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2021-01-15T18:24:38.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:38.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2021-01-15T18:24:38.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:38.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2021-01-15T18:24:40.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:40.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2021-01-15T18:24:40.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:40.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2021-01-15T18:24:40.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:40.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2021-01-15T18:24:41.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:41.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2021-01-15T18:24:41.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:41.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2021-01-15T18:24:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:41.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2021-01-15T18:24:42.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:42.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2021-01-15T18:24:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2021-01-15T18:24:43.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:43.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2021-01-15T18:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:43.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2021-01-15T18:24:44.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:44.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2021-01-15T18:24:44.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:44.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2021-01-15T18:24:44.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:44.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2021-01-15T18:24:45.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:45.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2021-01-15T18:24:45.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:45.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2021-01-15T18:24:45.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:45.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2021-01-15T18:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:46.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2021-01-15T18:24:46.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:46.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2021-01-15T18:24:47.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:47.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2021-01-15T18:24:47.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:47.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2021-01-15T18:24:47.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:47.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2021-01-15T18:24:48.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:48.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2021-01-15T18:24:48.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:48.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2021-01-15T18:24:50.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:50.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2021-01-15T18:24:50.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:50.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2021-01-15T18:24:51.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:51.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2021-01-15T18:24:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:52.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2021-01-15T18:24:53.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:53.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2021-01-15T18:24:54.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:54.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2021-01-15T18:24:55.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:55.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2021-01-15T18:24:55.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:55.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2021-01-15T18:24:55.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:55.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2021-01-15T18:24:56.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:56.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2021-01-15T18:24:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2021-01-15T18:24:56.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:56.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2021-01-15T18:24:57.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:57.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2021-01-15T18:24:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2021-01-15T18:24:57.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:57.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2021-01-15T18:24:58.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:58.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2021-01-15T18:24:58.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:58.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2021-01-15T18:24:59.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:59.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2021-01-15T18:24:59.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:24:59.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2021-01-15T18:25:00.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:00.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2021-01-15T18:25:00.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:00.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2021-01-15T18:25:00.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:00.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2021-01-15T18:25:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2021-01-15T18:25:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:01.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2021-01-15T18:25:02.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:02.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2021-01-15T18:25:03.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:03.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2021-01-15T18:25:03.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:03.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2021-01-15T18:25:04.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:04.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2021-01-15T18:25:04.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:04.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2021-01-15T18:25:04.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:04.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2021-01-15T18:25:05.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:05.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2021-01-15T18:25:05.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:05.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2021-01-15T18:25:05.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:05.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2021-01-15T18:25:06.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:06.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2021-01-15T18:25:07.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2021-01-15T18:25:07.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:416: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2021-01-15T18:25:07.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: expr 2 + 1 2021-01-15T18:25:07.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:418: TEST_recovery_pool_priority: p=3 2021-01-15T18:25:07.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2021-01-15T18:25:07.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2021-01-15T18:25:07.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2021-01-15T18:25:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2021-01-15T18:25:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2021-01-15T18:25:07.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2021-01-15T18:25:07.485 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T18:25:07.485 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:25:07.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:25:07.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T18:25:07.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:25:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: local otherosd=null 2021-01-15T18:25:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:423: TEST_recovery_pool_priority: ceph pg dump pgs 2021-01-15T18:25:08.691 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:25:08.691 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:22:04.666767+0000 34'600 34:638 [1] 1 [1] 1 0'0 2021-01-15T18:21:45.793733+0000 0'0 2021-01-15T18:21:45.793733+0000 0 2021-01-15T18:25:08.691 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 190 0 0 0 0 1986002944 0 0 568 568 active+clean 2021-01-15T18:22:06.114210+0000 34'568 34:601 [0] 0 [0] 0 0'0 2021-01-15T18:21:48.017069+0000 0'0 2021-01-15T18:21:48.017069+0000 0 2021-01-15T18:25:08.692 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:25:08.692 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:25:08.693 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:25:08.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: ERRORS=0 2021-01-15T18:25:08.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2021-01-15T18:25:09.685 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:25:09.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2021-01-15T18:25:10.505 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T18:25:10.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:428: TEST_recovery_pool_priority: sleep 10 2021-01-15T18:25:20.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: get_asok_path osd.1 2021-01-15T18:25:20.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:25:20.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:25:20.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:25:20.521 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:25:20.521 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:25:20.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:25:20.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: CEPH_ARGS= 2021-01-15T18:25:20.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:25:20.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: echo osd.1 2021-01-15T18:25:20.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:431: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-01-15T18:25:20.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: get_asok_path osd.0 2021-01-15T18:25:20.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T18:25:20.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T18:25:20.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:25:20.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:25:20.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:25:20.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.0.asok 2021-01-15T18:25:20.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: CEPH_ARGS= 2021-01-15T18:25:20.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:432: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.0.asok dump_recovery_reservations 2021-01-15T18:25:20.653 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 2021-01-15T18:25:20.654 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:25:20.654 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:25:20.654 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:25:20.654 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:25:20.654 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:25:20.655 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:25:20.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:25:20.655 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:25:20.655 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 191, 2021-01-15T18:25:20.656 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:25:20.656 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.656 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:25:20.656 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:25:20.656 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:25:20.657 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:25:20.657 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:25:20.657 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:25:20.657 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:25:20.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:25:20.658 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2021-01-15T18:25:20.658 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 192, 2021-01-15T18:25:20.658 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:25:20.658 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.659 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:25:20.659 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.659 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:25:20.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: echo osd.0 2021-01-15T18:25:20.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:434: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-01-15T18:25:20.755 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 2021-01-15T18:25:20.755 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:25:20.756 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:25:20.756 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:25:20.757 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:25:20.757 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:25:20.758 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:25:20.758 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:25:20.759 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "2.0", 2021-01-15T18:25:20.759 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 192, 2021-01-15T18:25:20.759 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:25:20.760 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.760 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:25:20.760 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:25:20.761 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:25:20.761 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:25:20.762 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:25:20.762 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:25:20.762 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:25:20.763 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:25:20.763 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:25:20.764 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 191, 2021-01-15T18:25:20.764 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:25:20.764 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.765 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:25:20.765 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:25:20.766 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:25:20.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-01-15T18:25:20.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T18:25:20.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-01-15T18:25:20.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: ITEM=1.0 2021-01-15T18:25:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-01-15T18:25:20.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-01-15T18:25:20.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T18:25:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: PRIO=191 2021-01-15T18:25:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:444: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-01-15T18:25:20.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-01-15T18:25:20.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-01-15T18:25:20.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2021-01-15T18:25:20.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:452: TEST_recovery_pool_priority: ITEM=1.0 2021-01-15T18:25:20.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-01-15T18:25:20.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-01-15T18:25:20.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-01-15T18:25:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: PRIO=191 2021-01-15T18:25:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2021-01-15T18:25:20.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-01-15T18:25:20.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T18:25:20.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-01-15T18:25:20.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:467: TEST_recovery_pool_priority: ITEM=2.0 2021-01-15T18:25:20.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:468: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-01-15T18:25:20.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2021-01-15T18:25:20.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T18:25:20.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:473: TEST_recovery_pool_priority: PRIO=192 2021-01-15T18:25:20.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:474: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-01-15T18:25:20.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-01-15T18:25:20.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-01-15T18:25:20.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2021-01-15T18:25:20.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:482: TEST_recovery_pool_priority: ITEM=2.0 2021-01-15T18:25:20.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:483: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-01-15T18:25:20.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2021-01-15T18:25:20.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-01-15T18:25:20.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:488: TEST_recovery_pool_priority: PRIO=192 2021-01-15T18:25:20.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:489: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2021-01-15T18:25:20.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:496: TEST_recovery_pool_priority: wait_for_clean 2021-01-15T18:25:20.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:25:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:25:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:25:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:25:20.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:25:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:25:20.792 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:25:20.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:25:20.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:25:20.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:25:20.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:25:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:25:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:25:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:25:20.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:25:21.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:25:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T18:25:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:25:21.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:25:21.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:25:21.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836531 2021-01-15T18:25:21.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836531 2021-01-15T18:25:21.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836531' 2021-01-15T18:25:21.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:25:21.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:25:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673010 2021-01-15T18:25:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673010 2021-01-15T18:25:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836531 1-42949673010' 2021-01-15T18:25:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:25:21.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836531 2021-01-15T18:25:21.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:25:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:25:21.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836531 2021-01-15T18:25:21.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:25:21.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836531 2021-01-15T18:25:21.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2021-01-15T18:25:21.805 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836531 2021-01-15T18:25:21.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:25:22.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836529 -lt 21474836531 2021-01-15T18:25:22.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:25:23.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:25:23.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:25:24.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836532 -lt 21474836531 2021-01-15T18:25:24.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:25:24.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673010 2021-01-15T18:25:24.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:25:24.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:25:24.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673010 2021-01-15T18:25:24.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:25:24.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673010 2021-01-15T18:25:24.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2021-01-15T18:25:24.129 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673010 2021-01-15T18:25:24.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:25:24.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673010 -lt 42949673010 2021-01-15T18:25:24.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:25:24.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:24.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:25.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T18:25:25.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:25.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:25.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:25.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:25.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:25.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:25.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:25.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:25.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:25.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:25.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:26.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:26.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:25:26.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:25:26.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:25:26.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:26.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:26.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:26.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:26.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:26.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:26.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:26.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:26.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:26.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:26.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:26.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:26.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:26.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:27.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:27.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:27.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:27.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:27.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:27.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:27.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:27.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:27.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:28.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26208969 2021-01-15T18:25:28.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26208969 '!=' null 2021-01-15T18:25:28.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:28.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:28.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:28.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:28.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:28.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:28.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:28.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:28.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:28.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:28.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:28.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:28.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:28.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:28.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:29.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:29.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:29.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:29.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:29.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:29.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:29.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:29.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:29.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:30.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335577 2021-01-15T18:25:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335577 '!=' null 2021-01-15T18:25:30.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:30.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:30.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:30.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:30.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:30.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:30.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:30.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:30.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:30.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:30.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:30.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:30.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:30.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:30.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:31.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:31.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:31.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:31.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:31.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:31.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:31.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:32.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335891 2021-01-15T18:25:32.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335891 '!=' null 2021-01-15T18:25:32.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:32.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:32.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:32.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:32.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:32.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:32.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:32.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:32.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:32.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:32.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:32.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:32.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:32.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:32.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:33.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:33.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:33.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:33.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:34.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209580 2021-01-15T18:25:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209580 '!=' null 2021-01-15T18:25:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:34.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:34.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:34.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:34.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:34.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:34.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:34.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:34.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:34.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:34.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:34.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:34.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:35.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:35.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:35.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:35.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:35.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:35.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209580 2021-01-15T18:25:35.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209580 '!=' null 2021-01-15T18:25:35.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:35.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:35.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:35.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:35.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:35.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:35.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:35.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:35.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:35.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:35.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:36.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:36.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:36.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:36.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:36.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:36.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:36.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:36.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:37.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094255 2021-01-15T18:25:37.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094255 '!=' null 2021-01-15T18:25:37.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:37.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:37.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:37.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:37.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:37.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:37.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:37.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:37.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:37.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:37.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:37.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:37.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:37.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:38.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:38.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:38.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:38.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:38.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:38.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:38.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:38.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:38.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:39.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462668 2021-01-15T18:25:39.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462668 '!=' null 2021-01-15T18:25:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:39.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:39.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:39.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:39.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:39.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:39.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:39.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:39.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:39.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:39.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:39.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:39.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:40.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:40.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:40.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:40.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:40.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:40.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:41.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209982 2021-01-15T18:25:41.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209982 '!=' null 2021-01-15T18:25:41.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:41.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:41.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:41.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:41.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:41.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:41.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:41.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:41.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:41.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:41.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:41.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:41.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:41.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:41.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:42.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:42.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:42.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:42.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:42.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:42.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:42.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:42.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:42.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:42.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336237 2021-01-15T18:25:42.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336237 '!=' null 2021-01-15T18:25:42.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:42.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:42.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:42.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:42.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:42.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:42.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:42.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:42.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:42.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:42.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:43.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:43.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:43.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:43.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:44.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:44.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:44.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:44.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462568 2021-01-15T18:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462568 '!=' null 2021-01-15T18:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:44.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:44.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:44.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:44.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:44.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:44.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:44.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:44.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:45.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:45.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:45.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:45.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:45.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:45.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:45.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:45.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:45.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:45.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:46.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220601 2021-01-15T18:25:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220601 '!=' null 2021-01-15T18:25:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:46.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:46.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:46.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:46.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:46.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:46.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:46.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:47.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:47.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:47.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:47.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:47.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:47.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:47.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:47.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:47.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:48.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336157 2021-01-15T18:25:48.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336157 '!=' null 2021-01-15T18:25:48.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:48.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:48.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:48.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:48.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:48.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:48.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:48.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:48.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:48.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:48.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:48.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:48.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:48.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:48.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:49.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:49.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:49.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:49.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:49.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:49.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:49.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:49.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:49.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336157 2021-01-15T18:25:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336157 '!=' null 2021-01-15T18:25:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:49.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:49.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:49.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:49.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:49.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:49.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:49.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:49.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:49.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:49.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:50.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:50.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:50.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:50.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:50.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:50.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:50.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:50.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:50.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:50.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:50.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:50.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:50.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:51.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209824 2021-01-15T18:25:51.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209824 '!=' null 2021-01-15T18:25:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:51.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:51.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:51.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:51.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:51.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:51.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:51.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:51.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:51.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:51.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:51.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:51.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:52.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:52.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:52.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:52.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:52.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:53.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209844 2021-01-15T18:25:53.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209844 '!=' null 2021-01-15T18:25:53.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:53.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:53.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:53.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:53.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:53.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:53.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:53.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:53.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:53.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:53.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:53.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:53.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:54.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:54.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:54.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:55.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083525 2021-01-15T18:25:55.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083525 '!=' null 2021-01-15T18:25:55.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:55.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:55.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:55.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:55.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:55.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:55.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:55.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:55.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:55.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:55.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:56.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:56.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:56.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:56.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:56.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:56.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:56.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:56.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841480 2021-01-15T18:25:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841480 '!=' null 2021-01-15T18:25:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:57.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:57.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:57.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:57.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:57.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:57.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:57.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:57.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:57.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:57.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:57.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:25:58.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:25:58.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:25:58.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:25:58.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:25:58.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:25:58.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:25:58.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:25:58.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:25:58.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:25:59.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083347 2021-01-15T18:25:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083347 '!=' null 2021-01-15T18:25:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:25:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:25:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:25:59.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:25:59.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:25:59.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:25:59.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:25:59.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:25:59.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:25:59.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:25:59.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:25:59.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:25:59.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:25:59.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:25:59.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:00.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:00.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:00.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:00.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:00.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:00.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:01.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083451 2021-01-15T18:26:01.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083451 '!=' null 2021-01-15T18:26:01.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:01.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:01.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:01.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:01.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:01.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:01.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:01.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:01.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:01.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:01.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:01.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:01.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:01.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:01.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:02.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:02.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:02.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:02.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:02.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:02.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:02.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:02.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:02.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:03.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083352 2021-01-15T18:26:03.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083352 '!=' null 2021-01-15T18:26:03.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:03.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:03.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:03.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:03.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:03.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:03.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:03.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:04.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:04.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:04.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:04.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:04.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:04.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:04.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:04.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:04.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:04.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:05.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335976 2021-01-15T18:26:05.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335976 '!=' null 2021-01-15T18:26:05.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:05.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:05.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:05.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:06.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:06.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:06.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:06.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:06.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:06.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:07.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093894 2021-01-15T18:26:07.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093894 '!=' null 2021-01-15T18:26:07.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:07.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:07.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:07.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:07.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:07.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:07.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:07.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:07.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:07.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:07.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:08.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:08.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:08.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:08.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:08.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:08.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:08.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:08.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:08.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:08.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:08.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:08.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:09.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841240 2021-01-15T18:26:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841240 '!=' null 2021-01-15T18:26:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:09.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:09.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:09.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:09.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:09.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:09.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:09.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:09.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:09.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:10.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:10.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:10.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:10.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:11.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:11.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:11.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:11.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:11.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:11.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:11.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346613 2021-01-15T18:26:11.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346613 '!=' null 2021-01-15T18:26:11.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:11.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:11.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:11.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:11.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:11.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:11.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:11.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:11.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:11.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:11.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:12.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:12.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:12.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:12.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:12.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:12.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:12.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:12.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:12.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:13.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599264 2021-01-15T18:26:13.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599264 '!=' null 2021-01-15T18:26:13.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:13.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:13.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:13.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:13.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:13.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:13.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:13.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:13.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:13.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:13.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:14.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:14.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:14.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:14.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:14.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:14.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:14.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:14.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:14.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:15.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17472920 2021-01-15T18:26:15.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17472920 '!=' null 2021-01-15T18:26:15.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:15.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:15.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:15.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:15.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:15.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:15.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:15.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:15.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:15.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:15.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:16.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:16.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:16.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:16.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:16.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:16.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:16.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:16.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:16.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:17.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978186 2021-01-15T18:26:17.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978186 '!=' null 2021-01-15T18:26:17.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:17.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:17.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:17.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:17.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:17.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:17.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:17.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:17.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:17.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:17.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:17.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:17.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:17.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:17.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:18.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:18.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:18.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:18.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:18.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:18.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:18.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:18.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:18.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220095 2021-01-15T18:26:18.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220095 '!=' null 2021-01-15T18:26:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:18.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:19.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:19.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:19.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:19.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:19.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:19.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:19.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:19.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:19.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:19.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:19.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:19.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:20.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:20.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:20.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:20.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:20.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:20.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714755 2021-01-15T18:26:20.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714755 '!=' null 2021-01-15T18:26:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:21.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:21.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:21.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:21.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:21.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:21.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:21.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:21.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:21.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:21.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:21.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:21.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:21.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:21.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:21.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:21.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:21.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:21.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:21.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:22.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209214 2021-01-15T18:26:23.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209214 '!=' null 2021-01-15T18:26:23.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:23.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:23.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:23.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:23.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:23.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:23.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:23.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:23.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:23.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:23.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:24.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:24.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:24.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:24.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:24.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:24.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:24.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:24.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:24.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:24.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082679 2021-01-15T18:26:24.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082679 '!=' null 2021-01-15T18:26:24.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:24.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:24.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:25.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:25.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:25.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:25.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:25.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:25.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:25.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:25.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:25.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:25.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:25.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:25.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:26.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:26.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:26.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:26.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:26.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:26.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:26.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967305 2021-01-15T18:26:26.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967305 '!=' null 2021-01-15T18:26:26.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:26.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:26.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:26.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:26.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:26.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:26.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:27.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:27.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:27.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:27.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:27.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:27.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:27.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:27.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:27.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:27.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:27.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:27.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:27.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:28.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335750 2021-01-15T18:26:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335750 '!=' null 2021-01-15T18:26:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:28.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:28.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:28.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:28.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:28.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:28.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:28.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:28.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:28.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:28.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:28.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:28.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:29.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:29.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:29.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:29.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:30.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462236 2021-01-15T18:26:30.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462236 '!=' null 2021-01-15T18:26:30.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:30.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:30.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:30.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:30.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:30.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:30.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:30.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:30.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:30.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:30.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:30.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:30.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:30.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:30.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:31.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:31.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:31.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:31.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:31.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:31.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:31.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:31.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:31.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:32.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462236 2021-01-15T18:26:32.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462236 '!=' null 2021-01-15T18:26:32.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:32.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:32.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:32.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:32.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:32.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:32.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:32.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:32.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:32.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:32.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:32.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:32.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:33.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:33.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:33.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:33.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:33.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462290 2021-01-15T18:26:33.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462290 '!=' null 2021-01-15T18:26:33.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:33.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:33.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:33.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:33.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:33.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:33.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:33.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:33.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:33.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:33.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:34.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:34.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:34.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:34.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:35.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:35.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:35.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:35.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:35.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:35.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:35.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:35.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:35.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462398 2021-01-15T18:26:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462398 '!=' null 2021-01-15T18:26:35.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:35.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:35.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:35.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:35.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:35.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:35.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:35.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:35.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:35.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:35.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:36.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:36.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:36.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:36.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:37.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:37.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:37.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:37.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:37.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:37.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220607 2021-01-15T18:26:37.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220607 '!=' null 2021-01-15T18:26:37.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:37.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:37.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:37.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:37.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:37.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:37.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:37.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:37.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:37.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:37.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:38.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:38.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:38.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:38.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:39.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:39.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:39.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462524 2021-01-15T18:26:39.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462524 '!=' null 2021-01-15T18:26:39.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:39.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:39.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:39.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:39.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:39.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:39.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:39.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:39.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:39.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:40.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:40.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:40.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:40.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:40.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:40.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:40.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:40.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:40.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:40.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:40.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:40.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:41.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336171 2021-01-15T18:26:41.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336171 '!=' null 2021-01-15T18:26:41.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:41.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:41.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:41.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:41.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:41.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:41.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:41.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:41.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:41.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:41.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:42.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:42.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:42.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:42.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:42.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:42.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:42.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:42.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:42.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:42.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:42.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:42.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:42.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:43.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083022 2021-01-15T18:26:43.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083022 '!=' null 2021-01-15T18:26:43.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:43.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:43.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:43.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:43.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:43.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:43.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:43.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:43.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:43.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:43.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:43.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:43.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:44.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:44.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:44.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:44.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:44.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:44.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:44.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:44.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:44.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:45.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082920 2021-01-15T18:26:45.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082920 '!=' null 2021-01-15T18:26:45.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:45.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:45.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:45.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:45.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:45.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:45.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:45.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:45.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:45.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:45.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:45.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:45.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:46.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:46.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:46.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:46.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:46.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:47.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841173 2021-01-15T18:26:47.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841173 '!=' null 2021-01-15T18:26:47.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:47.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:47.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:47.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:47.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:47.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:47.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:47.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:47.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:47.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:47.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:48.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:48.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:48.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:48.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:49.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:49.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:49.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:49.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:49.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335899 2021-01-15T18:26:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335899 '!=' null 2021-01-15T18:26:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:50.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:50.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:50.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:50.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:50.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:50.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:50.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:50.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:50.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:51.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:51.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:51.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335899 2021-01-15T18:26:51.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335899 '!=' null 2021-01-15T18:26:51.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:51.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:51.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:51.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:51.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:51.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:51.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:51.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:51.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:51.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:52.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:52.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:52.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:52.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:52.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:52.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:52.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:52.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:52.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:52.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:52.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:53.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462306 2021-01-15T18:26:53.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462306 '!=' null 2021-01-15T18:26:53.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:53.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:53.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:53.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:53.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:53.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:53.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:53.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:53.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:53.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:53.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:54.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:54.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:54.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:54.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:54.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:54.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:54.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:54.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:54.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:54.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:54.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:55.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23589037 2021-01-15T18:26:55.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23589037 '!=' null 2021-01-15T18:26:55.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:55.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:55.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:55.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:55.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:55.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:55.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:55.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:55.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:55.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:55.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:55.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:55.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:55.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:55.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:56.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:56.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:56.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:56.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:56.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:56.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:57.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18347023 2021-01-15T18:26:57.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18347023 '!=' null 2021-01-15T18:26:57.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:57.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:57.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:57.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:57.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:57.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:57.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:57.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:57.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:57.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:57.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:58.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:58.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:58.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:58.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:26:58.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:26:58.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:26:58.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:26:58.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:26:58.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:26:58.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:26:58.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:26:58.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:26:58.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:26:59.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462663 2021-01-15T18:26:59.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462663 '!=' null 2021-01-15T18:26:59.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:26:59.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:26:59.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:26:59.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:26:59.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:26:59.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:26:59.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:26:59.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:26:59.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:26:59.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:26:59.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:26:59.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:26:59.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:26:59.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:26:59.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:00.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:00.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:00.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:00.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:00.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335921 2021-01-15T18:27:00.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335921 '!=' null 2021-01-15T18:27:00.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:00.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:00.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:00.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:00.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:01.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:01.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:01.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:01.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:01.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:01.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:01.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:01.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:01.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:01.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:01.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:01.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:01.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:01.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:01.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:01.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:01.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:01.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:01.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:02.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209533 2021-01-15T18:27:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209533 '!=' null 2021-01-15T18:27:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:02.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:02.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:02.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:02.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:02.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:02.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:02.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:02.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:03.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:03.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:03.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:03.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:03.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:03.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:03.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:03.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:03.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:03.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:03.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:03.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:04.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083196 2021-01-15T18:27:04.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083196 '!=' null 2021-01-15T18:27:04.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:04.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:04.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:04.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:04.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:04.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:04.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:04.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:04.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:04.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:04.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:05.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:05.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:05.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:05.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:05.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:05.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:05.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:05.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:05.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:05.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:05.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:05.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:05.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:06.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083196 2021-01-15T18:27:06.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083196 '!=' null 2021-01-15T18:27:06.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:06.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:06.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:06.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:06.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:06.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:06.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:06.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:06.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:07.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:07.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:07.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:07.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:07.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:07.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:07.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:07.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:07.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:08.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209385 2021-01-15T18:27:08.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209385 '!=' null 2021-01-15T18:27:08.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:08.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:08.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:08.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:08.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:08.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:08.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:08.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:08.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:08.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:08.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:08.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:08.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:08.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:08.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:09.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:09.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:09.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:09.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:09.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:10.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461954 2021-01-15T18:27:10.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461954 '!=' null 2021-01-15T18:27:10.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:10.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:10.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:10.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:10.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:10.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:10.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:10.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:10.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:10.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:10.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:10.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:10.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:10.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:10.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:11.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:11.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:12.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461954 2021-01-15T18:27:12.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461954 '!=' null 2021-01-15T18:27:12.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:12.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:12.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:12.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:12.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:12.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:12.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:12.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:12.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:12.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:12.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:12.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:12.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:13.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:13.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:13.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:13.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:13.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:13.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:13.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:13.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:13.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462479 2021-01-15T18:27:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462479 '!=' null 2021-01-15T18:27:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:13.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:13.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:14.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:14.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:14.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:14.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:14.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:14.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:14.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:14.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:14.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:14.429 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:14.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:15.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:15.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:15.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:15.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:15.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:15.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:15.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462479 2021-01-15T18:27:15.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462479 '!=' null 2021-01-15T18:27:15.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:15.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:15.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:15.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:15.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:15.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:15.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:15.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:15.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:15.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:15.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:16.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:16.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:16.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:16.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:16.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:16.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:16.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:16.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:16.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:16.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:17.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220616 2021-01-15T18:27:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220616 '!=' null 2021-01-15T18:27:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:17.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:17.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:17.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:17.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:17.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:17.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:17.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:17.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:17.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:17.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:17.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:17.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:17.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:18.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:18.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:18.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:18.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:19.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462706 2021-01-15T18:27:19.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462706 '!=' null 2021-01-15T18:27:19.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:19.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:19.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:19.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:19.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:19.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:19.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:19.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:19.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:19.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:19.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:19.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:19.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:19.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:19.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:20.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:20.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:20.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:20.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:20.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:20.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:20.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:20.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:20.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336405 2021-01-15T18:27:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336405 '!=' null 2021-01-15T18:27:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:21.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:21.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:21.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:21.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:21.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:21.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:21.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:21.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:21.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:21.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:21.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:21.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:22.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:22.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:22.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:22.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:22.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:22.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:22.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:22.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:22.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26210260 2021-01-15T18:27:23.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26210260 '!=' null 2021-01-15T18:27:23.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:23.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:23.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:23.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:23.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:23.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:23.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:23.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:23.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:23.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:23.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:24.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:24.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:24.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:24.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:24.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:24.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:24.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:25.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083831 2021-01-15T18:27:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083831 '!=' null 2021-01-15T18:27:25.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:25.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:25.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:25.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:25.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:25.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:26.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:26.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:26.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:26.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:26.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:26.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:26.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:26.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:27.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841736 2021-01-15T18:27:27.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841736 '!=' null 2021-01-15T18:27:27.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:27.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:27.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:27.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:27.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:27.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:27.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:27.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:27.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:27.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:27.425 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:27.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:27.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:27.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:27.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:28.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:28.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:28.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:28.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:28.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:28.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:28.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:28.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:28.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:29.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27083683 2021-01-15T18:27:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27083683 '!=' null 2021-01-15T18:27:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:29.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:29.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:29.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:29.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:29.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:29.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:29.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:29.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:29.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:29.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:29.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:29.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:29.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:30.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:30.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:30.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:30.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:30.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:31.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25336322 2021-01-15T18:27:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25336322 '!=' null 2021-01-15T18:27:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:31.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:31.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:31.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:31.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:31.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:31.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:31.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:31.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:31.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:31.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:31.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:31.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:31.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:32.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:32.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:32.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:32.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:32.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:33.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841592 2021-01-15T18:27:33.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841592 '!=' null 2021-01-15T18:27:33.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:33.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:33.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:33.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:33.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:33.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:33.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:33.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:33.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:33.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:33.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:33.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:33.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:33.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:33.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:34.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:34.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:34.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:34.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:34.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:34.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:34.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:34.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:34.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:34.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346947 2021-01-15T18:27:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346947 '!=' null 2021-01-15T18:27:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:34.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:34.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:34.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:34.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:34.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:34.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:34.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:34.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:34.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:35.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:35.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:35.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:35.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:35.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:35.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:35.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:35.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:35.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12231325 2021-01-15T18:27:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12231325 '!=' null 2021-01-15T18:27:36.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:36.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:36.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:36.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:36.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:36.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:36.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:36.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:36.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:36.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:36.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:36.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:36.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:36.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:36.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:37.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:37.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:37.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:37.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:37.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599715 2021-01-15T18:27:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599715 '!=' null 2021-01-15T18:27:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:38.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:38.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:38.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:38.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:38.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:38.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:38.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:38.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:38.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:38.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:38.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:38.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:39.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:39.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:39.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:39.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:39.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:39.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:39.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:39.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:39.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:39.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599715 2021-01-15T18:27:39.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599715 '!=' null 2021-01-15T18:27:39.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:39.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:39.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:39.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:39.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:39.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:39.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:39.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:39.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:39.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:39.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:40.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:40.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:40.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:40.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:40.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:40.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:40.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:40.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:40.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:40.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:40.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:40.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:40.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:41.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17473361 2021-01-15T18:27:41.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17473361 '!=' null 2021-01-15T18:27:41.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:41.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:41.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:41.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:41.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:41.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:41.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:41.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:41.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:41.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:41.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:42.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:42.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:42.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:42.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:42.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:42.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:42.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:42.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:42.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:42.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:42.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:42.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:42.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:43.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094102 2021-01-15T18:27:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094102 '!=' null 2021-01-15T18:27:43.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:43.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:43.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:43.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:43.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:43.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:43.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:43.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:43.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:43.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:43.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:43.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:43.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:43.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:44.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:44.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:44.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:44.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:44.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588802 2021-01-15T18:27:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588802 '!=' null 2021-01-15T18:27:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:44.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:45.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:45.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:45.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:45.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:45.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:45.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:45.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:45.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:45.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:45.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:45.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:46.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:46.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:46.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:46.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:46.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:46.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:46.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:46.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:46.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:46.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841443 2021-01-15T18:27:46.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841443 '!=' null 2021-01-15T18:27:46.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:46.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:46.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:46.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:46.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:46.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:46.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:47.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:47.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:47.277 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:47.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:48.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:48.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:48.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:48.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:48.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:48.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:48.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:48.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:49.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209619 2021-01-15T18:27:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209619 '!=' null 2021-01-15T18:27:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:49.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:49.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:49.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:49.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:49.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:49.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:49.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:49.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:49.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:49.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:49.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:49.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:49.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:50.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:50.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:50.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:50.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:50.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:50.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:50.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:50.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:50.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:50.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082990 2021-01-15T18:27:50.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082990 '!=' null 2021-01-15T18:27:50.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:50.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:50.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:50.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:50.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:50.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:50.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:50.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:50.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:50.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:50.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:51.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:51.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:51.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335510 2021-01-15T18:27:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335510 '!=' null 2021-01-15T18:27:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:52.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:52.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:52.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:52.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:52.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:52.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:52.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:53.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:53.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:53.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:53.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:53.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:53.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:53.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:53.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:53.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:53.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:54.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588322 2021-01-15T18:27:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588322 '!=' null 2021-01-15T18:27:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:54.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:54.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:54.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:54.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:54.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:54.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:54.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:54.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:54.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:54.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:54.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:54.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:54.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:55.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:55.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:55.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:55.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:55.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:55.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:55.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:55.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:55.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:56.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346394 2021-01-15T18:27:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346394 '!=' null 2021-01-15T18:27:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:56.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:56.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:56.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:56.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:56.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:56.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:56.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:56.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:56.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:56.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:56.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:56.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:56.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:56.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:57.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:57.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:57.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:57.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:57.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:57.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:57.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:58.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461873 2021-01-15T18:27:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461873 '!=' null 2021-01-15T18:27:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:58.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:58.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:58.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:58.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:27:58.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:27:58.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:27:58.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:27:58.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:27:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:27:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:27:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:27:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:27:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:27:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:27:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:27:59.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:27:59.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:27:59.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461873 2021-01-15T18:27:59.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461873 '!=' null 2021-01-15T18:27:59.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:27:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:27:59.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:27:59.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:27:59.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:27:59.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:27:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:27:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:27:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:27:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:27:59.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:00.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:00.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:00.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:00.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:00.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:00.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:00.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:00.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:00.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:00.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:00.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:00.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:00.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:01.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335420 2021-01-15T18:28:01.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335420 '!=' null 2021-01-15T18:28:01.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:01.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:01.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:01.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:01.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:01.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:01.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:01.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:01.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:01.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:02.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:02.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:02.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:02.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:02.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:02.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:02.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:02.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:02.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:02.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:02.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:02.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:02.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:03.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335151 2021-01-15T18:28:03.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335151 '!=' null 2021-01-15T18:28:03.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:03.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:03.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:03.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:03.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:03.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:03.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:03.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:03.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:03.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:04.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:04.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:04.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:04.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:04.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:04.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:04.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:04.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26208965 2021-01-15T18:28:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26208965 '!=' null 2021-01-15T18:28:04.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:04.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:04.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:04.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:04.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:04.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:04.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:04.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:04.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:04.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:04.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:05.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:05.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:05.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:05.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:05.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:05.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:05.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:05.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:05.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:05.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841049 2021-01-15T18:28:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841049 '!=' null 2021-01-15T18:28:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:06.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:06.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:06.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:06.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:06.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:06.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:06.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:06.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:06.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:07.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:07.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:07.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:07.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:07.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:07.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:07.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:07.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:07.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:08.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082979 2021-01-15T18:28:08.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082979 '!=' null 2021-01-15T18:28:08.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:08.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:08.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:08.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:08.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:08.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:08.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:08.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:08.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:09.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:09.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:09.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:09.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:09.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:09.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:09.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:09.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:09.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:09.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=27082979 2021-01-15T18:28:09.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 27082979 '!=' null 2021-01-15T18:28:09.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:09.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:09.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:09.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:09.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:09.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:09.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:09.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:09.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:09.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:09.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:10.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:28:10.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:10.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:10.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:10.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 2 2021-01-15T18:28:10.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:28:10.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:10.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:10.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:10.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:10.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:10.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:10.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715170 2021-01-15T18:28:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715170 '!=' null 2021-01-15T18:28:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:11.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:11.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:11.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:11.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:11.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:11.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:11.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:11.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:11.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:11.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:11.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:28:11.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:11.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:11.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:12.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2021-01-15T18:28:12.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2021-01-15T18:28:12.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:28:12.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2021-01-15T18:28:12.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:12.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:12.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:12.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:12.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:12.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:12.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:12.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:12.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:12.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:12.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:28:12.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:12.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:12.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:13.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 2 2021-01-15T18:28:13.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:28:13.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:28:13.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:28:13.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:28:13.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:28:13.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:28:13.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:28:13.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:28:13.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26209910 2021-01-15T18:28:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26209910 '!=' null 2021-01-15T18:28:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:28:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:28:13.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:28:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:28:13.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:28:13.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:28:13.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:28:13.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:28:13.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:28:13.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:28:13.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:28:14.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:28:14.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:28:14.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:28:14.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:28:14.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T18:28:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:28:14.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:28:14.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:498: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2021-01-15T18:28:14.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:502: TEST_recovery_pool_priority: echo TEST PASSED 2021-01-15T18:28:14.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:505: TEST_recovery_pool_priority: delete_pool test1 2021-01-15T18:28:14.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T18:28:14.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T18:28:14.831 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2021-01-15T18:28:15.354 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T18:28:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:506: TEST_recovery_pool_priority: delete_pool test2 2021-01-15T18:28:15.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T18:28:15.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T18:28:15.968 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T18:28:15.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:507: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2021-01-15T18:28:15.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:28:15.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:28:15.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:28:15.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:28:15.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:28:16.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:28:16.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:508: TEST_recovery_pool_priority: return 0 2021-01-15T18:28:16.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-01-15T18:28:16.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-01-15T18:28:16.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:28:16.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-01-15T18:28:16.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:28:16.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:28:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:28:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:28:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:28:16.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:28:16.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:28:16.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:28:16.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:28:16.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:28:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:28:16.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:28:16.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:28:16.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:28:16.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:28:16.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:28:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:28:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:28:16.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-01-15T18:28:16.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:28:16.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.763429 2021-01-15T18:28:16.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:28:16.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:28:16.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: for func in $funcs 2021-01-15T18:28:16.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:36: run: setup td/osd-recovery-prio 2021-01-15T18:28:16.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2021-01-15T18:28:16.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2021-01-15T18:28:16.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-01-15T18:28:16.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:28:16.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-01-15T18:28:16.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:28:16.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:28:16.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:28:16.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:28:16.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:28:16.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:28:16.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:28:16.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:28:16.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:28:16.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:28:16.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:28:16.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:28:16.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:28:16.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:28:16.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:28:16.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:28:16.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:28:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:28:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-01-15T18:28:16.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:28:16.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.763429 2021-01-15T18:28:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:28:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:28:16.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2021-01-15T18:28:16.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:28:16.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.763429 2021-01-15T18:28:16.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:28:16.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:28:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:28:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2021-01-15T18:28:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: TEST_recovery_priority td/osd-recovery-prio 2021-01-15T18:28:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:44: TEST_recovery_priority: local dir=td/osd-recovery-prio 2021-01-15T18:28:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:45: TEST_recovery_priority: local pools=10 2021-01-15T18:28:16.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:46: TEST_recovery_priority: local OSDS=5 2021-01-15T18:28:16.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local max_tries=10 2021-01-15T18:28:16.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2021-01-15T18:28:16.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2021-01-15T18:28:16.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:28:16.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:28:16.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:28:16.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2021-01-15T18:28:16.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2021-01-15T18:28:16.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:28:16.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:16.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:16.643 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:16.643 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.644 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:16.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:28:16.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:28:16.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:28:16.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:28:16.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:28:16.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:28:16.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:28:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:28:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:28:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:28:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:28:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.847 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.847 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-mon.a.asok 2021-01-15T18:28:16.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:28:16.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.763429/ceph-mon.a.asok config get fsid 2021-01-15T18:28:16.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:28:16.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:28:16.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:28:16.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:28:16.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:28:16.946 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:28:16.946 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:28:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:28:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:28:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:16.947 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-mon.a.asok 2021-01-15T18:28:16.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:28:16.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.763429/ceph-mon.a.asok config get mon_host 2021-01-15T18:28:17.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2021-01-15T18:28:17.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-prio 2021-01-15T18:28:17.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:28:17.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:28:17.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:28:17.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-prio/x 2021-01-15T18:28:17.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:28:17.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:28:17.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:17.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:17.556 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:17.557 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:17.557 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:17.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:17.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:28:17.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:28:17.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:51: TEST_recovery_priority: export CEPH_ARGS 2021-01-15T18:28:17.586 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: expr 5 - 1 2021-01-15T18:28:17.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: seq 0 4 2021-01-15T18:28:17.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:28:17.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2021-01-15T18:28:17.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:28:17.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:28:17.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:28:17.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:28:17.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/0 2021-01-15T18:28:17.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:28:17.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:28:17.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:28:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:28:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2021-01-15T18:28:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2021-01-15T18:28:17.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:28:17.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:28:17.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:28:17.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:28:17.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:17.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:17.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:17.596 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:17.597 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:17.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:17.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:17.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:28:17.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:28:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:28:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:28:17.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:28:17.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:28:17.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:28:17.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:17.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:28:17.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:28:17.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/0 2021-01-15T18:28:17.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:28:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=1cb5b93d-8c72-411b-8fc2-12530db151f9 2021-01-15T18:28:17.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 1cb5b93d-8c72-411b-8fc2-12530db151f9' 2021-01-15T18:28:17.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:28:17.607 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 1cb5b93d-8c72-411b-8fc2-12530db151f9 2021-01-15T18:28:17.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDB3gFgTvGHIxAA7ppew2UAdnnPS5XUuF41hg== 2021-01-15T18:28:17.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDB3gFgTvGHIxAA7ppew2UAdnnPS5XUuF41hg=="}' 2021-01-15T18:28:17.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 1cb5b93d-8c72-411b-8fc2-12530db151f9 -i td/osd-recovery-prio/0/new.json 2021-01-15T18:28:17.943 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:17.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/0/new.json 2021-01-15T18:28:17.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDB3gFgTvGHIxAA7ppew2UAdnnPS5XUuF41hg== --osd-uuid 1cb5b93d-8c72-411b-8fc2-12530db151f9 2021-01-15T18:28:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.977+0000 7fa510de6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:17.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.979+0000 7fa510de6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:17.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.980+0000 7fa510de6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:17.981 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.980+0000 7fa510de6080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:28:17.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.980+0000 7fa510de6080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:28:17.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:17.980+0000 7fa510de6080 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2021-01-15T18:28:18.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:18.025+0000 7fa510de6080 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2021-01-15T18:28:22.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2021-01-15T18:28:22.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:28:22.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:28:22.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:28:22.645 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:28:23.167 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:28:23.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:28:23.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:23.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:28:23.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:28:23.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:28:23.188 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:23.187+0000 7fd469be6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:23.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:23.207+0000 7fd469be6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:23.210 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:23.209+0000 7fd469be6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:28:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:28:23.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:28:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:28:23.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:28:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:23.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:28:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:23.567 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:23.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:28:23.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:23.726+0000 7fd469be6080 -1 Falling back to public interface 2021-01-15T18:28:23.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:28:24.518 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:24.517+0000 7fd469be6080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:28:24.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:28:24.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:24.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:28:24.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:24.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:28:24.973 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:25.374 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/784151,v1:127.0.0.1:6803/784151] [v2:127.0.0.1:6804/784151,v1:127.0.0.1:6805/784151] exists,up 1cb5b93d-8c72-411b-8fc2-12530db151f9 2021-01-15T18:28:25.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:28:25.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:28:25.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:28:25.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:28:25.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2021-01-15T18:28:25.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:28:25.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:28:25.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:28:25.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:28:25.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/1 2021-01-15T18:28:25.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:28:25.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:28:25.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:28:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:28:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2021-01-15T18:28:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2021-01-15T18:28:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:28:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:28:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:28:25.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:28:25.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:25.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:25.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:28:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:28:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:28:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:28:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:28:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:28:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:28:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:28:25.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:28:25.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/1 2021-01-15T18:28:25.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:28:25.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=13d2410d-76e3-47b4-91e2-b28de41356ea 2021-01-15T18:28:25.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 13d2410d-76e3-47b4-91e2-b28de41356ea' 2021-01-15T18:28:25.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:28:25.384 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 13d2410d-76e3-47b4-91e2-b28de41356ea 2021-01-15T18:28:25.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDJ3gFghb8mFxAAJnhaAQh/oYnV/7xurfmvCQ== 2021-01-15T18:28:25.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDJ3gFghb8mFxAAJnhaAQh/oYnV/7xurfmvCQ=="}' 2021-01-15T18:28:25.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 13d2410d-76e3-47b4-91e2-b28de41356ea -i td/osd-recovery-prio/1/new.json 2021-01-15T18:28:25.923 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:25.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/1/new.json 2021-01-15T18:28:25.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDJ3gFghb8mFxAAJnhaAQh/oYnV/7xurfmvCQ== --osd-uuid 13d2410d-76e3-47b4-91e2-b28de41356ea 2021-01-15T18:28:25.959 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.958+0000 7f3ca7900080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:25.961 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.960+0000 7f3ca7900080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:25.962 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.961+0000 7f3ca7900080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:25.963 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.962+0000 7f3ca7900080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:28:25.963 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.962+0000 7f3ca7900080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:28:25.963 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:25.962+0000 7f3ca7900080 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2021-01-15T18:28:26.040 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:26.040+0000 7f3ca7900080 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2021-01-15T18:28:30.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2021-01-15T18:28:30.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:28:30.633 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:28:30.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:28:30.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:28:31.165 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:28:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:28:31.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:28:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:28:31.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:28:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:31.187+0000 7f358cd34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:31.189 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:31.189+0000 7f358cd34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:31.191 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:31.190+0000 7f358cd34080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:31.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:28:31.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:28:31.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:28:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:28:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:28:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:28:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:31.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:28:31.557 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:31.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:31.707+0000 7f358cd34080 -1 Falling back to public interface 2021-01-15T18:28:31.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:28:32.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:32.497+0000 7f358cd34080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:28:32.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:28:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:28:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:32.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:28:32.980 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:33.380 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/785415,v1:127.0.0.1:6811/785415] [v2:127.0.0.1:6812/785415,v1:127.0.0.1:6813/785415] exists,up 13d2410d-76e3-47b4-91e2-b28de41356ea 2021-01-15T18:28:33.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:28:33.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:28:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:28:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:28:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2021-01-15T18:28:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:28:33.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:28:33.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:28:33.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:28:33.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/2 2021-01-15T18:28:33.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:28:33.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:28:33.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:28:33.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:28:33.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2021-01-15T18:28:33.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2021-01-15T18:28:33.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:28:33.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:28:33.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:28:33.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:28:33.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:33.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:33.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:33.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:33.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:33.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:28:33.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:28:33.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:28:33.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:28:33.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:28:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:28:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:28:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:28:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:28:33.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/2 2021-01-15T18:28:33.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:28:33.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=fe725b9e-54b3-422e-9a9b-4982d7e6a705 2021-01-15T18:28:33.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 fe725b9e-54b3-422e-9a9b-4982d7e6a705' 2021-01-15T18:28:33.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:28:33.390 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 fe725b9e-54b3-422e-9a9b-4982d7e6a705 2021-01-15T18:28:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDR3gFgaiKCFxAAK3IfYbde5Dbok2XYUvDLkg== 2021-01-15T18:28:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDR3gFgaiKCFxAAK3IfYbde5Dbok2XYUvDLkg=="}' 2021-01-15T18:28:33.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new fe725b9e-54b3-422e-9a9b-4982d7e6a705 -i td/osd-recovery-prio/2/new.json 2021-01-15T18:28:33.997 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:28:34.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/2/new.json 2021-01-15T18:28:34.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDR3gFgaiKCFxAAK3IfYbde5Dbok2XYUvDLkg== --osd-uuid fe725b9e-54b3-422e-9a9b-4982d7e6a705 2021-01-15T18:28:34.033 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.032+0000 7fedbf0be080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:34.034 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.034+0000 7fedbf0be080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:34.035 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.034+0000 7fedbf0be080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.035+0000 7fedbf0be080 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-01-15T18:28:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.035+0000 7fedbf0be080 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-01-15T18:28:34.036 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.035+0000 7fedbf0be080 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2021-01-15T18:28:34.092 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:34.091+0000 7fedbf0be080 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2021-01-15T18:28:38.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2021-01-15T18:28:38.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:28:38.617 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:28:38.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:28:38.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:28:39.138 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:28:39.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:28:39.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:39.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:28:39.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:28:39.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:28:39.160 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:39.159+0000 7fab76d79080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:39.195 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:39.194+0000 7fab76d79080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:39.196 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:39.195+0000 7fab76d79080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:39.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:28:39.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:28:39.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:28:39.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:28:39.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:28:39.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:39.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:28:39.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:39.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:28:39.543 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:39.716 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:39.715+0000 7fab76d79080 -1 Falling back to public interface 2021-01-15T18:28:39.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:28:40.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:40.570+0000 7fab76d79080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:28:40.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:28:40.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:40.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:28:40.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:40.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:28:40.933 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:41.348 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/786687,v1:127.0.0.1:6819/786687] [v2:127.0.0.1:6820/786687,v1:127.0.0.1:6821/786687] exists,up fe725b9e-54b3-422e-9a9b-4982d7e6a705 2021-01-15T18:28:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:28:41.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:28:41.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:28:41.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:28:41.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2021-01-15T18:28:41.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:28:41.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:28:41.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:28:41.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:28:41.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/3 2021-01-15T18:28:41.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:28:41.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:28:41.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:28:41.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:28:41.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2021-01-15T18:28:41.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2021-01-15T18:28:41.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:28:41.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:28:41.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:28:41.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:28:41.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:41.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:41.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:28:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:28:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:28:41.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:28:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:28:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:28:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:28:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:28:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:28:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/3 2021-01-15T18:28:41.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:28:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b80c3c0a-5cee-4dff-882b-2d354422e320 2021-01-15T18:28:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 b80c3c0a-5cee-4dff-882b-2d354422e320' 2021-01-15T18:28:41.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:28:41.359 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 b80c3c0a-5cee-4dff-882b-2d354422e320 2021-01-15T18:28:41.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDZ3gFgNCufFRAAg/2KrfM0j2axEJTSJ/jzyA== 2021-01-15T18:28:41.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDZ3gFgNCufFRAAg/2KrfM0j2axEJTSJ/jzyA=="}' 2021-01-15T18:28:41.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b80c3c0a-5cee-4dff-882b-2d354422e320 -i td/osd-recovery-prio/3/new.json 2021-01-15T18:28:41.846 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:28:41.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/3/new.json 2021-01-15T18:28:41.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZ3gFgNCufFRAAg/2KrfM0j2axEJTSJ/jzyA== --osd-uuid b80c3c0a-5cee-4dff-882b-2d354422e320 2021-01-15T18:28:41.882 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.881+0000 7f4b01dfe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:41.883 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.883+0000 7f4b01dfe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:41.885 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.884+0000 7f4b01dfe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:41.886 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.885+0000 7f4b01dfe080 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-01-15T18:28:41.886 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.885+0000 7f4b01dfe080 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-01-15T18:28:41.886 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.885+0000 7f4b01dfe080 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2021-01-15T18:28:41.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:41.925+0000 7f4b01dfe080 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2021-01-15T18:28:46.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2021-01-15T18:28:46.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:28:46.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:28:46.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:28:46.750 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:28:47.277 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:28:47.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:28:47.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:47.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:28:47.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:28:47.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:28:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:47.298+0000 7f2e2da62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:47.300+0000 7f2e2da62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:47.302 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:47.301+0000 7f2e2da62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:47.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:28:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:28:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:28:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:28:47.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:28:47.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:47.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:28:47.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:47.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:28:47.700 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:47.820 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:47.820+0000 7f2e2da62080 -1 Falling back to public interface 2021-01-15T18:28:48.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:28:49.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:48.646+0000 7f2e2da62080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:28:49.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:28:49.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:49.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:28:49.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:28:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:28:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:28:49.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:28:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:28:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:55: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2021-01-15T18:28:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-prio 2021-01-15T18:28:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:28:49.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:28:49.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:28:49.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-prio/4 2021-01-15T18:28:49.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20' 2021-01-15T18:28:49.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:28:49.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:28:49.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:28:49.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2021-01-15T18:28:49.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2021-01-15T18:28:49.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:28:49.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:28:49.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2021-01-15T18:28:49.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:28:49.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:28:49.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:28:49.887 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:28:49.888 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:28:49.888 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:28:49.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:49.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' 2021-01-15T18:28:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:28:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:28:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:28:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2021-01-15T18:28:49.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2021-01-15T18:28:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:28:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:28:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:28:49.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:28:49.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-prio/4 2021-01-15T18:28:49.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:28:49.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3ca4c0dd-031c-4730-95ba-1e1194e21c27 2021-01-15T18:28:49.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 3ca4c0dd-031c-4730-95ba-1e1194e21c27' 2021-01-15T18:28:49.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:28:49.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDh3gFgTWstHhAA6wipomN5lNqTWndRwwYUVA== 2021-01-15T18:28:49.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDh3gFgTWstHhAA6wipomN5lNqTWndRwwYUVA=="}' 2021-01-15T18:28:49.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3ca4c0dd-031c-4730-95ba-1e1194e21c27 -i td/osd-recovery-prio/4/new.json 2021-01-15T18:28:49.894 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:49.895 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/787951,v1:127.0.0.1:6827/787951] [v2:127.0.0.1:6828/787951,v1:127.0.0.1:6829/787951] exists,up b80c3c0a-5cee-4dff-882b-2d354422e320 2021-01-15T18:28:49.895 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 3ca4c0dd-031c-4730-95ba-1e1194e21c27 2021-01-15T18:28:50.023 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:28:50.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-prio/4/new.json 2021-01-15T18:28:50.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDh3gFgTWstHhAA6wipomN5lNqTWndRwwYUVA== --osd-uuid 3ca4c0dd-031c-4730-95ba-1e1194e21c27 2021-01-15T18:28:50.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.057+0000 7fdbbbbd5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:50.084 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.078+0000 7fdbbbbd5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:50.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.079+0000 7fdbbbbd5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:50.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.080+0000 7fdbbbbd5080 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-01-15T18:28:50.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.080+0000 7fdbbbbd5080 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-01-15T18:28:50.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.080+0000 7fdbbbbd5080 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2021-01-15T18:28:50.162 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:50.161+0000 7fdbbbbd5080 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2021-01-15T18:28:54.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2021-01-15T18:28:54.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:28:54.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:28:54.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:28:54.989 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:28:55.512 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:28:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:28:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=f6b440be-d8d8-4812-94ff-223734ca508e --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.763429/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:28:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:28:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:28:55.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:28:55.534 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:55.533+0000 7f9fe6420080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:55.553 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:55.553+0000 7f9fe6420080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:55.555 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:55.554+0000 7f9fe6420080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:28:55.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:28:55.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:28:55.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:28:55.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:28:55.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:28:55.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:55.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:28:55.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:55.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:28:55.911 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:28:56.071 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:56.070+0000 7f9fe6420080 -1 Falling back to public interface 2021-01-15T18:28:56.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:28:56.872 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:28:56.871+0000 7f9fe6420080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:28:57.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:28:57.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:28:57.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:28:57.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:28:57.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:28:57.308 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:28:57.717 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/789215,v1:127.0.0.1:6835/789215] [v2:127.0.0.1:6836/789215,v1:127.0.0.1:6837/789215] exists,up 3ca4c0dd-031c-4730-95ba-1e1194e21c27 2021-01-15T18:28:57.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:28:57.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:28:57.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:28:57.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: seq 1 10 2021-01-15T18:28:57.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:28:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test1 1 1 2021-01-15T18:28:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T18:28:58.359 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T18:28:58.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:28:59.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test1 size 2 2021-01-15T18:29:00.262 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:29:00.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:00.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test2 1 1 2021-01-15T18:29:00.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test2 1 1 2021-01-15T18:29:00.938 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' already exists 2021-01-15T18:29:00.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:01.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test2 size 2 2021-01-15T18:29:02.846 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T18:29:02.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:02.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test3 1 1 2021-01-15T18:29:02.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test3 1 1 2021-01-15T18:29:03.457 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' already exists 2021-01-15T18:29:03.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:04.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-01-15T18:29:05.215 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T18:29:05.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:05.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test4 1 1 2021-01-15T18:29:05.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test4 1 1 2021-01-15T18:29:05.970 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' already exists 2021-01-15T18:29:05.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:06.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test4 size 2 2021-01-15T18:29:08.036 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 4 size to 2 2021-01-15T18:29:08.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:08.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test5 1 1 2021-01-15T18:29:08.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test5 1 1 2021-01-15T18:29:08.889 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' already exists 2021-01-15T18:29:08.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:09.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test5 size 2 2021-01-15T18:29:10.991 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 5 size to 2 2021-01-15T18:29:11.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:11.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test6 1 1 2021-01-15T18:29:11.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test6 1 1 2021-01-15T18:29:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' already exists 2021-01-15T18:29:11.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:12.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test6 size 2 2021-01-15T18:29:13.476 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 6 size to 2 2021-01-15T18:29:13.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:13.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test7 1 1 2021-01-15T18:29:13.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test7 1 1 2021-01-15T18:29:14.192 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' already exists 2021-01-15T18:29:14.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:15.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-01-15T18:29:15.861 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T18:29:15.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:15.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test8 1 1 2021-01-15T18:29:15.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test8 1 1 2021-01-15T18:29:16.492 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' already exists 2021-01-15T18:29:16.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:17.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test8 size 2 2021-01-15T18:29:18.484 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 8 size to 2 2021-01-15T18:29:18.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:18.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test9 1 1 2021-01-15T18:29:18.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test9 1 1 2021-01-15T18:29:19.138 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' already exists 2021-01-15T18:29:19.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test9 size 2 2021-01-15T18:29:21.091 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 9 size to 2 2021-01-15T18:29:21.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:60: TEST_recovery_priority: create_pool test10 1 1 2021-01-15T18:29:21.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test10 1 1 2021-01-15T18:29:21.702 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' already exists 2021-01-15T18:29:21.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:29:22.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: ceph osd pool set test10 size 2 2021-01-15T18:29:23.678 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 10 size to 2 2021-01-15T18:29:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: sleep 5 2021-01-15T18:29:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:65: TEST_recovery_priority: wait_for_clean 2021-01-15T18:29:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:29:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:29:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:29:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:29:28.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:29:28.691 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:29:28.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:29:28.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:29:28.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:29:28.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:29:28.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:29:28.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:29:28.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:29:28.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:29:28.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:29:29.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:29:29.181 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:29:29.181 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:29:29.181 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:29:29.182 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:29:29.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:29:29.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:29.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:29:29.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T18:29:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T18:29:29.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T18:29:29.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:29.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:29:29.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T18:29:29.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T18:29:29.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T18:29:29.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:29.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:29:29.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T18:29:29.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T18:29:29.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T18:29:29.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:29.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:29:29.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378634 2021-01-15T18:29:29.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378634 2021-01-15T18:29:29.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-81604378634' 2021-01-15T18:29:29.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:29.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:29:29.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215114 2021-01-15T18:29:29.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215114 2021-01-15T18:29:29.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452 3-81604378634 4-103079215114' 2021-01-15T18:29:29.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:29.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T18:29:29.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:29.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:29:29.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T18:29:29.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:29.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T18:29:29.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T18:29:29.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:29:29.896 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T18:29:30.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2021-01-15T18:29:30.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:30.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T18:29:30.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:30.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:29:30.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T18:29:30.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:30.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T18:29:30.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T18:29:30.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:29:30.287 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T18:29:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672972 -lt 42949672974 2021-01-15T18:29:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:29:31.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:29:31.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:29:32.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T18:29:32.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:32.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T18:29:32.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:32.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:29:32.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T18:29:32.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:32.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T18:29:32.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T18:29:32.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:29:32.092 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T18:29:32.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T18:29:32.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:32.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378634 2021-01-15T18:29:32.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:32.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:29:32.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378634 2021-01-15T18:29:32.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:32.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378634 2021-01-15T18:29:32.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2021-01-15T18:29:32.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:29:32.494 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378634 2021-01-15T18:29:32.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378635 -lt 81604378634 2021-01-15T18:29:32.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:32.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215114 2021-01-15T18:29:32.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:32.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:29:32.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215114 2021-01-15T18:29:32.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:32.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215114 2021-01-15T18:29:32.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215114' 2021-01-15T18:29:32.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:29:32.909 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215114 2021-01-15T18:29:33.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215114 -lt 103079215114 2021-01-15T18:29:33.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:29:33.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:29:33.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:29:33.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 10 == 0 2021-01-15T18:29:33.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:29:33.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:29:33.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:29:33.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:29:33.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:29:33.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:29:33.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:29:34.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=10 2021-01-15T18:29:34.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:29:34.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:29:34.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:29:34.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 10 = 10 2021-01-15T18:29:34.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:29:34.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:29:34.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:67: TEST_recovery_priority: ceph pg dump pgs 2021-01-15T18:29:35.115 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:29:35.116 INFO:tasks.workunit.client.0.smithi145.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:25.506336+0000 0'0 83:20 [2,1] 2 [2,1] 2 0'0 2021-01-15T18:29:21.437944+0000 0'0 2021-01-15T18:29:21.437944+0000 0 2021-01-15T18:29:35.116 INFO:tasks.workunit.client.0.smithi145.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:21.505941+0000 0'0 82:24 [1,2] 1 [1,2] 1 0'0 2021-01-15T18:29:18.907734+0000 0'0 2021-01-15T18:29:18.907734+0000 0 2021-01-15T18:29:35.117 INFO:tasks.workunit.client.0.smithi145.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:19.209754+0000 0'0 82:30 [3,4] 3 [3,4] 3 0'0 2021-01-15T18:29:16.268514+0000 0'0 2021-01-15T18:29:16.268514+0000 0 2021-01-15T18:29:35.117 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:16.897566+0000 0'0 82:26 [1,4] 1 [1,4] 1 0'0 2021-01-15T18:29:13.919535+0000 0'0 2021-01-15T18:29:13.919535+0000 0 2021-01-15T18:29:35.117 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:03.819081+0000 0'0 82:50 [3,1] 3 [3,1] 3 0'0 2021-01-15T18:29:00.655897+0000 0'0 2021-01-15T18:29:00.655897+0000 0 2021-01-15T18:29:35.118 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:00.966918+0000 0'0 82:55 [1,0] 1 [1,0] 1 0'0 2021-01-15T18:28:58.110760+0000 0'0 2021-01-15T18:28:58.110760+0000 0 2021-01-15T18:29:35.118 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:05.765695+0000 0'0 82:47 [1,2] 1 [1,2] 1 0'0 2021-01-15T18:29:03.219749+0000 0'0 2021-01-15T18:29:03.219749+0000 0 2021-01-15T18:29:35.118 INFO:tasks.workunit.client.0.smithi145.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:08.601631+0000 0'0 82:38 [4,0] 4 [4,0] 4 0'0 2021-01-15T18:29:05.631003+0000 0'0 2021-01-15T18:29:05.631003+0000 0 2021-01-15T18:29:35.119 INFO:tasks.workunit.client.0.smithi145.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:11.975318+0000 0'0 82:40 [3,4] 3 [3,4] 3 0'0 2021-01-15T18:29:08.441916+0000 0'0 2021-01-15T18:29:08.441916+0000 0 2021-01-15T18:29:35.119 INFO:tasks.workunit.client.0.smithi145.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T18:29:14.620482+0000 0'0 82:35 [4,2] 4 [4,2] 4 0'0 2021-01-15T18:29:11.335279+0000 0'0 2021-01-15T18:29:11.335279+0000 0 2021-01-15T18:29:35.120 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:29:35.120 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:29:35.121 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:29:35.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:71: TEST_recovery_priority: local PG1 2021-01-15T18:29:35.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:72: TEST_recovery_priority: local POOLNUM1 2021-01-15T18:29:35.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:73: TEST_recovery_priority: local pool1 2021-01-15T18:29:35.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local chk_osd1_1 2021-01-15T18:29:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local chk_osd1_2 2021-01-15T18:29:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local PG2 2021-01-15T18:29:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local POOLNUM2 2021-01-15T18:29:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:79: TEST_recovery_priority: local pool2 2021-01-15T18:29:35.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local chk_osd2 2021-01-15T18:29:35.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local PG3 2021-01-15T18:29:35.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local POOLNUM3 2021-01-15T18:29:35.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:84: TEST_recovery_priority: local pool3 2021-01-15T18:29:35.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: seq 1 10 2021-01-15T18:29:35.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 1.0 --format=json 2021-01-15T18:29:35.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:35.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:35.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-01-15T18:29:35.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:35.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-01-15T18:29:35.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z '' ']' 2021-01-15T18:29:35.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: PG1=1.0 2021-01-15T18:29:35.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: POOLNUM1=1 2021-01-15T18:29:35.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:95: TEST_recovery_priority: pool1=test1 2021-01-15T18:29:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: chk_osd1_1=1 2021-01-15T18:29:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: chk_osd1_2=0 2021-01-15T18:29:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 2.0 --format=json 2021-01-15T18:29:35.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:35.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:35.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-01-15T18:29:35.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=1 2021-01-15T18:29:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-01-15T18:29:35.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-01-15T18:29:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 3.0 --format=json 2021-01-15T18:29:35.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:36.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:36.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-01-15T18:29:36.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:36.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-01-15T18:29:36.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:36.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-01-15T18:29:36.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: PG2=3.0 2021-01-15T18:29:36.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: POOLNUM2=3 2021-01-15T18:29:36.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:102: TEST_recovery_priority: pool2=test3 2021-01-15T18:29:36.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: chk_osd2=2 2021-01-15T18:29:36.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:36.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 4.0 --format=json 2021-01-15T18:29:36.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:36.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:36.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-01-15T18:29:36.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:36.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=0 2021-01-15T18:29:36.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-01-15T18:29:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-01-15T18:29:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:36.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 5.0 --format=json 2021-01-15T18:29:36.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:37.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=3 2021-01-15T18:29:37.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-01-15T18:29:37.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:37.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-01-15T18:29:37.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-01-15T18:29:37.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:37.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 6.0 --format=json 2021-01-15T18:29:37.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:37.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=4 2021-01-15T18:29:37.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=2 2021-01-15T18:29:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-01-15T18:29:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-01-15T18:29:37.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: ceph pg map 7.0 --format=json 2021-01-15T18:29:37.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:88: TEST_recovery_priority: jq '.acting[]' 2021-01-15T18:29:37.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: local test_osd1=1 2021-01-15T18:29:37.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2021-01-15T18:29:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:90: TEST_recovery_priority: local test_osd2=4 2021-01-15T18:29:37.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: '[' -z 1.0 ']' 2021-01-15T18:29:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-01-15T18:29:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-01-15T18:29:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: PG3=7.0 2021-01-15T18:29:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: POOLNUM3=7 2021-01-15T18:29:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:108: TEST_recovery_priority: pool3=test7 2021-01-15T18:29:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: break 2021-01-15T18:29:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2021-01-15T18:29:37.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:114: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2021-01-15T18:29:37.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: seq 1 10 2021-01-15T18:29:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:37.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-01-15T18:29:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-01-15T18:29:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test2 2021-01-15T18:29:37.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test2 2021-01-15T18:29:37.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-01-15T18:29:38.428 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test2' does not exist 2021-01-15T18:29:38.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:38.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-01-15T18:29:38.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:38.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-01-15T18:29:38.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test4 2021-01-15T18:29:38.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test4 2021-01-15T18:29:38.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-01-15T18:29:38.945 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test4' does not exist 2021-01-15T18:29:38.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:38.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-01-15T18:29:38.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test5 2021-01-15T18:29:38.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test5 2021-01-15T18:29:38.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-01-15T18:29:39.476 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test5' does not exist 2021-01-15T18:29:39.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-01-15T18:29:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test6 2021-01-15T18:29:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test6 2021-01-15T18:29:39.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-01-15T18:29:40.109 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test6' does not exist 2021-01-15T18:29:40.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:40.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-01-15T18:29:40.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:40.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-01-15T18:29:40.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test8 2021-01-15T18:29:40.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test8 2021-01-15T18:29:40.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-01-15T18:29:40.699 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test8' does not exist 2021-01-15T18:29:40.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:40.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-01-15T18:29:40.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test9 2021-01-15T18:29:40.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test9 2021-01-15T18:29:40.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-01-15T18:29:41.280 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test9' does not exist 2021-01-15T18:29:41.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:120: TEST_recovery_priority: for p in $(seq 1 $pools) 2021-01-15T18:29:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:122: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-01-15T18:29:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:124: TEST_recovery_priority: delete_pool test10 2021-01-15T18:29:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test10 2021-01-15T18:29:41.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-01-15T18:29:41.947 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test10' does not exist 2021-01-15T18:29:41.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:128: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-01-15T18:29:42.709 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 1 2021-01-15T18:29:42.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:129: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-01-15T18:29:43.748 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 1 2021-01-15T18:29:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:130: TEST_recovery_priority: wait_for_clean 2021-01-15T18:29:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:29:43.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:29:43.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:29:43.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:29:43.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:29:43.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:29:43.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:29:43.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:29:43.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:29:43.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:29:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:29:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:29:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:29:43.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:29:43.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:29:44.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:29:44.260 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:29:44.261 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:29:44.261 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:29:44.261 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:29:44.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:29:44.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:44.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:29:44.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2021-01-15T18:29:44.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2021-01-15T18:29:44.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2021-01-15T18:29:44.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:44.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:29:44.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T18:29:44.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T18:29:44.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2021-01-15T18:29:44.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:44.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:29:44.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T18:29:44.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T18:29:44.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457' 2021-01-15T18:29:44.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:44.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:29:44.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2021-01-15T18:29:44.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2021-01-15T18:29:44.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-81604378639' 2021-01-15T18:29:44.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:29:44.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:29:44.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215119 2021-01-15T18:29:44.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215119 2021-01-15T18:29:44.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-81604378639 4-103079215119' 2021-01-15T18:29:44.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:44.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2021-01-15T18:29:44.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:44.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:29:44.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-01-15T18:29:44.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:44.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2021-01-15T18:29:44.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-01-15T18:29:44.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:29:44.944 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836500 2021-01-15T18:29:45.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836500 2021-01-15T18:29:45.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:29:46.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:29:46.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:29:46.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2021-01-15T18:29:46.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:46.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T18:29:46.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:46.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:29:46.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T18:29:46.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:46.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T18:29:46.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T18:29:46.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:29:46.731 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T18:29:47.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672979 2021-01-15T18:29:47.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:47.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T18:29:47.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:47.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:29:47.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T18:29:47.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:47.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T18:29:47.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T18:29:47.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:29:47.123 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T18:29:47.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2021-01-15T18:29:47.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:47.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378639 2021-01-15T18:29:47.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:47.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:29:47.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378639 2021-01-15T18:29:47.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:47.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2021-01-15T18:29:47.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378639' 2021-01-15T18:29:47.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:29:47.524 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378639 2021-01-15T18:29:47.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378640 -lt 81604378639 2021-01-15T18:29:47.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:29:47.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215119 2021-01-15T18:29:47.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:29:47.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:29:47.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215119 2021-01-15T18:29:47.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:29:47.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215119 2021-01-15T18:29:47.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215119' 2021-01-15T18:29:47.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:29:47.915 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215119 2021-01-15T18:29:48.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215119 -lt 103079215119 2021-01-15T18:29:48.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:29:48.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:29:48.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:29:48.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T18:29:48.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:29:48.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:29:48.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:29:48.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:29:48.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:29:48.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:29:48.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:29:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T18:29:49.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:29:49.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:29:49.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:29:49.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T18:29:49.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:29:49.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:29:49.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2021-01-15T18:29:49.772 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records in 2021-01-15T18:29:49.772 INFO:tasks.workunit.client.0.smithi145.stderr:10+0 records out 2021-01-15T18:29:49.773 INFO:tasks.workunit.client.0.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0456006 s, 230 MB/s 2021-01-15T18:29:49.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: p=1 2021-01-15T18:29:49.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T18:29:49.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-01-15T18:29:49.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:49.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2021-01-15T18:29:50.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:50.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2021-01-15T18:29:50.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:50.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2021-01-15T18:29:51.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:51.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2021-01-15T18:29:52.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:52.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2021-01-15T18:29:52.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:52.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2021-01-15T18:29:53.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:53.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2021-01-15T18:29:54.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:54.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2021-01-15T18:29:55.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:55.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2021-01-15T18:29:55.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:55.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2021-01-15T18:29:56.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:56.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2021-01-15T18:29:57.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:57.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2021-01-15T18:29:57.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:57.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2021-01-15T18:29:58.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:58.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2021-01-15T18:29:59.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:29:59.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2021-01-15T18:30:00.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:00.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2021-01-15T18:30:00.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:00.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2021-01-15T18:30:01.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:01.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2021-01-15T18:30:02.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:02.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2021-01-15T18:30:03.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:03.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2021-01-15T18:30:03.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2021-01-15T18:30:04.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:04.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2021-01-15T18:30:05.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:05.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2021-01-15T18:30:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:05.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2021-01-15T18:30:07.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:07.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2021-01-15T18:30:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2021-01-15T18:30:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2021-01-15T18:30:09.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:09.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2021-01-15T18:30:10.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:10.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2021-01-15T18:30:11.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:11.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2021-01-15T18:30:11.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:11.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2021-01-15T18:30:12.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:12.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2021-01-15T18:30:13.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:13.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2021-01-15T18:30:13.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:13.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2021-01-15T18:30:14.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:14.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2021-01-15T18:30:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2021-01-15T18:30:15.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:15.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2021-01-15T18:30:16.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:16.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2021-01-15T18:30:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2021-01-15T18:30:18.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:18.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2021-01-15T18:30:18.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:18.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2021-01-15T18:30:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2021-01-15T18:30:20.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:20.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2021-01-15T18:30:21.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:21.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2021-01-15T18:30:21.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:21.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2021-01-15T18:30:22.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:23.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2021-01-15T18:30:23.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:23.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2021-01-15T18:30:25.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:25.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2021-01-15T18:30:25.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:25.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2021-01-15T18:30:26.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:26.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2021-01-15T18:30:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2021-01-15T18:30:27.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:27.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2021-01-15T18:30:28.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:28.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2021-01-15T18:30:29.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:29.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2021-01-15T18:30:29.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:29.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2021-01-15T18:30:30.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:30.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2021-01-15T18:30:31.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:31.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2021-01-15T18:30:32.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:32.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2021-01-15T18:30:32.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:32.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2021-01-15T18:30:33.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:33.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2021-01-15T18:30:34.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:34.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2021-01-15T18:30:34.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:34.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2021-01-15T18:30:35.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:35.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2021-01-15T18:30:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:36.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2021-01-15T18:30:36.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:36.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2021-01-15T18:30:37.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:37.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2021-01-15T18:30:38.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:38.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2021-01-15T18:30:38.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:38.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2021-01-15T18:30:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2021-01-15T18:30:40.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:40.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2021-01-15T18:30:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:41.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2021-01-15T18:30:42.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:42.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2021-01-15T18:30:43.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:43.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2021-01-15T18:30:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:44.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2021-01-15T18:30:44.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:44.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2021-01-15T18:30:45.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:45.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2021-01-15T18:30:46.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:46.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2021-01-15T18:30:47.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:47.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2021-01-15T18:30:47.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:47.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2021-01-15T18:30:48.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:48.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2021-01-15T18:30:49.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:49.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2021-01-15T18:30:50.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:50.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2021-01-15T18:30:50.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:50.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2021-01-15T18:30:51.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:51.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2021-01-15T18:30:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:52.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2021-01-15T18:30:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:52.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2021-01-15T18:30:53.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:53.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2021-01-15T18:30:53.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:53.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2021-01-15T18:30:54.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:54.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2021-01-15T18:30:55.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:55.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2021-01-15T18:30:56.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:56.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2021-01-15T18:30:56.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:56.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2021-01-15T18:30:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:57.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2021-01-15T18:30:58.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:30:58.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2021-01-15T18:31:00.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:00.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2021-01-15T18:31:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:01.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2021-01-15T18:31:04.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:04.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2021-01-15T18:31:05.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:05.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2021-01-15T18:31:05.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:05.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2021-01-15T18:31:06.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:06.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2021-01-15T18:31:07.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:07.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2021-01-15T18:31:08.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:08.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2021-01-15T18:31:08.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:08.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2021-01-15T18:31:09.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:09.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2021-01-15T18:31:10.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:10.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2021-01-15T18:31:11.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:11.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2021-01-15T18:31:11.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:11.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2021-01-15T18:31:12.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2021-01-15T18:31:13.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:13.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2021-01-15T18:31:14.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:14.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2021-01-15T18:31:14.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:14.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2021-01-15T18:31:15.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:15.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2021-01-15T18:31:16.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:16.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2021-01-15T18:31:16.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:16.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2021-01-15T18:31:17.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:17.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2021-01-15T18:31:18.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:18.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2021-01-15T18:31:19.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:19.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2021-01-15T18:31:19.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:19.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2021-01-15T18:31:21.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:21.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2021-01-15T18:31:22.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:22.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2021-01-15T18:31:22.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:22.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2021-01-15T18:31:23.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:23.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2021-01-15T18:31:24.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:24.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2021-01-15T18:31:24.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:24.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2021-01-15T18:31:25.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:25.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2021-01-15T18:31:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2021-01-15T18:31:27.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:27.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2021-01-15T18:31:27.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:27.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2021-01-15T18:31:28.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:28.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2021-01-15T18:31:29.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:29.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2021-01-15T18:31:29.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:29.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2021-01-15T18:31:30.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:30.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2021-01-15T18:31:31.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:31.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2021-01-15T18:31:32.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:32.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2021-01-15T18:31:32.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:32.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2021-01-15T18:31:33.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:33.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2021-01-15T18:31:34.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:34.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2021-01-15T18:31:34.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:34.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2021-01-15T18:31:35.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:35.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2021-01-15T18:31:36.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:36.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2021-01-15T18:31:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2021-01-15T18:31:38.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:38.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2021-01-15T18:31:39.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:39.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2021-01-15T18:31:39.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:39.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2021-01-15T18:31:40.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:40.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2021-01-15T18:31:41.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:41.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2021-01-15T18:31:41.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:41.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2021-01-15T18:31:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:42.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2021-01-15T18:31:43.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:43.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2021-01-15T18:31:44.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:44.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2021-01-15T18:31:44.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:44.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2021-01-15T18:31:45.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:45.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2021-01-15T18:31:46.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:46.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2021-01-15T18:31:46.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:46.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2021-01-15T18:31:47.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:47.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2021-01-15T18:31:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:48.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2021-01-15T18:31:49.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:49.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2021-01-15T18:31:49.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:49.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2021-01-15T18:31:50.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:50.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2021-01-15T18:31:51.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:51.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2021-01-15T18:31:52.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:52.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2021-01-15T18:31:52.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:52.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2021-01-15T18:31:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2021-01-15T18:31:54.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:54.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2021-01-15T18:31:55.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:55.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2021-01-15T18:31:56.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:56.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2021-01-15T18:31:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2021-01-15T18:31:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2021-01-15T18:31:58.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:58.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2021-01-15T18:31:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:31:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2021-01-15T18:32:00.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:00.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2021-01-15T18:32:00.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:00.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2021-01-15T18:32:01.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:01.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2021-01-15T18:32:02.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:02.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2021-01-15T18:32:02.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:02.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2021-01-15T18:32:03.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:03.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2021-01-15T18:32:04.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:04.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2021-01-15T18:32:05.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:05.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2021-01-15T18:32:06.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:06.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2021-01-15T18:32:06.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:06.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2021-01-15T18:32:07.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:07.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2021-01-15T18:32:07.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:07.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2021-01-15T18:32:08.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:08.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2021-01-15T18:32:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:09.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2021-01-15T18:32:10.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:10.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2021-01-15T18:32:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:10.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2021-01-15T18:32:11.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:11.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2021-01-15T18:32:12.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:12.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2021-01-15T18:32:14.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:14.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2021-01-15T18:32:16.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:16.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2021-01-15T18:32:18.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:18.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2021-01-15T18:32:19.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:19.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2021-01-15T18:32:20.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:20.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2021-01-15T18:32:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:20.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2021-01-15T18:32:21.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:21.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2021-01-15T18:32:22.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:22.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2021-01-15T18:32:22.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:22.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2021-01-15T18:32:23.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:23.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2021-01-15T18:32:24.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:24.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2021-01-15T18:32:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:25.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2021-01-15T18:32:25.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 1 + 1 2021-01-15T18:32:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=2 2021-01-15T18:32:25.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T18:32:25.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-01-15T18:32:26.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:26.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2021-01-15T18:32:26.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:26.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2021-01-15T18:32:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:26.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2021-01-15T18:32:27.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:27.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2021-01-15T18:32:27.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:27.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2021-01-15T18:32:27.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:27.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2021-01-15T18:32:28.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:28.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2021-01-15T18:32:29.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:29.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2021-01-15T18:32:29.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:29.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2021-01-15T18:32:30.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:30.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2021-01-15T18:32:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2021-01-15T18:32:30.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:32.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2021-01-15T18:32:32.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:32.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2021-01-15T18:32:32.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:32.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2021-01-15T18:32:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2021-01-15T18:32:33.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:33.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2021-01-15T18:32:33.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:33.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2021-01-15T18:32:33.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:33.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2021-01-15T18:32:34.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:34.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2021-01-15T18:32:34.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:34.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2021-01-15T18:32:35.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:35.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2021-01-15T18:32:35.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:35.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2021-01-15T18:32:35.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:35.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2021-01-15T18:32:36.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:36.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2021-01-15T18:32:36.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:36.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2021-01-15T18:32:36.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:36.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2021-01-15T18:32:37.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:37.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2021-01-15T18:32:37.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:37.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2021-01-15T18:32:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:38.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2021-01-15T18:32:38.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:38.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2021-01-15T18:32:39.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:39.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2021-01-15T18:32:39.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:39.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2021-01-15T18:32:39.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:39.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2021-01-15T18:32:40.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:40.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2021-01-15T18:32:40.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:40.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2021-01-15T18:32:41.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:41.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2021-01-15T18:32:42.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:42.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2021-01-15T18:32:42.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:42.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2021-01-15T18:32:43.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:43.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2021-01-15T18:32:43.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:43.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2021-01-15T18:32:43.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:43.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2021-01-15T18:32:44.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:44.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2021-01-15T18:32:44.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:44.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2021-01-15T18:32:44.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:44.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2021-01-15T18:32:45.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:45.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2021-01-15T18:32:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:46.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2021-01-15T18:32:46.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:46.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2021-01-15T18:32:46.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:46.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2021-01-15T18:32:47.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:47.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2021-01-15T18:32:47.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:47.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2021-01-15T18:32:47.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:47.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2021-01-15T18:32:48.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2021-01-15T18:32:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:48.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2021-01-15T18:32:49.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:49.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2021-01-15T18:32:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:49.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2021-01-15T18:32:49.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:49.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2021-01-15T18:32:50.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:50.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2021-01-15T18:32:50.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2021-01-15T18:32:50.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:50.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2021-01-15T18:32:52.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:52.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2021-01-15T18:32:52.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:52.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2021-01-15T18:32:53.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:53.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2021-01-15T18:32:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2021-01-15T18:32:53.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:53.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2021-01-15T18:32:54.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:54.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2021-01-15T18:32:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:54.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2021-01-15T18:32:55.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2021-01-15T18:32:55.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:55.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2021-01-15T18:32:55.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:55.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2021-01-15T18:32:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2021-01-15T18:32:56.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2021-01-15T18:32:56.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:56.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2021-01-15T18:32:57.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:57.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2021-01-15T18:32:58.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:58.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2021-01-15T18:32:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2021-01-15T18:32:58.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:58.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2021-01-15T18:32:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:59.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2021-01-15T18:32:59.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:59.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2021-01-15T18:32:59.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:32:59.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2021-01-15T18:33:00.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:00.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2021-01-15T18:33:00.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:00.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2021-01-15T18:33:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:00.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2021-01-15T18:33:02.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:02.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2021-01-15T18:33:03.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:03.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2021-01-15T18:33:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:03.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2021-01-15T18:33:04.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:04.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2021-01-15T18:33:06.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:06.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2021-01-15T18:33:06.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:06.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2021-01-15T18:33:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2021-01-15T18:33:09.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:09.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2021-01-15T18:33:09.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:09.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2021-01-15T18:33:09.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:09.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2021-01-15T18:33:10.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:10.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2021-01-15T18:33:10.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:10.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2021-01-15T18:33:11.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:11.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2021-01-15T18:33:11.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:11.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2021-01-15T18:33:12.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:12.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2021-01-15T18:33:12.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:12.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2021-01-15T18:33:12.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:12.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2021-01-15T18:33:12.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:12.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2021-01-15T18:33:13.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:13.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2021-01-15T18:33:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:13.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2021-01-15T18:33:14.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:14.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2021-01-15T18:33:14.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:14.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2021-01-15T18:33:15.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:15.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2021-01-15T18:33:15.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:15.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2021-01-15T18:33:16.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:16.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2021-01-15T18:33:17.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:17.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2021-01-15T18:33:17.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:17.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2021-01-15T18:33:18.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:18.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2021-01-15T18:33:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2021-01-15T18:33:18.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:18.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2021-01-15T18:33:19.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:19.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2021-01-15T18:33:19.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:19.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2021-01-15T18:33:19.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:19.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2021-01-15T18:33:20.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:20.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2021-01-15T18:33:20.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:20.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2021-01-15T18:33:21.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:21.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2021-01-15T18:33:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:21.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2021-01-15T18:33:22.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:22.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2021-01-15T18:33:22.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:22.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2021-01-15T18:33:22.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:22.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2021-01-15T18:33:22.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:22.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2021-01-15T18:33:23.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:23.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2021-01-15T18:33:24.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:24.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2021-01-15T18:33:24.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:24.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2021-01-15T18:33:24.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:24.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2021-01-15T18:33:25.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:25.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2021-01-15T18:33:25.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:25.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2021-01-15T18:33:26.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:26.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2021-01-15T18:33:27.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:27.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2021-01-15T18:33:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2021-01-15T18:33:28.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:28.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2021-01-15T18:33:28.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:28.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2021-01-15T18:33:28.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:28.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2021-01-15T18:33:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2021-01-15T18:33:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:29.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2021-01-15T18:33:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2021-01-15T18:33:30.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:30.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2021-01-15T18:33:31.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:31.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2021-01-15T18:33:31.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:31.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2021-01-15T18:33:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2021-01-15T18:33:32.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:32.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2021-01-15T18:33:32.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:32.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2021-01-15T18:33:33.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:33.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2021-01-15T18:33:33.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:33.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2021-01-15T18:33:34.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:34.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2021-01-15T18:33:34.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:34.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2021-01-15T18:33:34.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:34.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2021-01-15T18:33:35.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:35.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2021-01-15T18:33:35.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:35.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2021-01-15T18:33:35.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:35.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2021-01-15T18:33:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2021-01-15T18:33:37.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:37.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2021-01-15T18:33:38.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:38.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2021-01-15T18:33:38.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:38.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2021-01-15T18:33:40.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:40.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2021-01-15T18:33:40.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:40.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2021-01-15T18:33:41.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:41.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2021-01-15T18:33:41.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:41.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2021-01-15T18:33:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2021-01-15T18:33:42.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:42.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2021-01-15T18:33:42.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:42.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2021-01-15T18:33:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:43.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2021-01-15T18:33:43.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:43.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2021-01-15T18:33:43.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:43.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2021-01-15T18:33:44.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:44.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2021-01-15T18:33:44.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:44.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2021-01-15T18:33:44.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:44.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2021-01-15T18:33:45.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:45.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2021-01-15T18:33:46.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:46.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2021-01-15T18:33:46.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:46.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2021-01-15T18:33:46.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:46.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2021-01-15T18:33:46.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:46.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2021-01-15T18:33:47.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:47.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2021-01-15T18:33:47.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:47.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2021-01-15T18:33:48.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:48.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2021-01-15T18:33:49.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:49.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2021-01-15T18:33:50.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:50.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2021-01-15T18:33:50.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:50.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2021-01-15T18:33:50.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:50.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2021-01-15T18:33:50.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:50.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2021-01-15T18:33:51.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:51.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2021-01-15T18:33:51.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:51.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2021-01-15T18:33:51.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:51.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2021-01-15T18:33:53.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:53.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2021-01-15T18:33:53.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:53.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2021-01-15T18:33:53.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:53.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2021-01-15T18:33:54.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:54.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2021-01-15T18:33:54.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:54.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2021-01-15T18:33:54.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:54.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2021-01-15T18:33:54.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:54.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2021-01-15T18:33:55.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:55.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2021-01-15T18:33:56.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2021-01-15T18:33:56.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:56.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2021-01-15T18:33:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2021-01-15T18:33:57.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:57.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2021-01-15T18:33:57.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2021-01-15T18:33:57.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:57.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2021-01-15T18:33:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:33:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2021-01-15T18:34:00.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 2 + 1 2021-01-15T18:34:00.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=3 2021-01-15T18:34:00.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:134: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2021-01-15T18:34:00.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: seq 1 200 2021-01-15T18:34:00.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:00.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2021-01-15T18:34:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2021-01-15T18:34:00.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:00.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2021-01-15T18:34:00.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:00.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2021-01-15T18:34:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:01.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2021-01-15T18:34:01.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:01.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2021-01-15T18:34:02.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:02.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2021-01-15T18:34:02.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:02.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2021-01-15T18:34:02.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:02.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2021-01-15T18:34:03.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:03.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2021-01-15T18:34:03.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:03.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2021-01-15T18:34:04.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:04.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2021-01-15T18:34:05.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:05.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2021-01-15T18:34:05.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:05.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2021-01-15T18:34:06.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:06.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2021-01-15T18:34:06.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:06.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2021-01-15T18:34:06.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:06.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2021-01-15T18:34:06.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:06.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2021-01-15T18:34:07.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:07.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2021-01-15T18:34:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2021-01-15T18:34:08.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:08.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2021-01-15T18:34:08.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:08.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2021-01-15T18:34:09.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:09.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2021-01-15T18:34:10.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:10.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2021-01-15T18:34:10.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:10.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2021-01-15T18:34:11.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:11.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2021-01-15T18:34:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:12.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2021-01-15T18:34:13.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:13.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2021-01-15T18:34:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:13.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2021-01-15T18:34:14.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:14.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2021-01-15T18:34:14.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:14.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2021-01-15T18:34:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:15.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2021-01-15T18:34:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2021-01-15T18:34:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2021-01-15T18:34:16.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:16.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2021-01-15T18:34:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:16.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2021-01-15T18:34:16.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:16.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2021-01-15T18:34:17.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:17.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2021-01-15T18:34:18.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:18.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2021-01-15T18:34:18.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:18.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2021-01-15T18:34:18.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:18.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2021-01-15T18:34:18.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:18.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2021-01-15T18:34:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:19.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2021-01-15T18:34:19.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:19.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2021-01-15T18:34:20.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:20.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2021-01-15T18:34:20.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:20.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2021-01-15T18:34:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2021-01-15T18:34:22.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:22.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2021-01-15T18:34:23.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:23.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2021-01-15T18:34:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2021-01-15T18:34:24.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:24.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2021-01-15T18:34:24.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:24.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2021-01-15T18:34:24.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:24.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2021-01-15T18:34:25.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:25.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2021-01-15T18:34:25.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:25.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2021-01-15T18:34:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:25.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2021-01-15T18:34:26.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:26.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2021-01-15T18:34:26.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:26.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2021-01-15T18:34:27.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:27.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2021-01-15T18:34:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:27.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2021-01-15T18:34:27.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:27.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2021-01-15T18:34:28.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:28.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2021-01-15T18:34:28.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:28.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2021-01-15T18:34:28.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:28.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2021-01-15T18:34:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:29.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2021-01-15T18:34:29.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:29.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2021-01-15T18:34:29.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:29.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2021-01-15T18:34:30.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:30.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2021-01-15T18:34:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:31.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2021-01-15T18:34:31.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:31.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2021-01-15T18:34:32.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:32.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2021-01-15T18:34:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2021-01-15T18:34:33.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:33.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2021-01-15T18:34:33.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:33.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2021-01-15T18:34:33.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:33.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2021-01-15T18:34:34.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:34.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2021-01-15T18:34:34.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:34.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2021-01-15T18:34:35.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:35.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2021-01-15T18:34:35.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:35.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2021-01-15T18:34:35.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:35.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2021-01-15T18:34:36.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:36.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2021-01-15T18:34:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:36.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2021-01-15T18:34:36.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:36.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2021-01-15T18:34:37.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:37.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2021-01-15T18:34:38.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:38.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2021-01-15T18:34:38.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:38.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2021-01-15T18:34:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:38.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2021-01-15T18:34:39.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:39.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2021-01-15T18:34:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:39.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2021-01-15T18:34:40.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:40.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2021-01-15T18:34:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2021-01-15T18:34:41.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:41.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2021-01-15T18:34:41.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:41.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2021-01-15T18:34:41.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:41.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2021-01-15T18:34:42.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:42.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2021-01-15T18:34:44.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:44.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2021-01-15T18:34:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:44.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2021-01-15T18:34:45.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:45.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2021-01-15T18:34:46.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:46.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2021-01-15T18:34:47.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:47.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2021-01-15T18:34:47.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:47.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2021-01-15T18:34:47.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:47.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2021-01-15T18:34:48.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:48.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2021-01-15T18:34:48.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:48.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2021-01-15T18:34:48.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:48.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2021-01-15T18:34:49.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:49.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2021-01-15T18:34:50.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:50.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2021-01-15T18:34:50.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:50.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2021-01-15T18:34:50.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:50.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2021-01-15T18:34:51.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:51.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2021-01-15T18:34:51.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:51.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2021-01-15T18:34:51.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:51.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2021-01-15T18:34:52.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:52.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2021-01-15T18:34:52.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:52.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2021-01-15T18:34:53.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:53.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2021-01-15T18:34:53.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:53.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2021-01-15T18:34:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:54.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2021-01-15T18:34:55.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:55.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2021-01-15T18:34:55.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:55.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2021-01-15T18:34:55.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:55.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2021-01-15T18:34:56.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:56.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2021-01-15T18:34:57.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:57.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2021-01-15T18:34:57.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:57.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2021-01-15T18:34:57.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:57.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2021-01-15T18:34:58.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:58.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2021-01-15T18:34:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2021-01-15T18:34:58.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:58.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2021-01-15T18:34:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:34:59.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2021-01-15T18:35:00.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:00.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2021-01-15T18:35:00.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:00.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2021-01-15T18:35:00.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:00.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2021-01-15T18:35:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:01.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2021-01-15T18:35:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:01.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2021-01-15T18:35:01.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:01.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2021-01-15T18:35:01.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:01.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2021-01-15T18:35:02.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:02.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2021-01-15T18:35:03.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:03.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2021-01-15T18:35:03.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:03.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2021-01-15T18:35:04.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:04.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2021-01-15T18:35:04.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:04.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2021-01-15T18:35:04.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:04.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2021-01-15T18:35:05.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:05.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2021-01-15T18:35:06.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:06.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2021-01-15T18:35:06.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:06.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2021-01-15T18:35:06.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:06.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2021-01-15T18:35:07.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:07.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2021-01-15T18:35:07.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:07.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2021-01-15T18:35:08.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:08.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2021-01-15T18:35:08.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:08.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2021-01-15T18:35:08.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:08.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2021-01-15T18:35:09.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:09.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2021-01-15T18:35:10.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:10.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2021-01-15T18:35:10.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:10.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2021-01-15T18:35:10.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:11.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2021-01-15T18:35:11.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2021-01-15T18:35:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2021-01-15T18:35:11.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:11.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2021-01-15T18:35:11.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:11.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2021-01-15T18:35:12.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:12.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2021-01-15T18:35:13.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:13.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2021-01-15T18:35:13.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:13.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2021-01-15T18:35:13.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:13.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2021-01-15T18:35:14.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:14.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2021-01-15T18:35:14.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:14.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2021-01-15T18:35:15.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:15.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2021-01-15T18:35:16.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2021-01-15T18:35:16.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:16.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2021-01-15T18:35:17.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:17.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2021-01-15T18:35:17.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:17.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2021-01-15T18:35:17.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:17.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2021-01-15T18:35:18.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:18.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2021-01-15T18:35:18.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:18.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2021-01-15T18:35:19.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:19.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2021-01-15T18:35:19.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:19.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2021-01-15T18:35:20.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:20.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2021-01-15T18:35:20.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:20.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2021-01-15T18:35:20.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:20.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2021-01-15T18:35:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:21.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2021-01-15T18:35:22.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:22.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2021-01-15T18:35:22.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:22.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2021-01-15T18:35:22.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:22.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2021-01-15T18:35:23.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:23.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2021-01-15T18:35:23.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:23.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2021-01-15T18:35:23.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:23.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2021-01-15T18:35:24.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:24.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2021-01-15T18:35:24.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:24.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2021-01-15T18:35:25.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:25.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2021-01-15T18:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2021-01-15T18:35:26.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:26.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2021-01-15T18:35:27.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:27.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2021-01-15T18:35:28.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:28.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2021-01-15T18:35:29.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:29.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2021-01-15T18:35:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:29.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2021-01-15T18:35:30.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:30.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2021-01-15T18:35:30.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:30.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2021-01-15T18:35:31.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:31.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2021-01-15T18:35:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2021-01-15T18:35:32.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:32.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2021-01-15T18:35:32.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:32.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2021-01-15T18:35:32.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: for i in $(seq 1 $objects) 2021-01-15T18:35:32.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:138: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2021-01-15T18:35:32.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: expr 3 + 1 2021-01-15T18:35:32.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:140: TEST_recovery_priority: p=4 2021-01-15T18:35:32.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: get_not_primary test1 obj1-p1 2021-01-15T18:35:32.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test1 2021-01-15T18:35:32.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1-p1 2021-01-15T18:35:32.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test1 obj1-p1 2021-01-15T18:35:32.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test1 2021-01-15T18:35:32.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1-p1 2021-01-15T18:35:32.924 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T18:35:32.925 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:35:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:35:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-01-15T18:35:33.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:35:33.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: local otherosd=0 2021-01-15T18:35:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:145: TEST_recovery_priority: ceph pg dump pgs 2021-01-15T18:35:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:35:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 191 0 0 0 0 1996488704 0 0 571 571 active+clean 2021-01-15T18:29:44.180882+0000 96'571 96:613 [1] 1 [1] 1 0'0 2021-01-15T18:29:13.919535+0000 0'0 2021-01-15T18:29:13.919535+0000 0 2021-01-15T18:35:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:29:00.966918+0000 96'600 96:664 [1,0] 1 [1,0] 1 0'0 2021-01-15T18:28:58.110760+0000 0'0 2021-01-15T18:28:58.110760+0000 0 2021-01-15T18:35:34.475 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:29:43.351242+0000 96'600 96:663 [1] 1 [1] 1 0'0 2021-01-15T18:29:03.219749+0000 0'0 2021-01-15T18:29:03.219749+0000 0 2021-01-15T18:35:34.476 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:35:34.476 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:35:34.477 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:35:34.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: ERRORS=0 2021-01-15T18:35:34.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph osd set norecover 2021-01-15T18:35:35.335 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:35:35.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ceph osd set noout 2021-01-15T18:35:36.104 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T18:35:36.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:153: TEST_recovery_priority: ceph osd pool set test7 size 2 2021-01-15T18:35:36.841 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 7 size to 2 2021-01-15T18:35:36.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:154: TEST_recovery_priority: sleep 2 2021-01-15T18:35:38.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:35:38.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:35:38.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:35:38.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:35:38.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:35:38.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:35:38.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:35:38.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:35:38.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:155: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:35:38.962 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:35:38.962 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:35:38.962 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:38.963 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:38.963 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:35:38.963 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:35:38.963 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:35:38.964 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:35:38.964 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:35:38.964 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:35:38.964 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:38.964 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:35:38.965 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:35:38.965 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:35:38.965 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:38.965 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:38.966 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:35:38.966 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:35:38.966 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:38.966 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:35:38.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: seq 1 10 2021-01-15T18:35:38.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-01-15T18:35:38.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-01-15T18:35:38.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-01-15T18:35:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: '[' 1 = 10 ']' 2021-01-15T18:35:39.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:167: TEST_recovery_priority: sleep 2 2021-01-15T18:35:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-01-15T18:35:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-01-15T18:35:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:160: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-01-15T18:35:41.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: break 2021-01-15T18:35:41.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:169: TEST_recovery_priority: flush_pg_stats 2021-01-15T18:35:41.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:35:41.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:35:42.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:35:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:35:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:35:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:35:42.142 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:35:42.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:35:42.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:42.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:35:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836574 2021-01-15T18:35:42.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836574 2021-01-15T18:35:42.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836574' 2021-01-15T18:35:42.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:42.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:35:42.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673052 2021-01-15T18:35:42.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673052 2021-01-15T18:35:42.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836574 1-42949673052' 2021-01-15T18:35:42.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:42.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:35:42.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509531 2021-01-15T18:35:42.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509531 2021-01-15T18:35:42.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509531' 2021-01-15T18:35:42.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:42.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:35:42.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378713 2021-01-15T18:35:42.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378713 2021-01-15T18:35:42.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509531 3-81604378713' 2021-01-15T18:35:42.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:42.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:35:42.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215192 2021-01-15T18:35:42.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215192 2021-01-15T18:35:42.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836574 1-42949673052 2-64424509531 3-81604378713 4-103079215192' 2021-01-15T18:35:42.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:42.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836574 2021-01-15T18:35:42.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:42.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:35:42.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836574 2021-01-15T18:35:42.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:42.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836574 2021-01-15T18:35:42.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836574' 2021-01-15T18:35:42.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:35:42.835 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836574 2021-01-15T18:35:43.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836572 -lt 21474836574 2021-01-15T18:35:43.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:35:44.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:35:44.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:35:44.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836574 -lt 21474836574 2021-01-15T18:35:44.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:44.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673052 2021-01-15T18:35:44.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:44.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:35:44.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673052 2021-01-15T18:35:44.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:44.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673052 2021-01-15T18:35:44.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673052' 2021-01-15T18:35:44.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:35:44.624 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673052 2021-01-15T18:35:45.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673052 -lt 42949673052 2021-01-15T18:35:45.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:45.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509531 2021-01-15T18:35:45.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:45.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:35:45.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509531 2021-01-15T18:35:45.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:45.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509531 2021-01-15T18:35:45.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2021-01-15T18:35:45.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:35:45.028 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509531 2021-01-15T18:35:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509531 -lt 64424509531 2021-01-15T18:35:45.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:45.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378713 2021-01-15T18:35:45.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:45.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:35:45.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378713 2021-01-15T18:35:45.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:45.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378713 2021-01-15T18:35:45.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378713' 2021-01-15T18:35:45.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:35:45.433 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378713 2021-01-15T18:35:45.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378714 -lt 81604378713 2021-01-15T18:35:45.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:45.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215192 2021-01-15T18:35:45.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:45.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:35:45.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215192 2021-01-15T18:35:45.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:45.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215192 2021-01-15T18:35:45.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215192' 2021-01-15T18:35:45.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:35:45.841 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215192 2021-01-15T18:35:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215193 -lt 103079215192 2021-01-15T18:35:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:35:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:35:46.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:35:46.230 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:35:46.231 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:35:46.231 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:35:46.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:35:46.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:35:46.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:35:46.330 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:35:46.331 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:35:46.331 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:46.332 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:46.332 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:35:46.332 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:35:46.333 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:35:46.333 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:35:46.334 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:35:46.334 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:35:46.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:46.335 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:35:46.335 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:35:46.335 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:35:46.336 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:46.336 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:46.336 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:35:46.337 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:35:46.337 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:46.338 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:35:46.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: ceph osd out osd.0 2021-01-15T18:35:46.868 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:35:46.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: sleep 2 2021-01-15T18:35:48.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:174: TEST_recovery_priority: flush_pg_stats 2021-01-15T18:35:48.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:35:48.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:35:49.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:35:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:35:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:35:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:35:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:35:49.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:35:49.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:49.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:35:49.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836577 2021-01-15T18:35:49.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836577 2021-01-15T18:35:49.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836577' 2021-01-15T18:35:49.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:49.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:35:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673056 2021-01-15T18:35:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673056 2021-01-15T18:35:49.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836577 1-42949673056' 2021-01-15T18:35:49.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:49.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:35:49.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509534 2021-01-15T18:35:49.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509534 2021-01-15T18:35:49.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836577 1-42949673056 2-64424509534' 2021-01-15T18:35:49.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:49.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:35:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378716 2021-01-15T18:35:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378716 2021-01-15T18:35:49.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836577 1-42949673056 2-64424509534 3-81604378716' 2021-01-15T18:35:49.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:49.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:35:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215196 2021-01-15T18:35:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215196 2021-01-15T18:35:49.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836577 1-42949673056 2-64424509534 3-81604378716 4-103079215196' 2021-01-15T18:35:49.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:49.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836577 2021-01-15T18:35:49.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:49.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:35:49.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836577 2021-01-15T18:35:49.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836577 2021-01-15T18:35:49.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2021-01-15T18:35:49.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:35:49.997 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836577 2021-01-15T18:35:50.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836577 -lt 21474836577 2021-01-15T18:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673056 2021-01-15T18:35:50.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:50.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:35:50.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673056 2021-01-15T18:35:50.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:50.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673056 2021-01-15T18:35:50.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673056' 2021-01-15T18:35:50.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:35:50.399 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673056 2021-01-15T18:35:50.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673054 -lt 42949673056 2021-01-15T18:35:50.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:35:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:35:51.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:35:52.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673056 -lt 42949673056 2021-01-15T18:35:52.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:52.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509534 2021-01-15T18:35:52.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:52.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:35:52.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509534 2021-01-15T18:35:52.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:52.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509534 2021-01-15T18:35:52.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509534' 2021-01-15T18:35:52.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:35:52.191 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509534 2021-01-15T18:35:52.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509534 -lt 64424509534 2021-01-15T18:35:52.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:52.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378716 2021-01-15T18:35:52.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:52.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:35:52.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378716 2021-01-15T18:35:52.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:52.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378716 2021-01-15T18:35:52.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378716' 2021-01-15T18:35:52.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:35:52.584 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378716 2021-01-15T18:35:52.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378717 -lt 81604378716 2021-01-15T18:35:52.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:52.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215196 2021-01-15T18:35:52.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:52.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:35:52.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215196 2021-01-15T18:35:52.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:52.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215196 2021-01-15T18:35:52.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215196' 2021-01-15T18:35:52.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:35:52.981 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215196 2021-01-15T18:35:53.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215196 -lt 103079215196 2021-01-15T18:35:53.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:35:53.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:35:53.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:35:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:35:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:35:53.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:35:53.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:35:53.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:35:53.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:35:53.483 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:35:53.483 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:35:53.483 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:53.484 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:53.484 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:35:53.484 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:35:53.484 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:35:53.485 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:35:53.485 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:35:53.485 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:35:53.486 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:35:53.486 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:35:53.486 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:53.486 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:35:53.487 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:53.487 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:35:53.487 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:35:53.487 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:35:53.487 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:35:53.488 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:35:53.488 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:35:53.488 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:53.488 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:35:53.488 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:35:53.489 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:35:53.489 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:35:53.489 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:35:53.489 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:35:53.489 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:35:53.490 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:35:53.490 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:35:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: ceph pg dump pgs 2021-01-15T18:35:53.860 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:35:53.861 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-01-15T18:35:41.630386+0000 96'600 107:660 [1,4] 1 [1] 1 0'0 2021-01-15T18:29:13.919535+0000 0'0 2021-01-15T18:29:13.919535+0000 0 2021-01-15T18:35:53.861 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-01-15T18:35:48.275004+0000 96'600 108:680 [1,2] 1 [1] 1 0'0 2021-01-15T18:28:58.110760+0000 0'0 2021-01-15T18:28:58.110760+0000 0 2021-01-15T18:35:53.861 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:29:43.351242+0000 96'600 107:672 [1] 1 [1] 1 0'0 2021-01-15T18:29:03.219749+0000 0'0 2021-01-15T18:29:03.219749+0000 0 2021-01-15T18:35:53.861 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:35:53.862 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:35:53.862 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:35:53.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph osd pool set test3 size 2 2021-01-15T18:35:54.567 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 3 size to 2 2021-01-15T18:35:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: sleep 2 2021-01-15T18:35:56.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:180: TEST_recovery_priority: flush_pg_stats 2021-01-15T18:35:56.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:35:56.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:35:56.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:35:56.983 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:35:56.983 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:35:56.983 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:35:56.984 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:35:56.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:35:56.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:56.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:35:57.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836581 2021-01-15T18:35:57.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836581 2021-01-15T18:35:57.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836581' 2021-01-15T18:35:57.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:57.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:35:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673059 2021-01-15T18:35:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673059 2021-01-15T18:35:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836581 1-42949673059' 2021-01-15T18:35:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:57.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:35:57.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509538 2021-01-15T18:35:57.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509538 2021-01-15T18:35:57.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509538' 2021-01-15T18:35:57.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:57.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:35:57.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378720 2021-01-15T18:35:57.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378720 2021-01-15T18:35:57.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509538 3-81604378720' 2021-01-15T18:35:57.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:35:57.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:35:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215199 2021-01-15T18:35:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215199 2021-01-15T18:35:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509538 3-81604378720 4-103079215199' 2021-01-15T18:35:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:57.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836581 2021-01-15T18:35:57.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:57.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:35:57.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836581 2021-01-15T18:35:57.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:57.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836581 2021-01-15T18:35:57.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2021-01-15T18:35:57.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:35:57.686 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836581 2021-01-15T18:35:58.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836581 -lt 21474836581 2021-01-15T18:35:58.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:58.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673059 2021-01-15T18:35:58.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:58.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:35:58.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673059 2021-01-15T18:35:58.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:58.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673059 2021-01-15T18:35:58.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2021-01-15T18:35:58.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:35:58.086 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673059 2021-01-15T18:35:58.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673059 -lt 42949673059 2021-01-15T18:35:58.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:58.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509538 2021-01-15T18:35:58.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:58.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:35:58.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509538 2021-01-15T18:35:58.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:58.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509538 2021-01-15T18:35:58.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509538' 2021-01-15T18:35:58.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:35:58.485 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509538 2021-01-15T18:35:58.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509538 -lt 64424509538 2021-01-15T18:35:58.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:35:58.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378720 2021-01-15T18:35:58.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:35:58.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:35:58.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378720 2021-01-15T18:35:58.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:35:58.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378720 2021-01-15T18:35:58.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378720' 2021-01-15T18:35:58.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:35:58.891 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378720 2021-01-15T18:35:59.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378718 -lt 81604378720 2021-01-15T18:35:59.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:36:00.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:36:00.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:36:00.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378720 -lt 81604378720 2021-01-15T18:36:00.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:00.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215199 2021-01-15T18:36:00.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:00.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:36:00.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215199 2021-01-15T18:36:00.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215199 2021-01-15T18:36:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215199' 2021-01-15T18:36:00.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:36:00.690 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215199 2021-01-15T18:36:01.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215200 -lt 103079215199 2021-01-15T18:36:01.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:01.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:01.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:01.079 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:01.080 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:01.080 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:01.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:01.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:36:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:01.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:01.191 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:01.192 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:01.192 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:01.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:01.192 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:01.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:01.193 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:01.193 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:01.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:01.193 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:01.194 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:01.194 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:01.194 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:01.194 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:01.194 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:01.195 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:01.195 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:01.195 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:01.195 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:01.195 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:01.196 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:01.196 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:01.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:01.196 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:01.196 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:01.197 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:01.197 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:01.197 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:01.197 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:01.197 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:01.198 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:01.198 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:01.198 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:01.198 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:01.198 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:01.199 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:01.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: ceph pg dump pgs 2021-01-15T18:36:01.555 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:36:01.555 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovering+forced_recovery+undersized+degraded+remapped 2021-01-15T18:35:41.630386+0000 96'600 112:664 [1,4] 1 [1] 1 0'0 2021-01-15T18:29:13.919535+0000 0'0 2021-01-15T18:29:13.919535+0000 0 2021-01-15T18:36:01.555 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-01-15T18:35:48.275004+0000 96'600 112:684 [1,2] 1 [1] 1 0'0 2021-01-15T18:28:58.110760+0000 0'0 2021-01-15T18:28:58.110760+0000 0 2021-01-15T18:36:01.555 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 600 active+recovery_wait+undersized+degraded+remapped 2021-01-15T18:35:55.806854+0000 96'600 113:683 [1,2] 1 [1] 1 0'0 2021-01-15T18:29:03.219749+0000 0'0 2021-01-15T18:29:03.219749+0000 0 2021-01-15T18:36:01.556 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:36:01.556 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:36:01.557 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:36:01.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:01.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-01-15T18:36:01.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: PRIO=190 2021-01-15T18:36:01.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-01-15T18:36:01.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:01.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T18:36:01.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: eval 'ITEM="7.0"' 2021-01-15T18:36:01.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:193: TEST_recovery_priority: ITEM=7.0 2021-01-15T18:36:01.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:194: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2021-01-15T18:36:01.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:01.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T18:36:01.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:199: TEST_recovery_priority: PRIO=255 2021-01-15T18:36:01.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:200: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-01-15T18:36:01.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: seq 1 10 2021-01-15T18:36:01.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:208: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2021-01-15T18:36:01.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: ceph pg force-recovery 3.0 2021-01-15T18:36:01.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:210: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2021-01-15T18:36:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: break 2021-01-15T18:36:01.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:219: TEST_recovery_priority: sleep 2 2021-01-15T18:36:03.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:03.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:03.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:03.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:03.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:03.960 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:03.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:03.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:36:03.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:220: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:04.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:221: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:04.079 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:04.080 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:04.080 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:04.080 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:04.080 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:04.080 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:04.081 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:04.081 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:04.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:04.081 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:04.081 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:04.082 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:04.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:04.082 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:04.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:04.082 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:04.083 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 255, 2021-01-15T18:36:04.083 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:04.083 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:04.083 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:04.083 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:04.084 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:04.084 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:04.084 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:04.084 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:04.084 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:04.085 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:04.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:04.085 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:04.085 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:04.086 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:04.086 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:04.086 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:04.086 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:04.087 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:04.087 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:04.087 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:04.087 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:04.087 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:04.088 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:04.088 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:04.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:04.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-01-15T18:36:04.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: PRIO=255 2021-01-15T18:36:04.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-01-15T18:36:04.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:228: TEST_recovery_priority: flush_pg_stats 2021-01-15T18:36:04.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:36:04.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:36:04.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:36:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:36:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:36:04.474 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:36:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:36:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:36:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:04.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:36:04.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836584 2021-01-15T18:36:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836584 2021-01-15T18:36:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836584' 2021-01-15T18:36:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:04.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:36:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673063 2021-01-15T18:36:04.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673063 2021-01-15T18:36:04.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836584 1-42949673063' 2021-01-15T18:36:04.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:04.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:36:04.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509541 2021-01-15T18:36:04.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509541 2021-01-15T18:36:04.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836584 1-42949673063 2-64424509541' 2021-01-15T18:36:04.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:04.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:36:05.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378723 2021-01-15T18:36:05.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378723 2021-01-15T18:36:05.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836584 1-42949673063 2-64424509541 3-81604378723' 2021-01-15T18:36:05.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:05.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:36:05.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215203 2021-01-15T18:36:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215203 2021-01-15T18:36:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836584 1-42949673063 2-64424509541 3-81604378723 4-103079215203' 2021-01-15T18:36:05.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:05.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836584 2021-01-15T18:36:05.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:36:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836584 2021-01-15T18:36:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836584 2021-01-15T18:36:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836584' 2021-01-15T18:36:05.166 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836584 2021-01-15T18:36:05.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:36:05.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836582 -lt 21474836584 2021-01-15T18:36:05.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:36:06.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:36:06.555 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:36:06.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836584 -lt 21474836584 2021-01-15T18:36:06.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:06.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673063 2021-01-15T18:36:06.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:06.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:36:06.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673063 2021-01-15T18:36:06.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:06.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673063 2021-01-15T18:36:06.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2021-01-15T18:36:06.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:36:06.958 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673063 2021-01-15T18:36:07.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673063 -lt 42949673063 2021-01-15T18:36:07.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:07.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509541 2021-01-15T18:36:07.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:07.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:36:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509541 2021-01-15T18:36:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:07.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509541 2021-01-15T18:36:07.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2021-01-15T18:36:07.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:36:07.371 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509541 2021-01-15T18:36:07.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509542 -lt 64424509541 2021-01-15T18:36:07.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:07.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378723 2021-01-15T18:36:07.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:07.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:36:07.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378723 2021-01-15T18:36:07.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378723 2021-01-15T18:36:07.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378723' 2021-01-15T18:36:07.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:36:07.778 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378723 2021-01-15T18:36:08.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378724 -lt 81604378723 2021-01-15T18:36:08.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:08.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215203 2021-01-15T18:36:08.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:08.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:36:08.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215203 2021-01-15T18:36:08.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215203 2021-01-15T18:36:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215203' 2021-01-15T18:36:08.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:36:08.168 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215203 2021-01-15T18:36:08.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215203 -lt 103079215203 2021-01-15T18:36:08.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:232: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2021-01-15T18:36:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2021-01-15T18:36:08.950 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T18:36:08.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:233: TEST_recovery_priority: sleep 2 2021-01-15T18:36:10.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:10.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:10.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:10.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:10.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:10.963 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:10.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:10.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:36:10.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:11.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:11.083 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:11.083 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:11.083 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:11.083 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:11.083 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:11.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:11.084 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:11.084 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:11.084 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:11.084 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:11.085 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:11.085 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:11.085 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:11.085 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:11.085 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:11.086 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:11.086 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:11.086 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:11.086 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:11.086 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:11.087 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:11.087 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:11.087 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:11.087 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:11.087 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:11.088 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:11.088 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:11.088 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:11.088 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:11.088 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:11.089 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:11.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:11.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-01-15T18:36:11.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:237: TEST_recovery_priority: PRIO=190 2021-01-15T18:36:11.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-01-15T18:36:11.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:11.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T18:36:11.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: eval 'ITEM="3.0"' 2021-01-15T18:36:11.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:244: TEST_recovery_priority: ITEM=3.0 2021-01-15T18:36:11.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:245: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2021-01-15T18:36:11.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:11.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T18:36:11.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:250: TEST_recovery_priority: PRIO=255 2021-01-15T18:36:11.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:251: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-01-15T18:36:11.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:258: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2021-01-15T18:36:11.462 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2021-01-15T18:36:11.462 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T18:36:11.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:259: TEST_recovery_priority: sleep 5 2021-01-15T18:36:16.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:16.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:16.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:16.475 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:16.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:16.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:36:16.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:260: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:16.584 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:16.584 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:16.584 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:16.585 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:16.585 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:16.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:16.585 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:16.585 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:16.586 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:16.586 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:16.586 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:16.586 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:16.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:16.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:16.587 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:16.587 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:16.587 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:16.587 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:16.588 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:16.588 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:16.588 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:16.588 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:16.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:16.589 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:16.590 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:16.590 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:16.590 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:16.590 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:16.590 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:16.591 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:16.591 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:16.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: flush_pg_stats 2021-01-15T18:36:16.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:36:16.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:36:16.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:36:16.981 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:36:16.981 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:36:16.982 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:36:16.982 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:36:16.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:36:16.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:16.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:36:17.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836589 2021-01-15T18:36:17.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836589 2021-01-15T18:36:17.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836589' 2021-01-15T18:36:17.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:17.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:36:17.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673067 2021-01-15T18:36:17.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673067 2021-01-15T18:36:17.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836589 1-42949673067' 2021-01-15T18:36:17.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:17.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:36:17.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509546 2021-01-15T18:36:17.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509546 2021-01-15T18:36:17.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836589 1-42949673067 2-64424509546' 2021-01-15T18:36:17.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:17.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:36:17.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378728 2021-01-15T18:36:17.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378728 2021-01-15T18:36:17.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836589 1-42949673067 2-64424509546 3-81604378728' 2021-01-15T18:36:17.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:17.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:36:17.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215207 2021-01-15T18:36:17.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215207 2021-01-15T18:36:17.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836589 1-42949673067 2-64424509546 3-81604378728 4-103079215207' 2021-01-15T18:36:17.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:17.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836589 2021-01-15T18:36:17.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:17.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:36:17.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836589 2021-01-15T18:36:17.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:17.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836589 2021-01-15T18:36:17.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2021-01-15T18:36:17.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:36:17.696 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836589 2021-01-15T18:36:18.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836589 -lt 21474836589 2021-01-15T18:36:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:18.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673067 2021-01-15T18:36:18.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:36:18.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673067 2021-01-15T18:36:18.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:18.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673067 2021-01-15T18:36:18.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673067' 2021-01-15T18:36:18.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:36:18.089 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673067 2021-01-15T18:36:18.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673067 -lt 42949673067 2021-01-15T18:36:18.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:18.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509546 2021-01-15T18:36:18.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:18.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:36:18.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509546 2021-01-15T18:36:18.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:18.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509546 2021-01-15T18:36:18.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509546' 2021-01-15T18:36:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:36:18.508 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509546 2021-01-15T18:36:18.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509546 -lt 64424509546 2021-01-15T18:36:18.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:18.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378728 2021-01-15T18:36:18.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:18.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:36:18.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378728 2021-01-15T18:36:18.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:18.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378728 2021-01-15T18:36:18.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378728' 2021-01-15T18:36:18.939 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:36:18.940 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378728 2021-01-15T18:36:19.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378726 -lt 81604378728 2021-01-15T18:36:19.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:36:20.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:36:20.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:36:20.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378728 -lt 81604378728 2021-01-15T18:36:20.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:20.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215207 2021-01-15T18:36:20.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:20.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:36:20.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215207 2021-01-15T18:36:20.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:20.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215207 2021-01-15T18:36:20.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215207' 2021-01-15T18:36:20.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:36:20.767 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215207 2021-01-15T18:36:21.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215208 -lt 103079215207 2021-01-15T18:36:21.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:264: TEST_recovery_priority: ceph pg force-recovery 7.0 2021-01-15T18:36:21.555 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2021-01-15T18:36:21.555 INFO:tasks.workunit.client.0.smithi145.stderr: 2021-01-15T18:36:21.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:265: TEST_recovery_priority: sleep 2 2021-01-15T18:36:23.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:23.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:23.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:23.567 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:23.568 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:23.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:23.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:36:23.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:23.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:23.688 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:23.688 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:23.688 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:23.688 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:23.688 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:23.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:23.689 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:23.689 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:23.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:23.689 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:23.690 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:23.690 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:23.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:23.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:23.690 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:23.691 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:23.691 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:23.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:23.691 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:23.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:23.692 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:23.692 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:23.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:23.692 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:23.692 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:23.693 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:23.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:23.693 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:23.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:23.693 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:23.694 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:23.694 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:23.694 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:23.694 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:23.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:23.695 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:23.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:23.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-01-15T18:36:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:269: TEST_recovery_priority: PRIO=190 2021-01-15T18:36:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2021-01-15T18:36:23.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:23.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2021-01-15T18:36:23.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: eval 'ITEM="7.0"' 2021-01-15T18:36:23.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:276: TEST_recovery_priority: ITEM=7.0 2021-01-15T18:36:23.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:277: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2021-01-15T18:36:23.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: cat td/osd-recovery-prio/out 2021-01-15T18:36:23.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2021-01-15T18:36:23.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:282: TEST_recovery_priority: PRIO=255 2021-01-15T18:36:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:283: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2021-01-15T18:36:23.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:290: TEST_recovery_priority: ceph osd unset noout 2021-01-15T18:36:24.401 INFO:tasks.workunit.client.0.smithi145.stderr:noout is unset 2021-01-15T18:36:24.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:291: TEST_recovery_priority: ceph osd unset norecover 2021-01-15T18:36:25.366 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:36:25.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:36:25.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:36:25.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:36:25.377 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:36:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:36:25.378 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:36:25.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:36:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations' 2021-01-15T18:36:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations' 2021-01-15T18:36:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:36:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:36:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:36:25.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:36:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:36:25.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:36:25.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:36:25.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:36:25.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:36:25.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:36:25.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:36:25.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:36:25.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:36:25.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:36:26.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:36:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:36:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:36:26.021 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:36:26.022 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:36:26.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:36:26.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:26.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:36:26.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836593 2021-01-15T18:36:26.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836593 2021-01-15T18:36:26.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593' 2021-01-15T18:36:26.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:26.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:36:26.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949673071 2021-01-15T18:36:26.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949673071 2021-01-15T18:36:26.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071' 2021-01-15T18:36:26.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:26.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:36:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509549 2021-01-15T18:36:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509549 2021-01-15T18:36:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549' 2021-01-15T18:36:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:36:26.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378732 2021-01-15T18:36:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378732 2021-01-15T18:36:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549 3-81604378732' 2021-01-15T18:36:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:36:26.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:36:26.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215211 2021-01-15T18:36:26.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215211 2021-01-15T18:36:26.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509549 3-81604378732 4-103079215211' 2021-01-15T18:36:26.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:26.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836593 2021-01-15T18:36:26.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:26.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:36:26.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836593 2021-01-15T18:36:26.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:26.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836593 2021-01-15T18:36:26.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2021-01-15T18:36:26.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:36:26.764 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836593 2021-01-15T18:36:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836590 -lt 21474836593 2021-01-15T18:36:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:36:28.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:36:28.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:36:28.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836593 -lt 21474836593 2021-01-15T18:36:28.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:28.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949673071 2021-01-15T18:36:28.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:28.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:36:28.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949673071 2021-01-15T18:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:28.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949673071 2021-01-15T18:36:28.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2021-01-15T18:36:28.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:36:28.881 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949673071 2021-01-15T18:36:29.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949673071 -lt 42949673071 2021-01-15T18:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509549 2021-01-15T18:36:29.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:29.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:36:29.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509549 2021-01-15T18:36:29.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:29.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509549 2021-01-15T18:36:29.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509549' 2021-01-15T18:36:29.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:36:29.303 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509549 2021-01-15T18:36:29.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509550 -lt 64424509549 2021-01-15T18:36:29.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:29.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378732 2021-01-15T18:36:29.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:29.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:36:29.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378732 2021-01-15T18:36:29.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:29.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378732 2021-01-15T18:36:29.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378732' 2021-01-15T18:36:29.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:36:29.773 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378732 2021-01-15T18:36:30.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378732 -lt 81604378732 2021-01-15T18:36:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:36:30.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215211 2021-01-15T18:36:30.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:36:30.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:36:30.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215211 2021-01-15T18:36:30.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:36:30.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215211 2021-01-15T18:36:30.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215211' 2021-01-15T18:36:30.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:36:30.220 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215211 2021-01-15T18:36:30.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215212 -lt 103079215211 2021-01-15T18:36:30.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:36:30.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:30.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 3 == 0 2021-01-15T18:36:31.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:31.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:31.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:31.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:31.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:32.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:32.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:32.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:32.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:32.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:36:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:36:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:36:32.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:32.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:32.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:32.681 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:32.682 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:32.683 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:32.684 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:32.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:32.686 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:32.687 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:32.688 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:32.689 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:32.690 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:32.691 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:32.692 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:32.693 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:32.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:32.694 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:32.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:32.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:32.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:32.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:32.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:32.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:32.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:32.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:32.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:33.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:33.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:33.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:33.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:33.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:33.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:33.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:33.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8513009 2021-01-15T18:36:34.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8513009 '!=' null 2021-01-15T18:36:34.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:34.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:34.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:34.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:34.802 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:34.802 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:34.802 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:34.803 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:34.803 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:34.803 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:34.803 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:34.804 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:34.805 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:34.805 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:34.805 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:34.805 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:34.805 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:34.806 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:34.806 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:34.806 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:34.806 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:34.806 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:34.807 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:34.807 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:34.808 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:34.809 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:34.809 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:34.810 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:34.810 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:34.810 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:34.810 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:34.811 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:34.811 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:34.811 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:34.811 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:34.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:34.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:34.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:34.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:34.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:34.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:34.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:34.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:34.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:35.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:35.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:35.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:35.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:35.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:35.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:35.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:35.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:35.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:35.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:35.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:35.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18623682 2021-01-15T18:36:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18623682 '!=' null 2021-01-15T18:36:36.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:36.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:36.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:36.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:36.552 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:36.553 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:36.554 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:36.555 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:36.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:36.556 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:36.556 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:36.556 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:36.556 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:36.557 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:36.557 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:36.557 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:36.557 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:36.557 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:36.558 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:36.558 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:36.558 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:36.558 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:36.559 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:36.559 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:36.559 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:36.559 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:36.559 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:36.560 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:36.560 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:36.560 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:36.560 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:36.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:36.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:36.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:36.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:36.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:36.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:37.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:37.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:37.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:37.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:37.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:37.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:37.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:37.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:37.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:37.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:37.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:37.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:37.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:38.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725289 2021-01-15T18:36:38.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725289 '!=' null 2021-01-15T18:36:38.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:38.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:38.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:38.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:38.515 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:38.515 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:38.516 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:38.516 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:38.516 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:38.516 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:38.516 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:38.517 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:38.517 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:38.517 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:38.517 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:38.517 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:38.518 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:38.518 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:38.518 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:38.518 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:38.519 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:38.519 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:38.519 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:38.519 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:38.519 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:38.520 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:38.520 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:38.520 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:38.521 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:38.521 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:38.521 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:38.521 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:38.521 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:38.522 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:38.522 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:38.522 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:38.522 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:38.522 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:38.523 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:38.523 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:38.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:38.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:38.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:38.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:38.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:38.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:38.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:38.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:38.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:39.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:39.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:39.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:39.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:39.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:39.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:39.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:39.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:39.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:39.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:39.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:39.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:39.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:40.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840963 2021-01-15T18:36:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840963 '!=' null 2021-01-15T18:36:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:40.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:40.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:40.217 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:40.217 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:40.218 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:40.218 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:40.218 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:40.218 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:40.218 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:40.219 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:40.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:40.219 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:40.219 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:40.219 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:40.220 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:40.220 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:40.220 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:40.220 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:40.220 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:40.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:40.221 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:40.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:40.221 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:40.222 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:40.222 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:40.222 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:40.222 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:40.222 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:40.223 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:40.224 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:40.224 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:40.224 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:40.224 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:40.224 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:40.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:40.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:40.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:40.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:40.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:40.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:40.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:40.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:40.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:40.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:40.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:40.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:40.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:41.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:41.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:41.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:41.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:41.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:41.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:41.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:41.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:41.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:41.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599289 2021-01-15T18:36:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599289 '!=' null 2021-01-15T18:36:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:41.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:41.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:41.897 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:41.897 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:41.897 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:41.897 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:41.898 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:41.898 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:41.898 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:41.898 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:41.899 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:41.900 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:41.901 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:41.902 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:41.903 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:41.904 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:41.904 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:41.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:42.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:42.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:42.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:42.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:42.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:42.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:42.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:42.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:42.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:42.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:42.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:42.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:42.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:42.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:42.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:42.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:42.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:42.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:42.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:42.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:43.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599289 2021-01-15T18:36:43.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599289 '!=' null 2021-01-15T18:36:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:43.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:43.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:43.984 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:43.984 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:43.985 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:43.985 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:43.985 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:43.985 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:43.985 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:43.986 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:43.986 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:43.986 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:43.986 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:43.987 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:43.987 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:43.987 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:43.987 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:43.987 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:43.988 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:43.988 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:43.988 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:43.988 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:43.988 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:43.989 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:43.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:43.989 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:43.989 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:43.989 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:43.990 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:43.990 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:43.990 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:43.991 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:43.991 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:43.991 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:43.991 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:43.991 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:43.992 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:43.994 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:43.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:44.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:44.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:44.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:44.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:44.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:44.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:44.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:44.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:44.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:44.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:44.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:44.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:45.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:45.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:45.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:45.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:45.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:45.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:45.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:45.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:45.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599208 2021-01-15T18:36:45.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599208 '!=' null 2021-01-15T18:36:45.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:45.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:45.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:45.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:45.629 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:45.629 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:45.629 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:45.630 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:45.630 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:45.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:45.630 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:45.630 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:45.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:45.631 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:45.631 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:45.631 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:45.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:45.632 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:45.632 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:45.632 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:45.633 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:45.633 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:45.633 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:45.633 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:45.634 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:45.634 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:45.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:45.634 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:45.634 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:45.635 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:45.635 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:45.635 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:45.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:45.635 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:45.636 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:45.636 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:45.636 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:45.636 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:45.636 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:45.637 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:45.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:45.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:45.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:45.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:46.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:46.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:46.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:46.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:46.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:46.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:46.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:46.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:46.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:46.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:46.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:46.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:46.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:47.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335614 2021-01-15T18:36:47.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335614 '!=' null 2021-01-15T18:36:47.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:47.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:47.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:47.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:47.496 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:47.496 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:47.497 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:47.497 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:47.497 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:47.497 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:47.498 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:47.498 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:47.498 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:47.499 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:47.499 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:47.499 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:47.499 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:47.500 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:47.500 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:47.500 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:47.500 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:47.500 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:47.501 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:47.502 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:47.502 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:47.502 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:47.502 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:47.503 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:47.504 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:47.504 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:47.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:47.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:47.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:47.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:47.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:47.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:47.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:47.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:47.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:47.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:47.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:47.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:47.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:48.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:48.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:48.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:48.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:48.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:48.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:48.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:48.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:48.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:49.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725676 2021-01-15T18:36:49.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725676 '!=' null 2021-01-15T18:36:49.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:49.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:49.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:49.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:49.195 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:49.195 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:49.195 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:49.196 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:49.196 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:49.196 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:49.196 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:49.196 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:49.197 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:49.197 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:49.197 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:49.197 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:49.197 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:49.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:49.198 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:49.198 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:49.198 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:49.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:49.199 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:49.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:49.199 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:49.199 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:49.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:49.200 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:49.200 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:49.200 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:49.200 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:49.201 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:49.201 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:49.201 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:49.201 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:49.202 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:49.202 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:49.203 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:49.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:49.203 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:49.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:49.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:49.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:49.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:49.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:49.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:49.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:49.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:49.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:49.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:49.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:49.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:49.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:50.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:50.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:50.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:50.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:50.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:50.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:50.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:50.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:50.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=25335692 2021-01-15T18:36:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 25335692 '!=' null 2021-01-15T18:36:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:50.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:50.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:50.953 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:50.953 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:50.954 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:50.954 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:50.954 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:50.954 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:50.954 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:50.955 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:50.955 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:50.955 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:50.955 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:50.955 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:50.956 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:50.956 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:50.956 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:50.956 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:50.956 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:50.957 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:50.957 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:50.957 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:50.957 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:50.957 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:50.958 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:50.958 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:50.958 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:50.958 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:50.958 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:50.959 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:50.959 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:50.959 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:50.959 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:50.959 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:50.960 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:50.960 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:50.960 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:50.960 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:51.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:51.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:51.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:51.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:51.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:51.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:51.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:51.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:51.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:51.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:51.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:51.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:51.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:51.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:51.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:51.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:51.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:51.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:51.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:51.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:51.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:52.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346457 2021-01-15T18:36:52.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346457 '!=' null 2021-01-15T18:36:52.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:52.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:52.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:52.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:52.794 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:52.795 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:52.795 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:52.795 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:52.795 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:52.796 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:52.796 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:52.796 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:52.796 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:52.796 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:52.797 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:52.797 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:52.797 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:52.797 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:52.797 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:52.798 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:52.798 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:52.798 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:52.798 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:52.799 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:52.799 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:52.799 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:52.799 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:52.799 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:52.800 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:52.800 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:52.800 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:52.800 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:52.800 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:52.801 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:52.801 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:52.801 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:52.801 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:52.801 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:52.802 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:52.802 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:52.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:52.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:52.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:52.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:52.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:52.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:52.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:52.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:53.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:53.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:53.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:53.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:53.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:53.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:53.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:53.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:53.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:53.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:53.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:53.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:53.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346448 2021-01-15T18:36:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346448 '!=' null 2021-01-15T18:36:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:54.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:54.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:54.476 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:54.477 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:54.477 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:54.477 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:54.477 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:54.477 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:54.478 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:54.478 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:54.478 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:54.478 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:54.478 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:54.479 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:54.479 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:54.479 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:54.479 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:54.479 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:54.480 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:54.480 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:54.480 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:54.480 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:54.481 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:54.481 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:54.481 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:54.481 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:54.482 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:54.482 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:54.482 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:54.482 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:54.482 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:54.483 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:54.483 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:54.483 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:54.483 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:54.483 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:54.484 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:54.484 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:54.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:54.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:54.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:54.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:54.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:54.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:54.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:54.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:54.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:54.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:54.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:54.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:55.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:55.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:55.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:55.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:55.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:56.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461968 2021-01-15T18:36:56.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461968 '!=' null 2021-01-15T18:36:56.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:56.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:56.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:56.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:56.577 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:56.577 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:56.578 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:56.578 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:56.578 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:56.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:56.578 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:56.579 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:56.579 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:56.579 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:56.580 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:56.580 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:56.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:56.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:56.580 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:56.581 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:56.581 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:56.581 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:56.581 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:56.581 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:56.582 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:56.582 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:56.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:56.582 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:56.582 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:56.583 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:56.583 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:56.583 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:56.583 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:56.583 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:56.584 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:56.584 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:56.584 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:56.584 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:56.584 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:56.585 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:56.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:56.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:56.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:56.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:56.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:56.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:56.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:56.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:56.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:57.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:57.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:57.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:57.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:57.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:57.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:57.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:57.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:57.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:57.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:36:58.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725566 2021-01-15T18:36:58.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725566 '!=' null 2021-01-15T18:36:58.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:36:58.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:58.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:36:58.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:36:58.238 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:36:58.239 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:36:58.239 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:58.239 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:58.239 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:36:58.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:58.240 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:36:58.240 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:36:58.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:58.240 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:36:58.241 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:58.241 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:58.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:58.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:58.241 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:36:58.242 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:36:58.242 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:58.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:58.242 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:58.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:58.243 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:36:58.243 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:36:58.243 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:36:58.243 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:36:58.243 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:36:58.244 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:36:58.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:58.244 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:36:58.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:36:58.244 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:36:58.245 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:36:58.245 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:36:58.245 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:36:58.245 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:36:58.245 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:36:58.246 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:36:58.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:36:58.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:36:58.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:36:58.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:36:58.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:36:58.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:36:58.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:36:58.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:36:58.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:36:58.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:36:58.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:36:58.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:36:58.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:36:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:36:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:36:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:36:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:36:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:36:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:36:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:36:59.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:36:59.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:00.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462092 2021-01-15T18:37:00.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462092 '!=' null 2021-01-15T18:37:00.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:00.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:00.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:00.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:00.192 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:00.192 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:00.192 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:00.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:00.193 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:00.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:00.193 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:00.193 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:00.193 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:00.194 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:00.194 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:00.194 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:00.194 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:00.195 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:00.195 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:00.195 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:00.195 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:00.195 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:00.196 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:00.196 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:00.196 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:00.196 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:00.197 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:00.197 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:00.197 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:00.197 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:00.197 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:00.198 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:00.198 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:00.198 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:00.198 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:00.198 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:00.199 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:00.199 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:00.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:00.199 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:00.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:00.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:00.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:00.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:00.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:00.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:00.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:00.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:00.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:00.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:00.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:00.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:00.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:01.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:01.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:01.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:01.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:01.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:01.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:01.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:01.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:01.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:02.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851923 2021-01-15T18:37:02.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851923 '!=' null 2021-01-15T18:37:02.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:02.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:02.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:02.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:02.312 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:02.312 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:02.312 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:02.312 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:02.313 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:02.313 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:02.313 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:02.313 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:02.313 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:02.314 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:02.314 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:02.314 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:02.314 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:02.315 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:02.316 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:02.316 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:02.316 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:02.316 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:02.317 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:02.317 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:02.317 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:02.317 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:02.317 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:02.318 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:02.318 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:02.318 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:02.318 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:02.318 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:02.319 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:02.319 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:02.319 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:02.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:02.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:02.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:02.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:02.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:02.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:02.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:02.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:02.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:02.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:02.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:02.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:02.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:03.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:03.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:03.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:03.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851954 2021-01-15T18:37:03.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851954 '!=' null 2021-01-15T18:37:03.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:03.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:03.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:03.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:03.942 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:03.942 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:03.942 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:03.943 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:03.943 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:03.943 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:03.943 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:03.944 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:03.944 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:03.944 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:03.944 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:03.944 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:03.945 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:03.945 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:03.945 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:03.945 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:03.945 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:03.946 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:03.946 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:03.946 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:03.946 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:03.947 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:03.947 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:03.947 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:03.947 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:03.947 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:03.948 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:03.948 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:03.948 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:03.949 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:03.949 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:03.949 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:03.949 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:03.949 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:03.950 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:03.950 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:03.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:04.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:04.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:04.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:04.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:04.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:04.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:04.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:04.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:04.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:04.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:04.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:04.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:05.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:05.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:05.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:05.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:05.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:05.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851954 2021-01-15T18:37:05.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851954 '!=' null 2021-01-15T18:37:05.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:05.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:05.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:05.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:05.940 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:05.940 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:05.940 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:05.941 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:05.941 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:05.941 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:05.941 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:05.941 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:05.942 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:05.942 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:05.942 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:05.942 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:05.943 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:05.944 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:05.944 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:05.944 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:05.944 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:05.944 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:05.945 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:05.945 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:05.945 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:05.945 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:05.945 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:05.946 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:05.946 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:05.946 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:05.946 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:05.946 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:05.947 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:05.947 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:05.947 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:05.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:06.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:06.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:06.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:06.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:06.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:06.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:06.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:06.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:06.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:07.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:07.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:07.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:07.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:07.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:07.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714766 2021-01-15T18:37:07.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714766 '!=' null 2021-01-15T18:37:07.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:07.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:07.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:07.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:07.743 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:07.743 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:07.743 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:07.744 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:07.744 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:07.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:07.744 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:07.744 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:07.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:07.745 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:07.745 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:07.745 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:07.745 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:07.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:07.746 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:07.746 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:07.746 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:07.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:07.747 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:07.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:07.747 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:07.747 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:07.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:07.748 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:07.748 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:07.748 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:07.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:07.749 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:07.749 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:07.749 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:07.749 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:07.750 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:07.750 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:07.750 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:07.750 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:07.750 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:07.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:07.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:07.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:07.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:07.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:07.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:07.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:07.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:07.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:08.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:08.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:08.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:08.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:09.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:09.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:09.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:09.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:09.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:09.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:09.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599290 2021-01-15T18:37:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599290 '!=' null 2021-01-15T18:37:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:09.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:09.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:09.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:09.675 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:09.675 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:09.676 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:09.676 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:09.677 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:09.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:09.677 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:09.678 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:09.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:09.678 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:09.679 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:09.679 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:09.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:09.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:09.680 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:09.680 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:09.680 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:09.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:09.681 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:09.681 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:09.682 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:09.682 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:09.682 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:09.683 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:09.683 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:09.683 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:09.684 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:09.684 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:09.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:09.685 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:09.685 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:09.686 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:09.686 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:09.686 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:09.687 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:09.687 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:09.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:09.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:09.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:09.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:09.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:09.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:10.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:10.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:10.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:10.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:10.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:10.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:10.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:10.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:10.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:10.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:10.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:10.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:10.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:11.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841148 2021-01-15T18:37:11.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841148 '!=' null 2021-01-15T18:37:11.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:11.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:11.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:11.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:11.585 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:11.586 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:11.586 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:11.586 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:11.587 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:11.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:11.587 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:11.587 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:11.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:11.588 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:11.588 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:11.588 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:11.588 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:11.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:11.589 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:11.590 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:11.590 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:11.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:11.590 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:11.590 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:11.591 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:11.591 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:11.591 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:11.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:11.591 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:11.592 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:11.592 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:11.592 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:11.592 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:11.592 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:11.593 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:11.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:11.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:11.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:11.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:11.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:11.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:11.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:11.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:11.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:12.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:12.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:12.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:12.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:12.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:12.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:12.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:12.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:12.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:12.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:13.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104626 2021-01-15T18:37:13.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104626 '!=' null 2021-01-15T18:37:13.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:13.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:13.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:13.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:13.273 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:13.274 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:13.274 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:13.274 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:13.274 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:13.274 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:13.275 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:13.275 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:13.275 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:13.275 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:13.275 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:13.276 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:13.276 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:13.276 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:13.276 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:13.276 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:13.277 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:13.277 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:13.277 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:13.277 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:13.277 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:13.278 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:13.278 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:13.278 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:13.278 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:13.278 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:13.279 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:13.279 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:13.279 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:13.279 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:13.280 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:13.280 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:13.280 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:13.280 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:13.280 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:13.281 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:13.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:13.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:13.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:13.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:13.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:13.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:13.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:13.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:13.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:13.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:14.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:14.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:14.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:14.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:14.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:15.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104708 2021-01-15T18:37:15.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104708 '!=' null 2021-01-15T18:37:15.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:15.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:15.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:15.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:15.124 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:15.124 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:15.124 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:15.125 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:15.125 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:15.126 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:15.126 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:15.126 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:15.127 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:15.127 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:15.128 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:15.128 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:15.128 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:15.129 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:15.129 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:15.129 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:15.130 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:15.130 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:15.131 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:15.131 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:15.131 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:15.132 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:15.132 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:15.133 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:15.133 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:15.133 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:15.134 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:15.134 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:15.134 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:15.135 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:15.135 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:15.136 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:15.136 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:15.136 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:15.137 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:15.137 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:15.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:15.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:15.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:15.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:15.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:15.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:15.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:15.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:15.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:15.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:15.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:15.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:15.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:16.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:16.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:16.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:16.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:16.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841209 2021-01-15T18:37:16.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841209 '!=' null 2021-01-15T18:37:16.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:16.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:16.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:16.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:16.856 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:16.856 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:16.857 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:16.857 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:16.857 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:16.857 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:16.858 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:16.858 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:16.858 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:16.858 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:16.858 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:16.859 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:16.859 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:16.859 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:16.859 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:16.860 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:16.860 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:16.860 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:16.860 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:16.861 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:16.861 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:16.861 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:16.862 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:16.862 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:16.862 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:16.862 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:16.862 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:16.863 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:16.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:16.863 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:16.863 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:16.864 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:16.864 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:16.864 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:16.864 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:16.865 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:16.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:16.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:16.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:16.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:16.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:16.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:16.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:16.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:16.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:17.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:17.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:17.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:17.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:18.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:18.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:18.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:18.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:18.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978374 2021-01-15T18:37:18.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978374 '!=' null 2021-01-15T18:37:18.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:18.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:18.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:18.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:18.740 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:18.740 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:18.741 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:18.741 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:18.741 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:18.741 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:18.742 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:18.742 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:18.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:18.742 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:18.742 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:18.743 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:18.743 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:18.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:18.743 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:18.743 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:18.744 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:18.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:18.744 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:18.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:18.744 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:18.745 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:18.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:18.745 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:18.745 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:18.745 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:18.746 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:18.746 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:18.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:18.746 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:18.746 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:18.747 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:18.747 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:18.747 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:18.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:18.747 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:18.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:18.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:18.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:18.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:18.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:18.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:18.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:18.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:18.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:19.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:19.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:19.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:19.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:19.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:19.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:19.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:19.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:19.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:19.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:19.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:19.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:19.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:20.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967631 2021-01-15T18:37:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967631 '!=' null 2021-01-15T18:37:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:20.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:20.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:20.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:20.519 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:20.519 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:20.520 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:20.520 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:20.520 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:20.520 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:20.521 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:20.521 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:20.521 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:20.522 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:20.522 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:20.522 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:20.522 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:20.523 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:20.523 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:20.523 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:20.523 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:20.524 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:20.524 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:20.524 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:20.524 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:20.524 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:20.525 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:20.525 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:20.525 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:20.525 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:20.525 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:20.526 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:20.526 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:20.526 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:20.526 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:20.526 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:20.527 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:20.527 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:20.527 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:20.527 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:20.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:20.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:20.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:20.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:20.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:20.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:20.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:20.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:20.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:21.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:21.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:21.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:21.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:21.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:21.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:21.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:21.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:21.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:21.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:21.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:21.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:21.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:22.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725798 2021-01-15T18:37:22.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725798 '!=' null 2021-01-15T18:37:22.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:22.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:22.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:22.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:22.197 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:22.197 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:22.198 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:22.198 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:22.198 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:22.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:22.198 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:22.199 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:22.199 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:22.199 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:22.199 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:22.200 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:22.200 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:22.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:22.200 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:22.200 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:22.201 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:22.201 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:22.201 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:22.201 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:22.201 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:22.202 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:22.202 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:22.202 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:22.202 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:22.202 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:22.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:22.203 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:22.203 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:22.203 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:22.204 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:22.204 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:22.205 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:22.205 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:22.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:22.205 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:22.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:22.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:22.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:22.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:22.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:22.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:22.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:22.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:22.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:22.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:22.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:22.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:22.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:23.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:23.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:23.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:23.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:23.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:23.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:24.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725898 2021-01-15T18:37:24.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725898 '!=' null 2021-01-15T18:37:24.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:24.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:24.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:24.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:24.281 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:24.282 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:24.282 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:24.283 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:24.283 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:24.283 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:24.284 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:24.284 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:24.285 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:24.285 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:24.285 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:24.286 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:24.286 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:24.287 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:24.287 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:24.287 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:24.288 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:24.288 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:24.289 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:24.289 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:24.289 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:24.290 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:24.290 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:24.290 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:24.291 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:24.291 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:24.292 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:24.292 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:24.292 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:24.293 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:24.293 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:24.293 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:24.293 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:24.293 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:24.294 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:24.294 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:24.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:24.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:24.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:24.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:24.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:24.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:24.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:24.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:24.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:24.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:24.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:24.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:24.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:25.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:25.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:25.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:25.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:25.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:25.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588985 2021-01-15T18:37:25.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588985 '!=' null 2021-01-15T18:37:25.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:25.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:25.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:25.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:25.957 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:25.957 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:25.958 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:25.958 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:25.958 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:25.958 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:25.959 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:25.959 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:25.959 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:25.959 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:25.959 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:25.960 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:25.961 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:25.961 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:25.961 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:25.962 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:25.962 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:25.962 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:25.962 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:25.962 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:25.963 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:25.963 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:25.963 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:25.963 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:25.963 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:25.964 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:25.964 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:25.964 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:25.964 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:25.964 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:25.965 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:25.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:26.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:26.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:26.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:26.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:26.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:26.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:26.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:26.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:26.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:26.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:26.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:26.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:27.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:27.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:27.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:27.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:27.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:27.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852294 2021-01-15T18:37:27.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852294 '!=' null 2021-01-15T18:37:27.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:27.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:27.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:27.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:27.954 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:27.954 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:27.954 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:27.955 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:27.955 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:27.955 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:27.955 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:27.956 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:27.956 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:27.956 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:27.956 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:27.956 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:27.957 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:27.957 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:27.957 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:27.957 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:27.957 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:27.958 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:27.958 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:27.958 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:27.958 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:27.958 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:27.959 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:27.959 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:27.959 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:27.959 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:27.959 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:27.960 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:27.960 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:27.960 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:27.960 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:27.960 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:27.961 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:27.961 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:27.961 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:27.961 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:27.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:28.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:28.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:28.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:28.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:28.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:28.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:28.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:28.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:28.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:28.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:28.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:28.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:28.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:28.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:28.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:28.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:28.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:28.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:28.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:28.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:28.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:29.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852294 2021-01-15T18:37:29.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852294 '!=' null 2021-01-15T18:37:29.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:29.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:29.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:29.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:29.632 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:29.632 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:29.632 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:29.633 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:29.633 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:29.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:29.634 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:29.634 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:29.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:29.635 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:29.636 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:29.636 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:29.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:29.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:29.637 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:29.637 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:29.638 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:29.638 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:29.638 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:29.639 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:29.639 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:29.639 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:29.640 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:29.640 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:29.640 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:29.641 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:29.641 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:29.641 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:29.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:29.642 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:29.642 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:29.643 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:29.643 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:29.643 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:29.643 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:29.644 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:29.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:29.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:29.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:29.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:29.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:29.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:29.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:29.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:29.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:30.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:30.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:30.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:30.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:30.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:30.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:30.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:30.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:30.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:30.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:30.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:30.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:31.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588969 2021-01-15T18:37:31.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588969 '!=' null 2021-01-15T18:37:31.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:31.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:31.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:31.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:31.276 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:31.276 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:31.277 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:31.277 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:31.277 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:31.277 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:31.277 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:31.278 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:31.278 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:31.278 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:31.278 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:31.278 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:31.279 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:31.279 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:31.279 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:31.279 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:31.279 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:31.280 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:31.280 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:31.280 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:31.280 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:31.281 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:31.281 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:31.281 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:31.281 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:31.281 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:31.282 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:31.282 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:31.282 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:31.282 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:31.282 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:31.283 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:31.283 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:31.283 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:31.283 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:31.283 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:31.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:31.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:31.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:31.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:31.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:31.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:31.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:31.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:31.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:31.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:31.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:31.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:31.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:32.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:32.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:32.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:32.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:32.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:32.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:32.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:32.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:32.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:33.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599467 2021-01-15T18:37:33.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599467 '!=' null 2021-01-15T18:37:33.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:33.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:33.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:33.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:33.105 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:33.105 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:33.106 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:33.107 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:33.108 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:33.109 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:33.110 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:33.111 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:33.112 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:33.112 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:33.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:33.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:33.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:33.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:33.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:33.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:33.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:33.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:33.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:33.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:33.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:33.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:34.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:34.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:34.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:34.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:34.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:34.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:34.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599475 2021-01-15T18:37:34.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599475 '!=' null 2021-01-15T18:37:34.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:34.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:34.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:34.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:34.747 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:34.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:34.749 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:34.750 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:34.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:34.752 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:34.753 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:34.754 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:34.754 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:34.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:34.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:34.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:34.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:34.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:34.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:34.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:34.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:34.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:35.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:35.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:35.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:35.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:35.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:35.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:35.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:35.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:35.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:35.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:35.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:35.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:35.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:36.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462301 2021-01-15T18:37:36.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462301 '!=' null 2021-01-15T18:37:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:36.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:36.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:36.577 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:36.577 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:36.578 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:36.578 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:36.578 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:36.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:36.579 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:36.579 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:36.579 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:36.579 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:36.580 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:36.580 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:36.580 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:36.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:36.580 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:36.581 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:36.581 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:36.581 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:36.581 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:36.582 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:36.582 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:36.582 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:36.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:36.583 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:36.583 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:36.583 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:36.583 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:36.584 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:36.584 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:36.584 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:36.584 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:36.585 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:36.585 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:36.585 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:36.585 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:36.586 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:36.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:36.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:36.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:36.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:36.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:36.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:36.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:36.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:36.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:37.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:37.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:37.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:37.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:37.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:37.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:37.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:37.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:38.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599329 2021-01-15T18:37:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599329 '!=' null 2021-01-15T18:37:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:38.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:38.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:38.346 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:38.347 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:38.347 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:38.347 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:38.347 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:38.347 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:38.348 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:38.348 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:38.348 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:38.348 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:38.348 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:38.349 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:38.349 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:38.349 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:38.349 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:38.349 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:38.350 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:38.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:38.350 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:38.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:38.351 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:38.351 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:38.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:38.351 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:38.351 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:38.352 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:38.353 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:38.353 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:38.353 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:38.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:38.354 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:38.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:38.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:38.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:38.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:38.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:38.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:38.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:38.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:38.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:38.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:38.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:38.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:38.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:39.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:39.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:39.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:39.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:39.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:39.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:39.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462169 2021-01-15T18:37:39.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462169 '!=' null 2021-01-15T18:37:39.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:39.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:39.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:39.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:40.031 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:40.032 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:40.032 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:40.032 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:40.032 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:40.033 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:40.033 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:40.033 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:40.033 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:40.033 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:40.034 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:40.034 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:40.034 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:40.034 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:40.035 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:40.035 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:40.035 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:40.035 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:40.035 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:40.036 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:40.036 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:40.036 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:40.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:40.037 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:40.037 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:40.037 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:40.037 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:40.038 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:40.038 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:40.038 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:40.038 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:40.039 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:40.039 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:40.040 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:40.040 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:40.040 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:40.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:40.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:40.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:40.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:40.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:40.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:40.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:40.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:40.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:40.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:40.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:40.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:40.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:41.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:41.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:41.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:41.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:41.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:41.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:41.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:41.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:41.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:41.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462169 2021-01-15T18:37:41.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462169 '!=' null 2021-01-15T18:37:41.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:41.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:41.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:41.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:41.689 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:41.689 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:41.689 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:41.689 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:41.690 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:41.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:41.690 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:41.690 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:41.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:41.691 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:41.691 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:41.691 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:41.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:41.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:41.692 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:41.692 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:41.692 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:41.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:41.692 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:41.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:41.693 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:41.693 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:41.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:41.693 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:41.694 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:41.694 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:41.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:41.694 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:41.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:41.695 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:41.695 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:41.695 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:41.695 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:41.695 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:41.696 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:41.696 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:41.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:41.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:41.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:41.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:41.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:42.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:42.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:42.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:42.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:42.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:42.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:42.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:42.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:42.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:42.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:42.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:42.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:42.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:43.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725677 2021-01-15T18:37:43.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725677 '!=' null 2021-01-15T18:37:43.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:43.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:43.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:43.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:43.503 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:43.503 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:43.504 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:43.504 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:43.504 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:43.504 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:43.505 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:43.505 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:43.505 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:43.505 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:43.506 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:43.506 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:43.506 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:43.506 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:43.506 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:43.507 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:43.507 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:43.507 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:43.507 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:43.507 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:43.508 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:43.508 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:43.508 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:43.508 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:43.509 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:43.509 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:43.509 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:43.509 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:43.509 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:43.510 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:43.510 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:43.510 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:43.510 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:43.511 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:43.511 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:43.511 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:43.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:43.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:43.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:43.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:43.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:43.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:43.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:43.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:43.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:44.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:44.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:44.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:44.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:44.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:44.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:44.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:44.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:45.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725856 2021-01-15T18:37:45.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725856 '!=' null 2021-01-15T18:37:45.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:45.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:45.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:45.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:45.684 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:45.685 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:45.685 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:45.685 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:45.685 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:45.685 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:45.686 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:45.686 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:45.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:45.686 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:45.686 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:45.687 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:45.687 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:45.687 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:45.687 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:45.687 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:45.688 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:45.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:45.688 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:45.688 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:45.688 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:45.689 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:45.690 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:45.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:45.690 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:45.690 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:45.690 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:45.691 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:45.691 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:45.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:45.691 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:45.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:45.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:45.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:45.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:45.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:46.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:46.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:46.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:46.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:46.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:46.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:46.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:46.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:46.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462388 2021-01-15T18:37:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462388 '!=' null 2021-01-15T18:37:47.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:47.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:47.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:47.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:47.401 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:47.401 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:47.401 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:47.402 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:47.402 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:47.402 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:47.402 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:47.402 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:47.403 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:47.403 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:47.403 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:47.403 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:47.403 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:47.404 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:47.404 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:47.404 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:47.404 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:47.404 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:47.405 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:47.405 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:47.405 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:47.405 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:47.405 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:47.406 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:47.406 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:47.406 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:47.406 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:47.406 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:47.407 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:47.407 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:47.407 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:47.407 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:47.407 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:47.408 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:47.408 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:47.408 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:47.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:47.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:47.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:47.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:47.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:47.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:47.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:47.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:47.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:47.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:47.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:47.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:47.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:48.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:48.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:48.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599420 2021-01-15T18:37:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599420 '!=' null 2021-01-15T18:37:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:49.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:49.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:49.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:49.328 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:49.329 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:49.329 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:49.329 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:49.329 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:49.330 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:49.330 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:49.330 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:49.330 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:49.331 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:49.331 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:49.331 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:49.331 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:49.331 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:49.332 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:49.332 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:49.332 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:49.332 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:49.333 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:49.333 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:49.333 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:49.333 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:49.333 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:49.334 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:49.334 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:49.334 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:49.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:49.335 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:49.335 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:49.335 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:49.335 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:49.335 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:49.336 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:49.336 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:49.336 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:49.336 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:49.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:49.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:49.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:49.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:49.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:49.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:49.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:49.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:49.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:49.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:49.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:49.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:49.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:50.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:50.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:50.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:50.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:50.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:50.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588202 2021-01-15T18:37:50.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588202 '!=' null 2021-01-15T18:37:50.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:50.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:50.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:50.987 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:50.988 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:50.988 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:50.988 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:50.988 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:50.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:50.989 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:50.989 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:50.989 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:50.989 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:50.990 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:50.990 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:50.990 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:50.990 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:50.990 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:50.991 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:50.991 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:50.991 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:50.991 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:50.991 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:50.992 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:50.992 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:50.992 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:50.992 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:50.992 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:50.993 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:50.993 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:50.993 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:50.993 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:50.993 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:50.994 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:50.994 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:50.994 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:50.994 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:50.994 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:50.995 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:50.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:51.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:51.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:51.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:51.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:51.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:51.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:51.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:51.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:51.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:51.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:51.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:51.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:52.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:52.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:52.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:52.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:52.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:52.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725258 2021-01-15T18:37:52.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725258 '!=' null 2021-01-15T18:37:52.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:52.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:52.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:52.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:53.047 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:53.047 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:53.047 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:53.048 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:53.048 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:53.048 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:53.048 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:53.048 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:53.049 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:53.049 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:53.049 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:53.049 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:53.049 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:53.050 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:53.050 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:53.050 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:53.050 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:53.050 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:53.051 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:53.051 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:53.051 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:53.051 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:53.051 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:53.052 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:53.052 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:53.052 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:53.052 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:53.053 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:53.054 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:53.054 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:53.054 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:53.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:53.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:53.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:53.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:53.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:53.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:53.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:53.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:53.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:53.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:53.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:53.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:54.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:54.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:54.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:54.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:54.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:54.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:54.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:54.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:54.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:54.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725203 2021-01-15T18:37:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725203 '!=' null 2021-01-15T18:37:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:54.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:54.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:54.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:54.676 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:54.676 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:54.676 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:54.677 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:54.677 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:54.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:54.677 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:54.677 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:54.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:54.678 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:54.678 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:54.678 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:54.678 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:54.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:54.679 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:54.679 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:54.679 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:54.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:54.680 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:54.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:54.680 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:54.680 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:54.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:54.681 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:54.681 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:54.681 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:54.681 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:54.682 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:54.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:54.682 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:54.682 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:54.682 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:54.683 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:54.683 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:54.683 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:54.683 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:54.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:54.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:54.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:54.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:54.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:54.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:54.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:54.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:54.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:55.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:55.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:55.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:55.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:55.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:55.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:55.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:55.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:55.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:55.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:56.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461480 2021-01-15T18:37:56.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461480 '!=' null 2021-01-15T18:37:56.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:56.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:56.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:56.379 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:56.380 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:56.380 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:56.380 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:56.380 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:56.381 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:56.382 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:56.382 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:56.382 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:56.382 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:56.383 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:56.383 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:56.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:56.383 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:56.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:56.384 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:56.384 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:56.384 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:56.384 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:56.384 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:56.385 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:56.386 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:56.386 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:56.386 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:56.386 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:56.386 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:56.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:56.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:56.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:56.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:56.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:56.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:56.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:56.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:56.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:56.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:56.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:56.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:56.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:57.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:57.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:57.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:57.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:57.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:58.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725209 2021-01-15T18:37:58.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725209 '!=' null 2021-01-15T18:37:58.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:58.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:58.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:58.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:58.226 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:58.226 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:58.227 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:58.227 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:58.227 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:58.227 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:58.227 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:58.228 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:58.228 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:58.228 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:58.228 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:58.228 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:58.229 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:58.230 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:58.230 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:58.230 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:58.230 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:58.230 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:58.231 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:58.232 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:58.232 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:58.232 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:58.232 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:58.232 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:58.233 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:58.233 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:58.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:58.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:58.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:58.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:58.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:58.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:58.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:58.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:58.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:37:58.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:37:58.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:37:58.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:37:58.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:37:59.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:37:59.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:37:59.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:37:59.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:37:59.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:37:59.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:37:59.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:37:59.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:37:59.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:37:59.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725209 2021-01-15T18:37:59.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725209 '!=' null 2021-01-15T18:37:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:37:59.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:59.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:37:59.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:37:59.862 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:37:59.862 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:37:59.862 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:59.863 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:59.863 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:37:59.863 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:59.863 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:37:59.864 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:37:59.864 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:59.864 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:37:59.864 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:59.864 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:59.865 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:59.865 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:59.865 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:37:59.865 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:37:59.865 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:59.866 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:59.866 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:59.866 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:59.866 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:37:59.866 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:37:59.867 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:37:59.867 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:37:59.867 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:37:59.867 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:37:59.867 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:59.868 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:37:59.868 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:37:59.868 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:37:59.868 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:37:59.869 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:37:59.869 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:37:59.869 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:37:59.869 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:37:59.869 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:37:59.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:37:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:37:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:37:59.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:37:59.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:37:59.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:37:59.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:37:59.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:37:59.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:00.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:00.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:00.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:00.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:01.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:01.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:01.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:01.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:01.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714294 2021-01-15T18:38:01.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714294 '!=' null 2021-01-15T18:38:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:01.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:01.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:01.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:01.843 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:01.843 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:01.844 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:01.844 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:01.844 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:01.844 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:01.845 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:01.845 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:01.845 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:01.845 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:01.845 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:01.846 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:01.846 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:01.846 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:01.846 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:01.847 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:01.847 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:01.847 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:01.847 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:01.848 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:01.848 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:01.848 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:01.848 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:01.848 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:01.849 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:01.849 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:01.849 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:01.849 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:01.849 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:01.850 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:01.850 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:01.850 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:01.850 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:01.851 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:01.851 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:01.851 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:01.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:01.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:01.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:01.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:01.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:01.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:01.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:01.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:01.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:02.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:02.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:02.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:02.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:03.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:03.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:03.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:03.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:03.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:03.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:03.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725391 2021-01-15T18:38:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725391 '!=' null 2021-01-15T18:38:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:03.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:03.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:03.652 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:03.653 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:03.653 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:03.653 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:03.654 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:03.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:03.654 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:03.654 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:03.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:03.655 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:03.655 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:03.655 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:03.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:03.656 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:03.656 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:03.656 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:03.656 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:03.656 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:03.657 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:03.657 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:03.657 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:03.657 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:03.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:03.658 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:03.658 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:03.658 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:03.658 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:03.659 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:03.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:03.659 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:03.659 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:03.659 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:03.660 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:03.660 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:03.660 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:03.660 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:03.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:03.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:03.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:03.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:03.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:03.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:03.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:04.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:04.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:04.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:04.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:04.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:04.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:04.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:04.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:04.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:04.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:04.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:04.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:05.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725607 2021-01-15T18:38:05.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725607 '!=' null 2021-01-15T18:38:05.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:05.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:05.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:05.416 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:05.416 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:05.416 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:05.417 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:05.417 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:05.417 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:05.417 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:05.418 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:05.418 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:05.418 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:05.418 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:05.419 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:05.419 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:05.419 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:05.419 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:05.419 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:05.420 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:05.420 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:05.420 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:05.420 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:05.421 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:05.421 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:05.421 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:05.421 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:05.422 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:05.422 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:05.422 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:05.422 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:05.422 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:05.423 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:05.423 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:05.423 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:05.423 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:05.423 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:05.424 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:05.424 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:05.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:05.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:05.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:05.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:05.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:05.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:05.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:05.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:05.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:05.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:06.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:06.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:06.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:06.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:06.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:06.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:06.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:06.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:06.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:07.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18346491 2021-01-15T18:38:07.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18346491 '!=' null 2021-01-15T18:38:07.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:07.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:07.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:07.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:07.238 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:07.239 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:07.239 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:07.239 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:07.239 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:07.239 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:07.240 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:07.240 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:07.240 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:07.240 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:07.240 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:07.241 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:07.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:07.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:07.241 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:07.241 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:07.242 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:07.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:07.242 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:07.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:07.242 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:07.243 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:07.243 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:07.243 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:07.243 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:07.243 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:07.244 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:07.245 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:07.245 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:07.245 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:07.245 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:07.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:07.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:07.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:07.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:07.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:07.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:07.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:07.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:07.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:07.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:07.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:07.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:08.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:08.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:08.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:08.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:08.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:08.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:08.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:08.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:08.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:08.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=9610037 2021-01-15T18:38:08.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 9610037 '!=' null 2021-01-15T18:38:08.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:08.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:08.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:08.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:08.908 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:08.908 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:08.908 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:08.909 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:08.909 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:08.909 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:08.909 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:08.909 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:08.910 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:08.910 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:08.910 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:08.910 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:08.910 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:08.911 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:08.912 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:08.912 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:08.912 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:08.912 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:08.912 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:08.913 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:08.913 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:08.913 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:08.913 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:08.914 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:08.914 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:08.914 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:08.914 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:08.914 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:08.915 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:08.915 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:08.915 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:08.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:09.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:09.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:09.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:09.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:09.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:09.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:09.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:09.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:09.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:09.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:09.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:09.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:10.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:10.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:10.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:10.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:10.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:10.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:10.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220068 2021-01-15T18:38:10.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220068 '!=' null 2021-01-15T18:38:10.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:10.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:10.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:10.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:10.810 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:10.810 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:10.810 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:10.811 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:10.811 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:10.811 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:10.811 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:10.811 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:10.812 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:10.813 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:10.813 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:10.813 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:10.813 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:10.813 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:10.814 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:10.814 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:10.814 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:10.814 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:10.814 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:10.815 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:10.815 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:10.815 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:10.815 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:10.815 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:10.816 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:10.817 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:10.817 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:10.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:10.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:10.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:10.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:10.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:10.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:10.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:10.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:10.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:11.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:11.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:11.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:11.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:11.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:11.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:11.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:11.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:11.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:11.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:11.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:11.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:11.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:12.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12230928 2021-01-15T18:38:12.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12230928 '!=' null 2021-01-15T18:38:12.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:12.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:12.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:12.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:12.478 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:12.478 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:12.479 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:12.479 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:12.479 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:12.479 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:12.479 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:12.480 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:12.480 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:12.480 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:12.480 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:12.480 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:12.481 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:12.482 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:12.482 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:12.482 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:12.482 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:12.482 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:12.483 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:12.483 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:12.483 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:12.483 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:12.483 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:12.484 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:12.484 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:12.484 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:12.484 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:12.484 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:12.485 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:12.485 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:12.485 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:12.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:12.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:12.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:12.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:12.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:12.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:12.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:12.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:12.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:13.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:13.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:13.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:13.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:13.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:13.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:13.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:14.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12231084 2021-01-15T18:38:14.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12231084 '!=' null 2021-01-15T18:38:14.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:14.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:14.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:14.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:14.511 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:14.511 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:14.511 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:14.511 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:14.512 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:14.512 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:14.512 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:14.512 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:14.512 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:14.513 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:14.514 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:14.514 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:14.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:14.514 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:14.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:14.515 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:14.516 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:14.516 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:14.516 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:14.516 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:14.516 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:14.517 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:14.517 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:14.517 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:14.517 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:14.517 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:14.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:14.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:14.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:14.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:14.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:14.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:14.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:14.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:14.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:15.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:15.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:15.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:15.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:15.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:15.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:16.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093960 2021-01-15T18:38:16.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093960 '!=' null 2021-01-15T18:38:16.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:16.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:16.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:16.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:16.543 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:16.543 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:16.544 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:16.544 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:16.544 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:16.544 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:16.544 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:16.545 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:16.545 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:16.545 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:16.545 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:16.546 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:16.546 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:16.546 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:16.546 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:16.546 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:16.547 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:16.547 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:16.547 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:16.547 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:16.547 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:16.548 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:16.548 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:16.548 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "7.0", 2021-01-15T18:38:16.548 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 255, 2021-01-15T18:38:16.548 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:16.549 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:16.549 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:16.549 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:16.549 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:16.549 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:16.550 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:16.550 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:16.550 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:16.550 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:16.551 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:16.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:16.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:16.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:16.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:16.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:16.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:16.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:16.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:16.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:17.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:17.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:17.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:17.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:17.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:17.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:17.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:17.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:17.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:17.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:17.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:17.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:17.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:18.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17472861 2021-01-15T18:38:18.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17472861 '!=' null 2021-01-15T18:38:18.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:18.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:18.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:18.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:18.344 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:18.344 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:18.344 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:18.345 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:18.345 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:18.345 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:18.345 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:18.345 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:18.346 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:18.346 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:18.346 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:18.346 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:18.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:18.347 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:18.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:18.347 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:18.347 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:18.348 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:18.348 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:18.348 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:18.348 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:18.348 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:18.349 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:18.349 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:18.349 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:18.349 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:18.350 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:18.350 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:18.350 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:18.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:18.350 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:18.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:18.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:18.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:18.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:18.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:18.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:18.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:18.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:18.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:18.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:18.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:18.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:19.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:19.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:19.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:19.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:19.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:19.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:19.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:19.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:19.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:20.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22277933 2021-01-15T18:38:20.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22277933 '!=' null 2021-01-15T18:38:20.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:20.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:20.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:20.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:20.265 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:20.265 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:20.266 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:20.266 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:20.266 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:20.266 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:20.267 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:20.267 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:20.267 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:20.267 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:20.267 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:20.268 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:20.268 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:20.268 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:20.268 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:20.269 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:20.269 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:20.269 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:20.269 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:20.270 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:20.270 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:20.270 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:20.270 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:20.270 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:20.271 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:20.271 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:20.271 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:20.271 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:20.272 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:20.272 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:20.272 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:20.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:20.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:20.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:20.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:20.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:20.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:20.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:20.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:20.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:20.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:20.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:20.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:20.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:21.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:21.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:21.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:21.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:21.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22277594 2021-01-15T18:38:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22277594 '!=' null 2021-01-15T18:38:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:21.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:21.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:22.028 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:22.029 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:22.029 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:22.029 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:22.029 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:22.029 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:22.030 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:22.030 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:22.030 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:22.030 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:22.030 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:22.031 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:22.031 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:22.031 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:22.031 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:22.031 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:22.032 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:22.032 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:22.032 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:22.032 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:22.032 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:22.033 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:22.034 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:22.034 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:22.034 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:22.034 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:22.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:22.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:22.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:22.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:22.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:22.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:22.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:22.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:22.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:22.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:22.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:22.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:22.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:23.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:23.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:23.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:23.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:23.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:23.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483555 2021-01-15T18:38:23.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483555 '!=' null 2021-01-15T18:38:23.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:23.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:23.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:23.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:23.900 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:23.901 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:23.901 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:23.901 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:23.901 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:23.902 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:23.902 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:23.902 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:23.902 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:23.903 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:23.903 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:23.903 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:23.903 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:23.903 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:23.904 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:23.904 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:23.904 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:23.904 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:23.904 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:23.905 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:23.905 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:23.905 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:23.905 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:23.905 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:23.906 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:23.906 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:23.906 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:23.906 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:23.906 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:23.907 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:23.907 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:23.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:24.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:24.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:24.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:24.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:24.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:24.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:24.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:24.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:38:24.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:24.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:24.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:24.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 3 2021-01-15T18:38:24.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:38:24.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:24.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:24.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:24.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:24.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:24.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:25.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=10483555 2021-01-15T18:38:25.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 10483555 '!=' null 2021-01-15T18:38:25.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:25.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:25.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:25.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:25.525 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:25.526 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:25.526 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:25.526 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:25.526 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:25.527 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:25.527 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:25.527 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:25.527 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:25.527 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:25.528 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:25.528 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:25.528 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:25.528 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:25.528 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:25.530 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:25.530 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:25.531 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:25.531 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:25.531 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:25.531 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:25.532 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:25.532 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:25.532 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:25.532 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:25.532 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:25.533 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:25.533 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:25.533 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:25.533 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:25.533 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:25.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:25.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:25.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:25.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:25.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:25.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:25.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:25.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:26.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:26.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:26.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:26.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:26.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:26.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 0 2021-01-15T18:38:26.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:38:26.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=1 2021-01-15T18:38:26.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:26.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:26.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:26.690 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:26.690 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:26.690 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:26.691 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:26.691 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:26.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:26.691 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:26.691 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:26.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:26.692 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:26.692 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:26.692 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:26.692 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:26.693 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:26.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:26.693 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:26.693 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:26.693 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:26.694 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:26.694 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:26.694 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:26.694 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:26.694 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:26.695 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:26.695 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:26.695 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:26.695 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:26.695 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:26.696 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:26.696 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:26.696 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:26.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:26.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:26.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:26.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:26.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:26.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:26.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:26.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:26.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:27.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:27.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:27.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:27.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:27.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:27.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:27.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:27.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:27.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:28.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20804833 2021-01-15T18:38:28.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20804833 '!=' null 2021-01-15T18:38:28.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:28.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:28.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:28.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:28.412 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:28.412 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:28.412 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:28.413 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:28.413 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:28.413 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:28.413 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:28.413 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:28.414 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:28.414 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:28.414 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:28.414 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:28.415 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:28.415 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:28.415 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:28.415 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:28.415 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:28.416 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:28.416 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:28.416 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:28.416 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:28.417 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:28.417 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:28.417 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:28.417 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:28.418 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:28.418 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:28.418 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:28.418 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:28.418 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:28.419 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:28.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:28.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:28.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:28.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:28.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:28.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:28.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:28.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:28.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:28.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:28.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:28.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:29.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:29.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:29.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:29.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:29.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:29.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:29.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:30.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=26917010 2021-01-15T18:38:30.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 26917010 '!=' null 2021-01-15T18:38:30.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:30.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:30.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:30.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:30.259 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:30.259 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:30.260 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:30.261 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:30.262 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:30.263 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:30.264 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:30.265 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:30.265 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:30.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:30.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:30.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:30.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:30.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:30.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:30.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:30.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:30.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:30.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:30.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:30.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:30.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:31.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:31.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:31.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:31.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:31.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:31.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17472922 2021-01-15T18:38:31.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17472922 '!=' null 2021-01-15T18:38:31.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:31.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:31.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:31.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:32.073 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:32.073 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:32.073 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:32.074 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:32.074 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:32.074 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:32.074 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:32.075 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:32.075 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:32.075 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:32.075 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:32.075 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:32.076 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:32.076 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:32.076 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:32.076 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:32.076 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:32.077 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:32.077 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:32.077 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:32.077 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:32.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:32.078 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:32.078 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:32.078 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:32.078 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:32.079 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:32.079 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:32.079 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:32.079 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:32.080 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:32.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:32.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:32.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:32.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:32.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:32.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:32.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:32.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:33.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:33.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:33.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:33.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:33.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:33.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17472922 2021-01-15T18:38:33.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17472922 '!=' null 2021-01-15T18:38:33.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:33.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:33.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:33.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:33.710 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:33.710 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:33.710 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:33.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:33.712 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:33.712 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:33.712 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:33.712 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:33.712 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:33.713 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:33.713 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:33.713 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:33.713 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:33.713 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:33.714 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:33.714 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:33.714 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:33.714 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:33.714 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:33.715 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:33.715 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:33.715 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:33.715 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:33.715 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:33.716 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:33.716 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:33.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:33.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:33.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:33.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:33.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:33.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:33.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:33.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:33.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:34.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:34.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:34.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:34.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:34.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:34.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:34.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:34.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:34.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:34.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:34.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:34.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:35.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=17473155 2021-01-15T18:38:35.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 17473155 '!=' null 2021-01-15T18:38:35.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:35.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:35.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:35.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:35.347 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:35.347 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:35.348 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:35.348 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:35.348 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:35.348 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:35.348 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:35.349 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:35.349 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:35.349 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:35.349 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:35.349 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:35.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:35.350 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:35.350 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:35.350 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:35.350 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:35.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:35.351 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:35.351 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:35.351 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:35.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:35.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:35.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:35.352 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:35.352 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:35.353 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:35.353 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:35.353 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:35.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:35.353 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:35.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:35.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:35.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:35.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:35.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:35.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:35.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:35.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:35.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:35.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:35.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:35.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:35.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:36.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:36.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:36.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:36.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:36.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:36.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:36.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:36.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:37.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462410 2021-01-15T18:38:37.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462410 '!=' null 2021-01-15T18:38:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:37.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:37.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:37.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:37.363 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:37.363 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:37.363 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:37.363 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:37.364 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:37.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:37.364 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:37.364 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:37.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:37.365 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:37.365 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:37.365 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:37.365 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:37.365 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:37.366 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:37.366 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:37.366 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:37.366 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:37.367 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:37.367 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:37.367 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:37.367 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:37.367 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:37.368 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:37.368 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:37.368 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:37.368 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:37.368 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:37.369 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:37.369 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:37.369 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:37.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:37.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:37.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:37.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:37.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:37.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:37.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:37.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:37.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:37.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:37.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:38.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:38.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:38.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:38.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:38.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:38.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:38.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:38.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:38.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:38.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852139 2021-01-15T18:38:38.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852139 '!=' null 2021-01-15T18:38:38.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:38.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:38.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:38.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:39.076 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:39.076 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:39.076 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:39.077 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:39.077 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:39.077 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:39.077 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:39.078 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:39.078 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:39.079 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:39.080 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:39.080 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:39.080 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:39.080 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:39.081 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:39.081 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:39.081 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:39.081 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:39.081 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:39.082 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:39.082 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:39.082 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:39.083 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:39.083 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:39.083 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:39.083 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:39.083 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:39.084 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:39.084 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:39.084 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:39.084 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:39.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:39.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:39.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:39.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:39.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:39.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:39.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:40.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:40.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715208 2021-01-15T18:38:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715208 '!=' null 2021-01-15T18:38:40.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:40.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:40.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:40.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:40.781 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:40.781 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:40.782 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:40.782 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:40.782 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:40.782 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:40.782 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:40.783 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:40.783 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:40.783 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:40.783 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:40.783 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:40.784 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:40.784 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:40.784 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:40.784 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:40.784 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:40.785 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:40.785 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:40.785 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:40.785 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:40.785 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:40.786 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:40.787 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:40.787 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:40.787 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:40.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:40.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:40.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:40.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:40.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:40.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:40.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:40.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:40.893 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:41.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:41.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:41.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:41.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:41.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:41.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:41.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:41.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:41.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:41.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:41.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:41.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:41.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:42.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852196 2021-01-15T18:38:42.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852196 '!=' null 2021-01-15T18:38:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:42.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:42.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:42.865 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:42.865 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:42.865 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:42.866 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:42.866 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:42.866 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:42.866 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:42.867 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:42.867 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:42.867 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:42.867 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:42.868 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:42.868 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:42.869 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:42.869 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:42.869 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:42.870 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:42.871 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:42.871 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:42.871 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:42.872 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:42.872 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:42.872 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:42.872 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:42.873 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:42.873 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:42.873 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:42.873 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:42.874 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:42.874 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:42.875 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:42.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:42.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:42.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:42.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:42.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:42.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:42.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:42.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:42.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:43.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:43.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:43.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:43.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:43.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:43.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:43.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:43.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:43.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:44.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852235 2021-01-15T18:38:44.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852235 '!=' null 2021-01-15T18:38:44.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:44.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:44.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:44.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:44.523 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:44.524 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:44.524 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:44.524 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:44.525 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:44.525 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:44.526 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:44.526 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:44.526 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:44.527 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:44.528 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:44.528 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:44.528 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:44.528 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:44.529 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:44.529 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:44.529 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:44.529 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:44.530 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:44.530 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:44.530 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:44.530 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:44.530 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:44.531 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:44.531 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:44.531 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:44.531 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:44.531 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:44.532 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:44.532 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:44.532 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:44.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:44.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:44.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:44.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:44.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:44.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:44.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:44.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:44.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:45.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:45.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:45.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:45.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:45.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:45.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:45.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:45.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:45.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:46.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588785 2021-01-15T18:38:46.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588785 '!=' null 2021-01-15T18:38:46.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:46.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:46.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:46.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:46.443 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:46.443 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:46.443 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:46.444 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:46.444 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:46.444 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:46.445 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:46.445 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:46.445 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:46.445 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:46.446 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:46.446 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:46.447 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:46.447 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:46.447 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:46.448 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:46.448 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:46.449 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:46.449 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:46.449 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:46.450 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:46.450 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:46.451 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:46.451 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:46.451 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:46.452 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:46.452 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:46.452 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:46.453 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:46.453 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:46.453 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:46.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:46.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:46.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:46.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:46.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:46.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:46.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:46.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:46.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:46.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:46.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:46.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:46.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:47.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:47.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:47.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:47.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:47.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:47.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:47.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:47.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:47.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:48.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599473 2021-01-15T18:38:48.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599473 '!=' null 2021-01-15T18:38:48.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:48.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:48.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:48.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:48.206 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:48.206 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:48.206 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:48.207 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:48.207 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:48.207 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:48.207 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:48.207 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:48.208 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:48.208 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:48.208 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:48.208 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:48.209 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:48.209 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:48.209 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:48.209 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:48.209 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:48.210 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:48.210 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:48.210 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:48.210 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:48.211 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:48.211 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:48.211 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:48.211 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:48.211 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:48.212 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:48.212 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:48.212 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:48.212 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:48.213 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:48.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:48.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:48.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:48.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:48.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:48.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:48.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:48.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:49.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:49.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:49.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:49.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:49.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:49.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:49.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:50.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588789 2021-01-15T18:38:50.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588789 '!=' null 2021-01-15T18:38:50.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:50.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:50.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:50.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:50.271 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:50.272 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:50.272 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:50.272 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:50.273 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:50.273 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:50.273 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:50.274 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:50.274 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:50.275 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:50.275 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:50.275 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:50.276 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:50.276 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:50.276 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:50.277 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:50.277 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:50.278 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:50.278 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:50.278 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:50.279 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:50.279 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:50.280 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:50.280 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:50.280 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:50.281 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:50.281 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:50.281 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:50.282 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:50.282 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:50.283 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:50.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:50.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:50.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:50.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:50.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:50.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:50.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:50.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:50.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:50.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:50.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:50.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:50.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:51.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:51.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:51.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:51.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:51.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:51.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:51.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588789 2021-01-15T18:38:51.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588789 '!=' null 2021-01-15T18:38:51.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:51.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:51.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:51.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:51.975 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:51.976 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:51.976 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:51.976 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:51.977 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:51.977 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:51.977 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:51.978 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:51.978 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:51.979 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:51.979 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:51.979 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:51.980 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:51.980 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:51.981 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:51.981 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:51.981 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:51.982 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:51.982 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:51.983 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:51.983 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:51.983 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:51.987 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:51.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:52.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:52.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:52.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:52.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:52.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:52.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:52.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:52.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:52.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:52.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:52.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:52.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:53.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:53.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:53.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:53.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:53.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:53.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:53.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:53.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:53.041 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725815 2021-01-15T18:38:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725815 '!=' null 2021-01-15T18:38:53.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:53.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:53.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:53.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:53.686 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:53.687 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:53.687 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:53.687 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:53.687 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:53.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:53.688 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:53.688 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:53.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:53.689 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:53.689 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:53.689 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:53.689 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:53.689 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:53.690 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:53.690 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:53.690 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:53.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:53.690 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:53.691 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:53.691 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:53.691 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:53.691 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:53.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:53.692 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:53.692 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:53.692 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:53.692 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:53.693 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:53.693 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:53.693 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:53.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:53.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:53.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:53.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:53.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:53.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:53.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:53.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:53.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:54.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:54.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:54.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:54.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:54.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:54.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:54.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:54.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:54.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:54.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:54.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:54.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:54.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725830 2021-01-15T18:38:55.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725830 '!=' null 2021-01-15T18:38:55.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:55.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:55.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:55.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:55.543 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:55.544 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:55.544 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:55.544 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:55.544 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:55.544 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:55.545 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:55.545 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:55.545 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:55.545 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:55.545 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:55.546 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:55.546 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:55.546 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:55.546 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:55.546 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:55.547 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:55.548 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:55.549 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:55.549 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:55.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:55.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:55.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:55.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:55.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:55.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:55.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:55.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:55.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:56.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:56.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:56.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:56.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:56.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:56.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:56.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:56.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:57.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462417 2021-01-15T18:38:57.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462417 '!=' null 2021-01-15T18:38:57.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:57.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:57.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:57.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:57.303 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:57.304 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:57.304 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:57.304 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:57.304 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:57.305 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:57.306 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:57.306 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:57.306 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:57.306 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:57.306 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:57.307 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:57.307 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:57.307 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:57.307 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:57.307 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:57.308 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:57.308 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:57.308 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:57.308 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:57.308 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:57.309 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:57.309 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:57.309 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:57.309 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:57.309 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:57.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:57.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:57.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:57.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:57.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:57.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:57.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:57.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:57.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:57.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:57.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:57.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:57.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:38:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:38:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:38:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:38:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:38:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:38:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:38:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:38:58.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:38:58.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:38:58.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725637 2021-01-15T18:38:58.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725637 '!=' null 2021-01-15T18:38:58.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:38:58.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:58.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:38:58.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:38:59.025 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:38:59.025 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:38:59.025 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:59.026 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:59.026 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:38:59.026 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:59.026 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:38:59.027 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:38:59.027 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:59.027 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:38:59.027 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:59.028 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:59.028 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:59.028 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:59.029 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:59.029 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:38:59.029 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:38:59.029 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:38:59.029 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:38:59.030 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:38:59.030 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:38:59.030 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:59.031 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:38:59.031 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:38:59.031 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:38:59.031 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:38:59.032 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:38:59.032 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:38:59.032 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:38:59.032 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:38:59.032 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:38:59.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:38:59.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:38:59.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:38:59.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:38:59.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:38:59.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:38:59.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:38:59.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:38:59.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:38:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:38:59.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:38:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:38:59.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:00.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:00.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:00.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:00.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:00.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:00.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:00.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:00.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:00.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841267 2021-01-15T18:39:00.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841267 '!=' null 2021-01-15T18:39:00.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:00.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:00.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:00.658 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:00.756 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:00.757 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:00.757 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:00.757 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:00.757 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:00.757 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:00.758 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:00.758 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:00.758 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:00.758 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:00.759 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:00.759 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:00.759 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:00.759 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:00.759 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:00.760 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:00.760 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:00.760 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:00.760 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:00.760 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:00.761 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:00.761 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:00.761 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:00.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:00.761 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:00.762 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:00.762 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:00.762 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:00.762 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:00.762 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:00.763 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:00.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:00.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:00.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:00.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:00.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:00.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:00.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:00.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:00.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:01.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:01.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:01.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:01.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:01.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:01.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:01.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:01.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:01.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:01.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:01.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:01.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:01.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:02.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852048 2021-01-15T18:39:02.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852048 '!=' null 2021-01-15T18:39:02.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:02.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:02.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:02.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:02.471 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:02.472 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:02.472 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:02.472 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:02.472 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:02.473 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:02.473 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:02.473 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:02.473 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:02.474 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:02.474 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:02.475 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:02.475 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:02.475 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:02.475 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:02.476 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:02.476 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:02.476 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:02.476 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:02.477 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:02.477 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:02.477 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:02.477 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:02.478 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:02.478 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:02.478 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:02.478 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:02.478 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:02.479 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:02.479 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:02.479 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:02.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:02.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:02.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:02.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:02.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:02.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:02.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:02.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:02.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:02.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:02.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:02.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:02.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:03.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:03.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:03.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:03.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:03.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:03.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:03.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:03.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:03.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:04.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852048 2021-01-15T18:39:04.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852048 '!=' null 2021-01-15T18:39:04.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:04.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:04.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:04.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:04.200 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:04.201 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:04.201 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:04.202 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:04.202 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:04.202 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:04.202 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:04.203 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:04.203 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:04.203 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:04.203 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:04.203 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:04.204 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:04.204 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:04.204 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:04.204 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:04.205 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:04.205 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:04.205 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:04.205 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:04.205 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:04.206 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:04.206 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:04.206 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:04.206 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:04.206 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:04.207 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:04.207 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:04.207 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:04.207 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:04.207 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:04.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:04.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:04.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:04.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:04.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:04.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:04.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:04.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:04.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:04.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:04.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:04.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:04.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:05.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:05.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:05.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:05.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:05.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:05.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714997 2021-01-15T18:39:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714997 '!=' null 2021-01-15T18:39:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:05.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:05.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:05.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:06.009 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:06.009 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:06.010 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:06.010 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:06.010 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:06.010 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:06.011 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:06.011 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:06.011 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:06.011 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:06.011 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:06.012 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:06.012 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:06.012 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:06.012 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:06.012 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:06.013 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:06.013 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:06.013 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:06.013 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:06.013 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:06.014 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:06.014 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:06.014 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:06.014 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:06.014 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:06.015 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:06.015 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:06.015 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:06.015 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:06.015 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:06.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:06.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:06.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:06.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:06.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:06.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:06.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:06.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:06.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:06.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:06.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:06.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:06.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:07.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:07.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:07.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:07.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:07.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:07.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:07.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:07.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:07.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:07.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978399 2021-01-15T18:39:07.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978399 '!=' null 2021-01-15T18:39:07.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:07.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:07.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:07.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:07.918 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:07.919 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:07.919 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:07.919 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:07.919 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:07.919 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:07.920 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:07.920 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:07.920 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:07.920 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:07.920 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:07.921 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:07.921 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:07.921 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:07.921 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:07.921 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:07.922 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:07.922 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:07.922 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:07.922 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:07.923 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:07.923 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:07.923 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:07.923 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:07.923 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:07.924 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:07.924 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:07.924 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:07.924 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:07.924 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:07.925 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:07.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:08.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:08.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:08.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:08.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:08.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:08.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:08.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:08.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:08.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:08.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:08.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:08.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:09.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:09.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:09.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:09.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978399 2021-01-15T18:39:09.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978399 '!=' null 2021-01-15T18:39:09.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:09.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:09.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:09.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:09.746 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:09.747 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:09.747 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:09.747 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:09.747 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:09.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:09.748 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:09.748 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:09.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:09.748 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:09.749 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:09.749 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:09.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:09.749 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:09.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:09.750 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:09.750 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:09.750 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:09.750 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:09.751 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:09.751 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:09.751 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:09.751 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:09.752 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:09.752 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:09.752 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:09.752 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:09.752 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:09.753 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:09.753 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:09.753 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:09.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:09.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:09.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:09.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:09.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:09.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:09.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:09.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:09.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:10.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:10.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:10.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:10.584 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:11.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:11.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:11.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:11.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:11.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:11.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:11.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:11.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:11.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715207 2021-01-15T18:39:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715207 '!=' null 2021-01-15T18:39:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:11.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:11.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:11.759 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:11.760 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:11.760 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:11.760 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:11.760 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:11.761 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:11.762 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:11.762 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:11.762 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:11.762 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:11.762 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:11.763 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:11.763 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:11.763 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:11.763 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:11.763 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:11.764 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:11.764 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:11.764 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:11.764 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:11.764 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:11.765 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:11.765 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:11.765 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:11.765 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:11.765 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:11.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:11.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:11.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:11.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:11.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:11.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:11.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:11.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:11.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:12.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:12.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:12.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:12.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:12.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:12.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:12.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:12.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:12.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:12.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:12.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:12.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:12.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:13.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599448 2021-01-15T18:39:13.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599448 '!=' null 2021-01-15T18:39:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:13.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:13.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:13.725 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:13.725 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:13.726 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:13.727 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:13.728 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:13.729 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:13.730 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:13.731 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:13.731 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:13.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:13.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:13.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:13.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:13.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:13.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:13.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:13.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:14.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:14.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:14.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:14.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:14.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:14.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:14.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:14.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:14.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:15.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599415 2021-01-15T18:39:15.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599415 '!=' null 2021-01-15T18:39:15.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:15.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:15.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:15.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:15.466 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:15.466 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:15.466 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:15.467 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:15.467 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:15.467 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:15.467 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:15.468 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:15.468 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:15.468 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:15.468 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:15.468 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:15.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:15.469 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:15.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:15.469 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:15.469 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:15.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:15.470 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:15.470 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:15.470 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:15.470 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:15.471 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:15.471 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:15.471 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:15.471 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:15.471 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:15.472 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:15.472 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:15.472 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:15.472 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:15.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:15.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:15.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:15.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:15.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:15.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:15.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:15.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:15.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:16.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:16.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:16.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:16.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:16.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:16.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:16.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:16.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841122 2021-01-15T18:39:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841122 '!=' null 2021-01-15T18:39:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:17.303 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:17.303 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:17.304 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:17.305 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:17.306 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:17.307 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:17.308 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:17.309 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:17.309 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:17.309 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:17.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:17.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:17.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:17.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:17.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:17.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:17.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:17.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:17.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:17.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:17.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:17.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:17.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:18.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:18.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:18.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:18.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:18.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:18.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:18.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:18.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:18.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:18.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978090 2021-01-15T18:39:18.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978090 '!=' null 2021-01-15T18:39:18.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:18.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:18.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:18.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:19.034 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:19.034 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:19.035 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:19.035 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:19.035 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:19.035 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:19.035 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:19.036 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:19.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:19.036 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:19.036 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:19.037 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:19.037 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:19.037 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:19.037 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:19.037 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:19.038 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:19.038 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:19.038 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:19.038 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:19.038 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:19.039 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:19.040 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:19.040 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:19.040 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:19.040 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:19.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:19.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:19.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:19.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:19.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:19.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:19.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:19.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:19.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:19.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:19.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:19.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:19.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:20.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:20.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20093592 2021-01-15T18:39:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20093592 '!=' null 2021-01-15T18:39:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:20.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:20.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:20.876 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:20.876 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:20.877 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:20.877 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:20.877 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:20.877 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:20.878 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:20.878 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:20.878 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:20.878 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:20.878 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:20.879 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:20.879 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:20.879 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:20.879 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:20.879 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:20.880 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:20.880 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:20.880 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:20.880 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:20.880 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:20.881 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:20.882 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:20.882 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:20.882 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:20.882 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:20.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:20.986 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:20.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:20.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:20.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:20.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:20.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:21.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:21.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:21.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:21.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:21.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:21.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:21.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:21.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:21.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:21.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:21.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:21.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:21.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11357271 2021-01-15T18:39:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11357271 '!=' null 2021-01-15T18:39:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:23.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:23.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:23.159 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:23.159 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:23.159 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:23.160 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:23.160 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:23.160 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:23.160 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:23.161 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:23.161 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:23.161 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:23.161 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:23.161 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:23.162 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:23.162 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:23.162 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:23.162 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:23.162 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:23.163 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:23.163 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:23.163 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:23.163 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:23.163 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:23.164 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:23.164 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:23.164 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:23.164 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:23.164 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:23.165 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:23.165 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:23.165 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:23.165 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:23.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:23.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:23.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:23.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:23.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:23.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:23.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:23.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:23.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:23.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:23.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:23.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:24.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:24.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:24.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:24.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:24.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:24.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:24.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:24.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:24.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11357222 2021-01-15T18:39:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11357222 '!=' null 2021-01-15T18:39:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:24.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:24.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:24.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:24.829 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:24.829 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:24.830 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:24.830 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:24.830 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:24.830 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:24.830 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:24.831 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:24.831 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:24.831 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:24.831 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:24.831 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:24.832 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:24.832 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:24.832 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:24.832 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:24.832 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:24.833 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:24.833 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:24.833 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:24.833 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:24.833 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:24.834 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:24.834 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:24.834 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:24.834 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:24.834 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:24.835 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:24.835 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:24.835 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:24.835 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:24.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:24.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:24.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:24.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:24.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:24.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:24.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:24.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:24.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:25.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:25.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:25.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:25.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:25.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:25.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:25.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:25.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:25.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:25.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:25.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:25.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:25.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:26.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19219920 2021-01-15T18:39:26.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19219920 '!=' null 2021-01-15T18:39:26.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:26.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:26.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:26.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:26.522 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:26.522 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:26.522 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:26.522 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:26.522 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:26.523 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:26.523 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:26.523 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:26.523 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:26.523 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:26.524 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:26.524 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:26.524 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:26.524 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:26.524 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:26.525 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:26.525 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:26.525 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:26.525 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:26.525 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:26.526 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:26.526 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:26.526 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:26.526 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:26.526 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:26.527 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:26.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:26.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:26.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:26.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:26.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:26.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:26.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:26.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:26.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:27.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:27.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:27.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:27.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:27.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:27.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:27.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:27.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:27.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:27.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:28.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978189 2021-01-15T18:39:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978189 '!=' null 2021-01-15T18:39:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:28.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:28.247 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:28.251 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:28.251 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:28.251 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:28.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:28.252 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:28.252 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:28.252 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:28.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:28.252 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:28.253 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:28.253 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:28.253 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:28.253 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:28.253 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:28.254 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:28.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:28.254 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:28.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:28.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:28.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:28.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:28.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:28.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:28.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:28.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:28.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:29.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:29.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:29.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:29.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:29.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:29.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:29.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:29.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:29.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:29.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:29.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:29.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:29.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:30.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967443 2021-01-15T18:39:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967443 '!=' null 2021-01-15T18:39:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:30.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:30.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:30.235 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:30.239 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:30.240 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:30.241 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:30.242 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:30.242 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:30.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:30.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:30.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:30.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:30.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:30.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:30.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:30.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:31.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:31.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:31.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:31.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:31.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:31.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:31.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:31.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:31.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:32.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851819 2021-01-15T18:39:32.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851819 '!=' null 2021-01-15T18:39:32.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:32.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:32.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:32.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:32.102 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:32.102 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:32.103 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:32.103 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:32.103 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:32.103 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:32.104 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:32.104 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:32.104 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:32.104 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:32.104 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:32.105 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:32.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:32.105 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:32.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:32.105 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:32.106 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:32.106 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:32.106 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:32.107 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:32.107 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:32.107 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:32.107 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:32.108 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:32.108 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:32.108 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:32.108 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:32.108 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:32.109 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:32.109 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:32.109 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:32.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:32.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:32.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:32.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:32.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:32.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:32.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:32.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:32.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:32.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:32.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:32.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:32.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:33.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:33.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:33.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:33.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:33.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:33.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:33.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:33.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:33.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:33.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851834 2021-01-15T18:39:33.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851834 '!=' null 2021-01-15T18:39:33.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:33.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:33.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:33.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:33.931 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:33.932 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:33.932 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:33.932 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:33.932 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:33.933 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:33.934 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:33.934 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:33.934 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:33.934 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:33.934 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:33.935 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:33.935 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:33.935 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:33.935 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:33.935 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:33.936 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:33.937 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:33.937 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:33.937 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:33.937 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:33.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:34.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:34.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:34.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:34.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:34.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:34.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:34.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:34.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:35.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:35.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:35.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:35.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:35.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851834 2021-01-15T18:39:35.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851834 '!=' null 2021-01-15T18:39:35.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:35.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:35.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:35.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:35.621 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:35.622 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:35.622 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:35.623 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:35.623 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:35.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:35.624 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:35.624 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:35.624 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:35.625 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:35.625 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:35.626 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:35.626 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:35.626 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:35.627 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:35.627 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:35.628 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:35.628 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:35.628 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:35.629 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:35.629 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:35.629 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:35.629 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:35.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:35.630 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:35.630 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:35.630 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:35.631 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:35.631 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:35.631 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:35.631 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:35.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:35.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:35.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:35.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:35.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:35.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:35.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:35.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:36.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:36.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:36.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:36.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:36.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:36.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:36.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:36.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:36.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:36.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:36.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:36.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:36.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:37.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588230 2021-01-15T18:39:37.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588230 '!=' null 2021-01-15T18:39:37.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:37.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:37.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:37.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:37.280 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:37.280 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:37.281 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:37.281 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:37.281 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:37.282 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:37.282 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:37.283 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:37.283 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:37.283 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:37.283 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:37.284 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:37.284 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:37.284 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:37.284 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:37.285 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:37.285 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:37.285 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:37.285 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:37.285 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:37.286 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:37.286 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:37.286 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:37.286 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:37.286 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:37.287 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:37.287 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:37.287 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:37.287 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:37.288 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:37.288 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:37.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:37.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:37.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:37.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:37.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:37.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:37.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:37.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:37.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:37.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:37.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:38.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:38.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:38.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:39.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725503 2021-01-15T18:39:39.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725503 '!=' null 2021-01-15T18:39:39.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:39.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:39.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:39.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:39.180 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:39.180 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:39.180 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:39.181 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:39.181 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:39.181 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:39.181 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:39.181 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:39.182 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:39.182 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:39.182 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:39.182 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:39.182 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:39.183 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:39.183 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:39.183 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:39.183 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:39.183 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:39.184 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:39.184 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:39.184 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:39.184 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:39.184 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:39.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:39.185 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:39.185 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:39.185 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:39.185 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:39.186 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:39.186 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:39.186 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:39.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:39.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:39.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:39.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:39.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:39.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:39.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:39.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:39.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:40.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:40.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:40.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:40.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:40.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:40.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:40.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:40.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:40.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588414 2021-01-15T18:39:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588414 '!=' null 2021-01-15T18:39:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:40.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:40.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:41.033 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:41.033 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:41.033 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:41.033 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:41.034 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:41.034 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:41.034 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:41.034 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:41.035 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:41.035 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:41.035 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:41.035 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:41.035 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:41.036 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:41.036 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:41.036 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:41.036 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:41.036 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:41.037 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:41.037 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:41.037 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:41.037 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:41.037 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:41.038 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:41.038 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:41.038 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:41.038 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:41.039 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:41.039 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:41.039 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:41.039 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:41.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:41.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:41.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:41.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:41.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:41.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:41.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:41.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:41.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:41.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:41.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:41.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:41.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:42.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:42.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:42.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:42.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:42.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:42.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:42.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:42.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:42.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:42.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599160 2021-01-15T18:39:42.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599160 '!=' null 2021-01-15T18:39:42.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:42.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:42.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:42.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:42.881 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:42.881 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:42.882 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:42.882 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:42.882 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:42.882 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:42.882 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:42.883 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:42.883 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:42.883 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:42.883 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:42.883 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:42.884 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:42.884 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:42.884 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:42.884 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:42.884 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:42.885 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:42.886 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:42.886 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:42.886 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:42.886 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:42.887 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:42.887 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:42.887 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:42.887 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:42.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:42.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:42.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:42.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:42.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:42.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:42.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:42.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:42.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:43.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:43.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:43.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:43.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:43.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:43.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:44.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599183 2021-01-15T18:39:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599183 '!=' null 2021-01-15T18:39:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:44.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:44.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:44.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:44.611 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:44.611 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:44.612 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:44.612 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:44.612 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:44.612 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:44.612 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:44.613 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:44.613 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:44.613 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:44.613 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:44.614 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:44.614 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:44.614 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:44.614 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:44.614 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:44.615 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:44.615 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:44.615 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:44.615 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:44.615 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:44.616 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:44.616 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:44.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:44.616 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:44.617 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:44.617 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:44.617 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:44.617 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:44.617 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:44.618 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:44.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:44.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:44.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:44.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:44.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:44.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:44.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:44.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:45.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:45.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:45.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:45.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:45.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:45.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:45.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:45.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:45.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:45.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:45.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:45.659 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:45.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:46.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24461881 2021-01-15T18:39:46.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24461881 '!=' null 2021-01-15T18:39:46.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:46.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:46.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:46.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:46.294 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:46.295 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:46.295 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:46.295 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:46.295 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:46.296 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:46.296 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:46.296 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:46.296 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:46.297 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:46.297 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:46.297 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:46.298 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:46.298 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:46.298 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:46.298 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:46.298 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:46.299 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:46.299 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:46.299 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:46.299 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:46.300 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:46.300 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:46.300 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:46.300 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:46.300 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:46.301 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:46.301 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:46.301 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:46.301 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:46.301 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:46.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:46.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:46.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:46.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:46.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:46.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:46.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:46.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:46.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:46.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:46.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:46.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:46.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:47.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:47.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:47.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:47.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:47.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:47.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:47.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725460 2021-01-15T18:39:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725460 '!=' null 2021-01-15T18:39:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:47.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:47.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:47.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:47.985 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:47.985 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:47.986 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:47.986 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:47.986 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:47.987 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:47.987 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:47.988 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:47.988 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:47.989 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:47.989 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:47.990 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:47.990 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:47.990 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:47.991 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:47.991 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:47.992 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:47.992 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:47.992 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:47.993 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:47.993 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:47.993 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:47.994 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:47.994 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:47.995 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:47.995 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:47.996 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:47.996 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:47.996 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:47.997 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:47.998 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:47.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:48.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:48.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:48.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:48.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:48.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:48.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:48.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:48.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:48.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:48.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:48.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:48.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:49.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:49.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:49.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:49.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:49.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:49.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:50.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725460 2021-01-15T18:39:50.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725460 '!=' null 2021-01-15T18:39:50.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:50.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:50.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:50.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:50.198 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:50.199 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:50.199 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:50.199 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:50.199 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:50.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:50.200 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:50.200 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:50.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:50.200 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:50.201 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:50.201 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:50.201 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:50.201 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:50.201 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:50.202 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:50.202 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:50.202 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:50.202 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:50.203 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:50.203 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:50.203 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:50.203 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:50.203 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:50.204 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:50.205 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:50.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:50.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:50.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:50.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:50.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:50.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:50.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:50.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:50.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:50.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:50.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:50.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:50.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:51.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:51.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:51.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:51.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:51.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599038 2021-01-15T18:39:51.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599038 '!=' null 2021-01-15T18:39:51.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:51.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:51.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:51.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:51.983 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:51.984 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:51.985 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:51.986 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:51.987 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:51.987 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:51.987 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:51.987 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:51.987 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:51.988 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:51.988 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:51.988 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:51.988 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:51.988 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:51.989 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:51.989 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:51.989 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:51.989 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:51.989 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:51.990 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:51.990 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:51.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:52.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:52.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:52.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:52.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:52.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:52.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:52.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:53.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:53.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:53.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:53.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:53.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:53.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:53.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:53.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599038 2021-01-15T18:39:53.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599038 '!=' null 2021-01-15T18:39:53.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:53.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:53.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:53.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:53.730 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:53.730 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:53.730 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:53.731 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:53.731 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:53.731 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:53.731 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:53.731 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:53.732 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:53.732 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:53.732 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:53.732 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:53.732 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:53.733 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:53.733 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:53.733 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:53.733 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:53.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:53.734 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:53.734 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:53.734 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:53.734 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:53.734 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:53.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:53.735 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:53.735 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:53.735 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:53.735 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:53.736 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:53.736 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:53.736 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:53.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:53.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:53.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:53.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:53.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:53.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:54.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:54.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:54.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:54.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:54.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:54.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:54.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:54.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:54.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:54.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:55.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599181 2021-01-15T18:39:55.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599181 '!=' null 2021-01-15T18:39:55.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:55.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:55.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:55.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:55.375 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:55.376 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:55.377 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:55.378 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:55.379 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:55.380 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:55.381 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:55.382 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:55.382 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:55.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:55.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:55.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:55.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:55.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:55.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:55.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:55.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:55.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:55.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:55.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:55.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:56.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:56.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:56.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:56.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:56.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:56.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:57.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=24462104 2021-01-15T18:39:57.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 24462104 '!=' null 2021-01-15T18:39:57.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:57.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:57.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:57.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:57.215 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:57.215 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:57.216 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:57.216 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:57.216 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:57.216 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:57.216 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:57.217 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:57.217 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:57.217 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:57.217 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:57.217 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:57.218 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:57.218 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:57.218 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:57.218 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:57.218 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:57.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:57.219 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:57.219 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:57.219 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:57.219 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:57.220 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:57.220 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:57.220 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:57.220 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:57.220 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:57.221 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:57.221 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:57.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:57.221 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:57.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:57.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:57.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:57.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:57.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:57.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:57.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:57.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:57.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:57.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:57.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:57.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:57.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:39:58.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:39:58.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:39:58.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:39:58.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:39:58.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:39:58.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:39:58.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:39:58.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:39:58.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:39:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16599103 2021-01-15T18:39:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16599103 '!=' null 2021-01-15T18:39:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:39:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:58.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:39:58.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:39:59.092 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:39:59.092 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:39:59.092 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:59.093 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:59.093 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:39:59.093 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:59.093 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:39:59.093 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:39:59.094 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:59.094 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:39:59.094 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:59.094 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:59.094 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:59.095 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:59.095 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:59.095 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:39:59.095 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:39:59.095 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:39:59.096 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:39:59.096 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:39:59.096 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:39:59.096 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:59.096 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:39:59.097 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:39:59.097 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:39:59.097 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:39:59.097 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:39:59.097 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:39:59.098 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:39:59.098 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:39:59.098 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:39:59.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:39:59.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:39:59.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:39:59.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:39:59.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:39:59.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:39:59.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:39:59.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:39:59.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:39:59.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:39:59.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:39:59.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:39:59.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:00.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:00.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:00.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:00.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:00.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:00.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:00.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:00.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:00.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714694 2021-01-15T18:40:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714694 '!=' null 2021-01-15T18:40:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:00.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:00.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:00.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:00.789 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:00.790 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:00.790 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:00.790 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:00.790 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:00.791 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:00.791 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:00.791 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:00.791 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:00.791 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:00.792 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:00.792 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:00.792 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:00.792 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:00.792 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:00.793 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:00.793 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:00.793 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:00.793 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:00.793 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:00.794 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:00.794 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:00.794 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:00.794 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:00.794 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:00.795 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:00.795 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:00.795 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:00.795 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:00.795 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:00.796 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:00.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:00.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:00.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:00.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:00.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:00.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:00.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:00.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:00.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:01.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:01.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:01.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:01.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:02.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:02.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:02.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:02.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:02.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:02.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:02.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:02.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:02.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:02.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978280 2021-01-15T18:40:02.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978280 '!=' null 2021-01-15T18:40:02.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:02.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:02.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:02.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:02.840 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:02.841 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:02.841 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:02.841 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:02.841 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:02.841 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:02.842 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:02.842 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:02.842 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:02.842 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:02.843 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:02.843 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:02.843 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:02.843 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:02.843 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:02.844 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:02.845 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:02.845 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:02.845 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:02.845 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:02.845 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:02.846 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:02.846 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:02.846 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:02.846 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:02.846 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:02.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:02.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:02.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:02.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:02.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:02.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:02.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:02.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:02.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:03.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:03.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:03.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:03.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:03.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:03.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:03.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:03.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:03.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:03.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:04.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978154 2021-01-15T18:40:04.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978154 '!=' null 2021-01-15T18:40:04.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:04.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:04.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:04.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:04.509 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:04.509 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:04.510 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:04.510 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:04.511 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:04.511 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:04.511 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:04.512 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:04.512 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:04.513 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:04.513 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:04.513 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:04.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:04.514 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:04.514 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:04.515 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:04.515 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:04.516 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:04.516 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:04.517 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:04.517 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:04.518 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:04.518 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:04.518 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:04.519 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:04.519 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:04.520 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:04.520 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:04.520 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:04.521 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:04.521 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:04.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:04.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:04.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:04.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:04.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:04.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:04.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:04.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:04.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:04.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:04.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:05.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:05.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:05.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:05.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:05.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:06.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714359 2021-01-15T18:40:06.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714359 '!=' null 2021-01-15T18:40:06.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:06.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:06.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:06.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:06.193 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:06.194 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:06.194 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:06.194 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:06.195 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:06.195 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:06.195 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:06.195 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:06.195 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:06.196 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:06.196 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:06.196 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:06.196 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:06.196 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:06.197 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:06.197 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:06.197 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:06.197 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:06.197 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:06.198 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:06.198 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:06.198 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:06.198 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:06.198 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:06.199 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:06.199 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:06.199 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:06.199 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:06.199 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:06.200 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:06.200 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:06.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:06.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:06.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:06.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:06.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:06.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:06.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:06.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:06.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:06.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:07.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:07.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:07.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:07.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:07.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:07.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714359 2021-01-15T18:40:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714359 '!=' null 2021-01-15T18:40:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:07.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:07.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:07.939 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:07.939 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:07.940 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:07.940 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:07.940 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:07.940 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:07.940 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:07.941 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:07.941 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:07.941 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:07.941 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:07.941 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:07.942 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:07.942 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:07.942 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:07.942 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:07.942 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:07.943 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:07.943 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:07.943 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:07.943 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:07.943 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:07.944 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:07.944 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:07.944 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:07.944 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:07.944 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:07.945 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:07.945 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:07.945 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:07.945 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:08.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:08.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:08.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:08.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:08.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:08.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:08.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:08.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:08.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:08.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:08.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:08.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:08.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:08.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:08.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:08.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:09.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851518 2021-01-15T18:40:09.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851518 '!=' null 2021-01-15T18:40:09.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:09.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:09.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:09.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:09.633 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:09.633 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:09.634 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:09.634 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:09.634 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:09.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:09.635 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:09.635 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:09.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:09.635 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:09.636 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:09.636 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:09.636 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:09.636 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:09.637 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:09.637 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:09.637 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:09.637 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:09.637 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:09.638 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:09.638 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:09.638 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:09.638 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:09.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:09.639 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:09.639 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:09.639 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:09.639 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:09.640 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:09.640 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:09.640 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:09.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:09.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:09.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:09.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:09.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:09.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:09.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:09.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:09.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:10.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:10.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:10.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:10.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:10.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:10.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:10.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:10.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:10.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:10.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:10.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:10.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:11.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840537 2021-01-15T18:40:11.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840537 '!=' null 2021-01-15T18:40:11.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:11.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:11.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:11.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:11.580 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:11.580 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:11.580 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:11.580 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:11.581 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:11.581 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:11.581 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:11.581 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:11.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:11.582 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:11.582 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:11.582 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:11.582 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:11.583 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:11.584 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:11.584 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:11.584 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:11.584 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:11.584 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:11.585 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:11.585 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:11.585 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:11.585 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:11.586 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:11.586 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:11.586 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:11.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:11.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:11.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:11.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:11.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:11.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:11.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:11.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:12.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:12.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:12.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:12.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:12.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:12.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:12.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:12.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:12.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:13.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725281 2021-01-15T18:40:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725281 '!=' null 2021-01-15T18:40:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:13.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:13.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:13.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:13.384 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:13.384 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:13.384 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:13.385 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:13.385 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [ 2021-01-15T18:40:13.385 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:13.385 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 190, 2021-01-15T18:40:13.385 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T18:40:13.386 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:13.386 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:13.386 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:13.386 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:13.387 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:13.388 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "1.0", 2021-01-15T18:40:13.388 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:13.388 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:13.388 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:13.388 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:13.389 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:13.389 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:13.389 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:13.389 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:13.389 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:13.390 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:13.390 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:13.390 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:13.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:13.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:13.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:13.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:13.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:13.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:13.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:13.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:13.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:13.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:13.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:13.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:13.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:14.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:14.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:14.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:14.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:14.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:14.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:14.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:14.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:14.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:15.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725242 2021-01-15T18:40:15.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725242 '!=' null 2021-01-15T18:40:15.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:15.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:15.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:15.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:15.218 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:15.219 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:15.219 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:15.219 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:15.219 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:15.220 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:15.220 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:15.220 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:15.220 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:15.221 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:15.221 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:15.221 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:15.221 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:15.221 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:15.222 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:15.222 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:15.222 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:15.222 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:15.223 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:15.223 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:15.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:15.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:15.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:15.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:15.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:15.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:15.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:15.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:15.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:15.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:15.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:15.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:15.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:16.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:16.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:16.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:16.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:16.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:16.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:17.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13628595 2021-01-15T18:40:17.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13628595 '!=' null 2021-01-15T18:40:17.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:17.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:17.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:17.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:17.659 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:17.663 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:17.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:17.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:17.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:17.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:17.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:17.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:17.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:17.770 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:17.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:18.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:40:18.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:18.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:18.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:18.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 3 2021-01-15T18:40:18.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 1 '!=' 1 2021-01-15T18:40:18.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:18.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:18.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:18.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:18.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:18.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:18.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6552185 2021-01-15T18:40:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6552185 '!=' null 2021-01-15T18:40:19.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:19.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:19.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:19.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:19.465 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:19.465 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:19.466 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:19.466 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:19.466 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:19.466 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:19.466 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:19.467 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:19.467 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:19.467 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:19.467 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:19.467 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:19.468 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:19.468 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:19.468 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:19.468 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:19.468 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:19.469 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:19.469 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:19.469 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:19.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:19.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:19.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:19.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:19.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:19.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:19.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:19.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:19.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:19.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:19.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:19.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:19.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:20.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:20.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 1 2021-01-15T18:40:20.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:40:20.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=2 2021-01-15T18:40:20.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:20.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:20.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:20.573 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:20.574 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:20.574 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:20.574 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:20.574 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:20.575 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:20.575 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:20.575 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:20.576 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:20.576 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:20.576 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:20.576 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:20.577 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:20.577 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:20.577 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:20.577 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:20.578 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:20.578 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:20.578 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:20.578 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:20.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:20.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:20.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:20.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:20.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:20.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:20.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:20.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:20.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:21.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:21.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:21.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:21.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:21.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:21.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:22.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19656776 2021-01-15T18:40:22.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19656776 '!=' null 2021-01-15T18:40:22.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:22.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:22.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:22.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:22.446 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:22.446 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:22.446 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:22.446 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:22.447 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:22.447 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:22.447 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:22.447 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:22.448 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:22.448 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:22.448 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:22.448 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:22.448 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:22.449 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:22.449 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:22.449 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:22.449 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:22.449 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:22.450 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:22.450 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:22.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:22.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:22.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:22.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:22.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:22.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:22.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:22.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:22.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:23.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:23.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:23.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:23.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:23.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:23.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:23.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:24.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12140944 2021-01-15T18:40:24.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12140944 '!=' null 2021-01-15T18:40:24.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:24.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:24.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:24.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:24.381 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:24.381 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:24.382 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:24.382 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:24.382 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:24.382 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:24.383 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:24.383 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:24.383 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:24.383 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:24.383 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:24.384 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:24.384 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:24.384 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:24.384 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:24.384 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:24.385 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:24.385 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:24.385 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:24.385 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:24.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:24.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:24.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:24.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:24.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:24.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:24.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:24.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:24.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:24.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:24.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:24.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:24.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:25.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:25.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:25.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:25.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:25.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:25.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:25.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:26.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18870904 2021-01-15T18:40:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18870904 '!=' null 2021-01-15T18:40:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:26.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:26.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:26.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:26.497 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:26.497 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:26.498 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:26.498 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:26.498 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:26.498 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:26.499 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:26.499 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:26.499 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:26.499 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:26.499 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:26.500 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:26.500 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:26.500 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:26.500 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:26.500 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:26.501 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:26.501 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:26.501 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:26.501 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:26.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:26.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:26.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:26.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:26.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:26.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:26.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:26.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:26.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:26.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:26.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:26.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:27.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:27.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:27.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:27.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15754505 2021-01-15T18:40:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15754505 '!=' null 2021-01-15T18:40:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:28.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:28.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:28.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:28.246 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:28.246 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:28.246 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:28.247 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:28.247 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:28.247 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:28.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:28.247 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:28.248 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:28.249 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:28.250 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:28.250 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:28.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:28.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:28.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:28.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:28.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:28.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:28.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:28.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:28.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:28.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:28.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:28.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:28.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:29.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:29.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:29.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:29.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:29.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588910 2021-01-15T18:40:30.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588910 '!=' null 2021-01-15T18:40:30.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:30.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:30.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:30.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:30.234 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:30.235 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:30.236 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:30.237 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:30.238 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:30.238 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:30.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:30.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:30.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:30.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:30.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:30.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:30.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:30.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:30.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:30.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:30.733 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:31.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:31.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:31.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:31.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:31.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:31.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852191 2021-01-15T18:40:31.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852191 '!=' null 2021-01-15T18:40:31.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:31.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:31.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:31.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:31.909 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:31.909 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:31.909 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:31.910 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:31.910 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:31.910 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:31.911 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:31.911 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:31.911 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:31.911 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:31.911 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:31.912 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:31.912 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:31.912 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:31.912 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:31.913 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:31.913 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:31.913 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:31.913 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:31.913 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:31.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:32.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:32.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:32.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:32.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:32.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:32.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:32.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:32.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:32.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:32.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:32.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:32.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:32.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:32.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:32.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:32.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:32.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:32.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:32.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:32.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852191 2021-01-15T18:40:33.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852191 '!=' null 2021-01-15T18:40:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:33.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:33.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:33.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:33.629 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:33.630 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:33.630 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:33.630 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:33.630 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:33.631 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:33.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:33.631 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:33.631 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:33.631 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:33.632 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:33.632 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:33.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:33.632 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:33.632 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:33.633 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:33.633 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:33.633 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:33.633 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:33.633 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:33.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:33.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:33.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:33.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:33.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:34.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:34.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:34.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:34.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:34.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:34.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:34.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:34.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:34.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:34.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:34.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:34.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:34.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:35.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852187 2021-01-15T18:40:35.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852187 '!=' null 2021-01-15T18:40:35.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:35.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:35.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:35.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:35.723 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:35.724 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:35.724 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:35.724 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:35.724 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:35.724 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:35.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:35.725 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:35.725 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:35.725 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:35.725 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:35.726 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:35.726 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:35.726 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:35.726 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:35.727 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:35.727 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:35.727 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:35.727 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:35.727 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:35.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:35.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:35.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:35.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:35.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:35.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:35.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:35.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:35.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:36.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:36.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:36.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:36.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:36.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:36.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:36.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:36.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:36.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714972 2021-01-15T18:40:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714972 '!=' null 2021-01-15T18:40:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:37.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:37.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:37.370 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:37.370 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:37.370 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:37.371 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:37.371 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:37.371 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:37.371 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:37.371 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:37.372 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:37.373 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:37.373 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:37.373 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:37.373 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:37.373 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:37.374 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:37.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:37.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:37.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:37.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:37.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:37.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:37.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:37.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:37.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:37.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:37.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:37.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:37.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:38.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:38.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:38.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:38.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:38.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:38.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:38.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:38.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:38.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:39.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725532 2021-01-15T18:40:39.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725532 '!=' null 2021-01-15T18:40:39.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:39.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:39.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:39.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:39.623 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:39.624 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:39.624 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:39.624 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:39.624 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:39.624 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:39.625 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:39.625 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:39.625 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:39.625 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:39.625 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:39.626 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:39.626 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:39.626 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:39.626 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:39.627 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:39.627 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:39.627 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:39.627 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:39.627 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:39.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:39.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:39.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:39.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:39.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:39.734 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:39.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:39.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:39.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:40.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:40.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:40.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:40.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:40.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:40.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:40.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:40.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:40.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:40.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:40.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:40.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:40.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:41.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714693 2021-01-15T18:40:41.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714693 '!=' null 2021-01-15T18:40:41.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:41.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:41.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:41.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:41.264 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:41.264 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:41.264 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:41.265 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:41.265 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:41.265 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:41.265 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:41.265 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:41.266 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:41.266 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:41.266 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:41.266 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:41.267 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:41.267 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:41.267 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:41.267 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:41.267 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:41.268 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:41.268 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:41.268 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:41.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:41.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:41.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:41.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:41.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:41.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:41.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:41.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:41.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:41.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:41.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:41.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:41.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:42.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:42.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:42.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:42.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:42.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:42.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:42.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:42.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:42.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:42.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851908 2021-01-15T18:40:42.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851908 '!=' null 2021-01-15T18:40:42.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:42.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:42.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:42.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:43.029 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:43.029 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:43.029 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:43.029 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:43.030 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:43.030 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:43.030 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:43.030 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:43.030 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:43.031 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:43.031 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:43.031 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:43.031 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:43.031 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:43.032 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:43.032 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:43.032 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:43.032 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:43.032 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:43.033 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:43.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:43.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:43.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:43.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:43.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:43.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:43.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:43.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:43.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:43.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:44.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:44.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:44.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:44.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:44.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:44.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:44.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851953 2021-01-15T18:40:44.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851953 '!=' null 2021-01-15T18:40:44.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:44.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:44.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:44.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:44.695 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:44.696 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:44.696 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:44.696 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:44.696 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:44.697 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:44.697 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:44.697 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:44.697 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:44.697 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:44.698 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:44.698 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:44.698 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:44.698 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:44.699 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:44.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:44.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:44.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:44.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:44.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:44.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:44.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:44.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:44.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:45.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:45.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:45.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:45.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:45.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:45.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:45.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:45.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:45.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:45.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:46.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588258 2021-01-15T18:40:46.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588258 '!=' null 2021-01-15T18:40:46.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:46.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:46.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:46.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:46.468 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:46.468 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:46.469 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:46.469 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:46.469 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:46.469 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:46.470 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:46.470 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:46.470 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:46.470 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:46.470 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:46.471 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:46.471 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:46.471 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:46.471 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:46.472 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:46.472 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:46.472 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:46.472 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:46.472 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:46.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:46.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:46.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:46.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:46.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:46.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:46.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:46.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:46.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:46.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:46.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:47.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:47.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:47.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:47.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:47.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:47.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:47.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:47.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:47.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725497 2021-01-15T18:40:48.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725497 '!=' null 2021-01-15T18:40:48.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:48.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:48.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:48.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:48.414 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:48.414 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:48.414 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:48.414 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:48.415 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:48.415 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:48.415 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:48.415 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:48.415 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:48.416 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:48.416 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:48.416 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:48.416 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:48.416 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:48.417 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:48.417 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:48.417 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:48.417 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:48.417 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:48.418 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:48.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:48.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:48.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:48.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:48.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:48.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:48.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:48.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:48.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:48.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:48.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:48.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:48.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:49.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:49.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:49.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:49.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:49.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:49.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:50.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588182 2021-01-15T18:40:50.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588182 '!=' null 2021-01-15T18:40:50.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:50.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:50.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:50.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:50.170 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:50.170 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:50.170 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:50.171 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:50.171 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:50.171 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:50.171 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:50.172 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:50.172 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:50.172 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:50.172 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:50.172 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:50.173 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:50.173 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:50.173 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:50.173 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:50.173 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:50.174 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:50.174 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:50.174 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:50.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:50.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:50.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:50.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:50.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:50.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:50.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:50.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:50.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:50.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:50.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:50.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:50.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:51.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:51.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:51.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:51.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:51.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:51.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598982 2021-01-15T18:40:51.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598982 '!=' null 2021-01-15T18:40:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:51.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:51.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:51.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:52.047 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:52.047 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:52.047 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:52.047 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:52.048 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:52.048 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:52.048 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:52.048 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:52.048 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:52.049 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:52.049 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:52.049 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:52.049 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:52.049 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:52.050 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:52.050 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:52.050 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:52.050 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:52.051 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:52.051 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:52.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:52.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:52.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:52.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:52.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:52.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:52.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:52.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:52.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:52.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:53.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:53.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:53.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:53.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:53.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:53.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:53.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:53.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:53.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:53.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=16598982 2021-01-15T18:40:53.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 16598982 '!=' null 2021-01-15T18:40:53.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:53.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:53.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:53.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:54.060 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:54.061 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:54.061 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:54.061 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:54.061 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:54.062 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:54.062 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:54.062 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:54.062 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:54.062 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:54.063 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:54.063 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:54.063 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:54.063 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:54.063 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:54.064 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:54.064 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:54.064 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:54.064 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:54.065 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:54.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:54.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:54.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:54.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:54.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:54.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:54.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:54.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:55.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:55.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:55.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:55.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:55.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:55.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21840815 2021-01-15T18:40:55.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21840815 '!=' null 2021-01-15T18:40:55.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:55.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:55.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:55.940 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:56.041 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:56.114 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:56.114 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:56.114 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:56.115 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:56.115 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:56.115 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:56.115 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:56.115 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:56.116 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:56.116 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:56.116 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:56.116 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:56.116 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:56.117 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:56.117 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:56.117 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:56.117 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:56.118 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:56.118 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:56.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:56.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:56.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:56.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:56.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:56.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:56.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:56.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:56.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:56.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:56.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:56.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:56.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:57.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:57.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:57.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:57.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:57.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:57.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104571 2021-01-15T18:40:57.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104571 '!=' null 2021-01-15T18:40:57.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:57.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:57.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:57.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:57.860 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:57.861 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:57.861 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:57.861 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:57.861 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:57.861 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:57.862 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:57.862 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:57.862 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:57.862 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:57.862 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:57.863 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:57.863 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:57.863 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:57.863 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:57.863 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:57.864 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:57.864 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:57.864 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:57.864 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:57.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:57.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:57.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:57.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:57.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:57.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:57.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:57.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:57.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:40:58.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:40:58.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:40:58.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:40:58.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:40:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:40:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:40:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:40:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:40:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:40:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:40:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:40:58.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:40:58.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:40:59.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104571 2021-01-15T18:40:59.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104571 '!=' null 2021-01-15T18:40:59.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:40:59.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:59.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:40:59.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:40:59.529 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:40:59.530 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:40:59.530 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:59.530 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:59.531 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:59.531 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:40:59.531 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:40:59.531 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:40:59.531 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:40:59.532 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:40:59.532 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:59.532 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:40:59.532 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:40:59.532 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:40:59.533 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:40:59.533 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:40:59.533 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:40:59.533 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:40:59.533 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:40:59.534 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:40:59.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:40:59.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:40:59.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:40:59.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:40:59.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:40:59.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:40:59.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:40:59.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:40:59.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:00.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:00.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:00.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:00.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:01.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:01.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:01.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:01.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:01.014 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:01.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967381 2021-01-15T18:41:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967381 '!=' null 2021-01-15T18:41:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:01.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:01.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:01.642 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:01.643 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:01.643 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:01.643 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:01.643 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:01.643 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:01.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:01.644 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:01.644 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:01.644 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:01.644 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:01.645 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:01.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:01.645 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:01.645 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:01.645 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:01.646 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:01.646 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:01.646 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:01.646 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:01.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:01.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:01.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:01.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:01.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:01.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:01.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:01.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:01.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:02.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:02.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:02.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:02.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:02.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:02.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:02.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:02.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:02.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104819 2021-01-15T18:41:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104819 '!=' null 2021-01-15T18:41:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:03.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:03.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:03.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:03.841 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:03.842 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:03.842 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:03.842 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:03.842 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:03.843 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:03.843 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:03.843 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:03.843 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:03.843 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:03.844 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:03.844 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:03.844 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:03.844 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:03.844 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:03.845 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:03.845 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:03.845 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:03.845 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:03.845 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:03.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:03.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:03.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:03.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:03.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:03.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:03.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:03.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:03.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:04.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:04.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:04.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:04.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:04.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:04.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:04.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:05.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104967 2021-01-15T18:41:05.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104967 '!=' null 2021-01-15T18:41:05.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:05.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:05.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:05.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:05.600 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:05.600 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:05.600 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:05.600 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:05.601 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:05.601 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:05.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:05.601 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:05.601 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:05.602 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:05.602 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:05.602 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:05.602 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:05.602 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:05.603 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:05.606 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:05.606 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:05.606 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:05.606 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:05.606 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:05.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:05.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:05.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:05.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:05.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:05.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:05.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:05.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:05.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:06.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:06.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:06.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:06.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:06.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:06.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:06.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:06.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:06.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:06.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:06.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:06.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:06.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:07.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20967962 2021-01-15T18:41:07.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20967962 '!=' null 2021-01-15T18:41:07.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:07.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:07.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:07.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:07.348 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:07.348 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:07.348 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:07.349 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:07.349 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:07.349 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:07.350 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:07.350 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:07.351 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:07.351 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:07.351 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:07.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:07.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:07.353 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:07.353 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:07.353 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:07.353 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:07.354 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:07.354 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:07.354 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:07.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:07.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:07.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:07.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:07.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:07.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:07.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:07.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:07.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:07.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:07.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:07.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:07.843 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:08.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:08.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:08.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:08.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:08.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:09.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725974 2021-01-15T18:41:09.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725974 '!=' null 2021-01-15T18:41:09.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:09.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:09.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:09.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:09.245 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:09.246 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:09.246 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:09.246 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:09.246 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:09.247 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:09.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:09.247 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:09.247 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:09.247 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:09.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:09.248 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:09.248 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:09.248 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:09.248 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:09.249 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:09.249 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:09.249 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:09.249 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:09.249 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:09.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:09.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:09.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:09.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:09.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:09.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:09.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:09.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:09.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:10.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:10.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:10.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:10.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:10.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:10.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:10.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715105 2021-01-15T18:41:10.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715105 '!=' null 2021-01-15T18:41:10.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:10.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:10.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:10.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:10.975 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:10.975 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:10.975 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:10.976 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:10.976 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:10.976 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:10.976 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:10.976 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:10.977 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:10.977 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:10.977 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:10.977 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:10.977 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:10.978 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:10.978 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:10.978 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:10.978 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:10.978 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:10.979 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:10.979 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:10.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:11.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:11.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:11.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:11.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:11.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:11.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:11.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:11.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:11.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:11.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:11.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:12.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:12.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:12.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:12.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:12.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:12.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:12.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:12.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:12.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:12.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852075 2021-01-15T18:41:12.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852075 '!=' null 2021-01-15T18:41:12.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:12.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:12.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:12.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:12.678 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:12.678 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:12.678 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:12.679 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:12.679 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:12.679 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:12.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:12.679 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:12.680 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:12.681 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:12.681 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:12.681 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:12.681 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:12.682 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:12.682 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:12.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:12.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:12.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:12.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:12.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:12.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:13.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:13.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:13.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:13.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:13.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:13.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:13.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:13.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:13.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:13.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:13.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:13.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:13.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:14.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851928 2021-01-15T18:41:14.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851928 '!=' null 2021-01-15T18:41:14.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:14.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:14.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:14.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:14.313 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:14.313 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:14.313 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:14.313 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:14.314 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:14.315 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:14.315 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:14.315 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:14.315 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:14.315 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:14.316 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:14.316 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:14.316 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:14.316 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:14.316 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:14.317 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:14.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:14.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:14.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:14.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:14.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:14.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:14.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:14.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:14.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:14.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:14.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:14.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:14.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:15.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:15.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:15.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714601 2021-01-15T18:41:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714601 '!=' null 2021-01-15T18:41:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:16.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:16.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:16.301 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:16.301 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:16.301 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:16.302 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:16.302 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:16.302 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:16.302 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:16.302 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:16.303 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:16.303 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:16.303 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:16.303 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:16.304 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:16.305 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:16.305 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:16.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:16.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:16.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:16.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:16.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:16.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:16.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:16.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:16.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:16.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:17.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:17.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:17.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:17.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:17.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:17.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:17.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:17.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:17.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:17.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851593 2021-01-15T18:41:17.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851593 '!=' null 2021-01-15T18:41:17.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:17.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:17.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:17.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:18.015 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:18.015 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:18.016 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:18.016 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:18.016 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:18.016 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:18.017 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:18.017 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:18.017 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:18.017 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:18.018 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:18.018 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:18.018 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:18.018 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:18.019 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:18.019 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:18.019 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:18.019 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:18.020 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:18.020 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:18.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:18.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:18.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:18.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:18.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:18.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:18.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:18.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:18.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:18.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:19.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:19.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:19.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:19.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:19.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:19.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:19.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14851593 2021-01-15T18:41:19.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14851593 '!=' null 2021-01-15T18:41:19.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:19.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:19.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:19.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:19.662 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:19.662 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:19.663 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:19.664 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:19.665 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:19.666 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:19.666 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:19.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:19.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:19.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:19.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:19.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:19.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:19.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:19.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:19.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:20.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:20.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:20.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:20.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:20.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:21.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714223 2021-01-15T18:41:21.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714223 '!=' null 2021-01-15T18:41:21.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:21.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:21.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:21.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:21.588 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:21.589 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:21.589 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:21.589 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:21.589 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:21.590 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:21.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:21.590 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:21.590 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:21.590 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:21.591 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:21.591 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:21.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:21.591 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:21.591 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:21.592 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:21.592 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:21.592 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:21.592 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:21.592 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:21.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:21.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:21.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:21.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:21.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:21.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:21.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:21.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:22.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:22.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:22.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:22.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:22.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:22.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:22.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:22.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:22.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725323 2021-01-15T18:41:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725323 '!=' null 2021-01-15T18:41:23.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:23.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:23.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:23.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:23.249 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:23.250 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:23.250 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:23.250 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:23.250 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:23.250 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:23.251 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:23.251 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:23.251 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:23.251 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:23.252 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:23.253 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:23.253 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:23.253 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:23.253 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:23.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:23.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:23.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:23.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:23.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:23.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:23.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:23.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:23.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:23.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:23.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:23.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:23.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:24.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:24.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:24.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:24.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:24.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:24.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:24.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:24.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:24.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725440 2021-01-15T18:41:24.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725440 '!=' null 2021-01-15T18:41:24.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:24.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:24.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:24.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:24.923 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:24.923 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:24.924 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:24.924 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:24.924 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:24.924 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:24.924 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:24.925 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:24.925 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:24.925 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:24.925 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:24.925 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:24.926 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:24.927 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:24.927 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:24.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:25.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:25.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:25.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:25.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:25.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:25.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:25.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:25.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:25.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:25.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:25.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:25.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:25.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:25.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:25.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:25.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:25.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:25.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:25.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:25.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:25.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588448 2021-01-15T18:41:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588448 '!=' null 2021-01-15T18:41:26.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:26.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:26.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:26.583 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:26.583 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:26.584 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:26.584 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:26.584 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:26.584 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:26.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:26.585 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:26.585 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:26.585 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:26.586 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:26.586 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:26.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:26.586 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:26.587 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:26.587 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:26.587 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:26.587 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:26.587 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:26.588 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:26.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:26.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:26.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:26.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:26.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:26.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:26.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:26.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:26.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:27.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:27.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:27.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:27.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:27.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:27.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:27.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:27.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:27.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:27.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:27.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:28.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725681 2021-01-15T18:41:28.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725681 '!=' null 2021-01-15T18:41:28.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:28.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:28.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:28.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:28.528 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:28.529 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:28.529 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:28.529 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:28.529 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:28.530 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:28.530 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:28.530 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:28.530 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:28.530 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:28.531 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:28.531 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:28.531 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:28.531 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:28.531 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:28.532 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:28.532 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:28.532 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:28.532 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:28.532 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:28.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:28.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:28.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:28.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:28.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:28.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:28.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:28.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:28.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:29.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:29.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:29.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:29.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:29.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:29.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:29.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:29.545 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:30.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23588670 2021-01-15T18:41:30.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23588670 '!=' null 2021-01-15T18:41:30.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:30.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:30.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:30.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:30.220 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:30.220 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:30.221 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:30.222 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:30.223 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:30.224 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:30.224 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:30.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:30.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:30.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:30.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:30.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:30.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:30.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:30.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:30.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:30.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:30.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:30.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:30.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:31.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:31.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:31.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:31.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:31.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:31.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:31.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:31.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:31.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725761 2021-01-15T18:41:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725761 '!=' null 2021-01-15T18:41:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:31.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:31.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:31.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:31.975 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:31.975 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:31.976 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:31.976 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:31.976 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:31.976 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:31.976 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:31.977 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:31.977 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:31.977 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:31.977 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:31.977 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:31.978 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:31.978 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:31.978 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:31.979 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:31.979 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:31.979 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:31.979 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:31.979 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:31.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:32.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:32.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:32.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:32.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:32.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:32.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:32.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:32.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:32.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:32.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:32.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:32.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:33.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:33.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:33.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:33.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:33.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:33.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725761 2021-01-15T18:41:33.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725761 '!=' null 2021-01-15T18:41:33.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:33.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:33.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:33.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:33.977 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:33.978 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:33.978 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:33.978 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:33.978 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:33.979 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:33.979 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:33.979 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:33.979 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:33.980 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:33.980 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:33.980 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:33.980 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:33.980 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:33.981 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:33.981 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:33.981 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:33.981 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:33.982 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:33.982 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:33.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:34.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:34.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:34.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:34.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:34.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:34.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:34.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:34.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:34.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:34.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:34.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:34.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:35.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:35.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:35.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725769 2021-01-15T18:41:35.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725769 '!=' null 2021-01-15T18:41:35.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:35.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:35.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:35.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:35.678 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:35.679 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:35.679 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:35.679 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:35.679 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:35.680 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:35.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:35.680 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:35.680 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:35.681 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:35.681 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:35.681 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:35.681 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:35.681 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:35.682 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:35.682 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:35.682 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:35.682 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:35.683 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:35.683 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:35.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:35.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:35.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:35.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:36.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:36.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:36.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:36.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:37.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:37.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:37.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:37.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:37.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:37.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:37.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22715019 2021-01-15T18:41:37.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22715019 '!=' null 2021-01-15T18:41:37.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:37.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:37.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:37.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:37.738 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:37.739 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:37.739 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:37.739 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:37.740 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:37.740 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:37.740 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:37.741 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:37.741 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:37.741 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:37.742 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:37.742 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:37.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:37.743 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:37.743 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:37.743 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:37.744 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:37.744 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:37.744 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:37.744 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:37.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:37.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:37.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:37.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:37.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:37.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:37.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:37.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:37.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:38.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:38.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:38.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:38.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:38.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:38.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:38.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:38.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:38.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:38.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:38.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:38.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:38.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852116 2021-01-15T18:41:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852116 '!=' null 2021-01-15T18:41:39.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:39.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:39.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:39.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:39.375 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:39.376 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:39.377 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:39.377 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:39.377 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:39.377 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:39.378 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:39.378 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:39.378 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:39.378 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:39.378 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:39.379 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:39.379 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:39.379 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:39.379 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:39.379 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:39.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:39.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:39.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:39.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:39.486 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:39.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:39.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:39.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:39.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:39.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:40.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:40.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:40.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:40.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:40.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:40.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:40.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:40.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:40.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:40.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714805 2021-01-15T18:41:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714805 '!=' null 2021-01-15T18:41:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:40.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:40.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:40.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:41.021 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:41.022 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:41.022 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:41.022 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:41.022 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:41.023 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:41.023 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:41.023 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:41.023 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:41.023 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:41.024 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:41.024 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:41.024 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:41.024 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:41.024 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:41.025 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:41.025 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:41.025 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:41.025 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:41.025 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:41.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:41.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:41.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:41.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:41.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:41.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:41.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:41.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:41.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:41.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:41.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:41.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:41.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:42.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:42.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:42.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:42.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:42.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:42.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:42.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:42.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852112 2021-01-15T18:41:42.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852112 '!=' null 2021-01-15T18:41:42.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:42.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:42.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:42.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:43.101 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:43.101 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:43.101 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:43.102 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:43.102 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:43.102 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:43.102 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:43.102 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:43.103 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:43.103 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:43.103 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:43.103 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:43.103 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:43.104 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:43.104 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:43.104 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:43.104 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:43.104 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:43.105 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:43.105 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:43.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:43.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:43.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:43.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:43.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:43.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:43.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:43.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:43.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:43.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:43.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:43.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:43.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:44.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:44.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:44.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:44.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:44.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:44.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:44.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:44.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:44.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:44.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852122 2021-01-15T18:41:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852122 '!=' null 2021-01-15T18:41:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:44.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:44.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:44.858 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:44.859 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:44.859 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:44.859 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:44.859 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:44.860 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:44.860 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:44.860 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:44.860 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:44.861 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:44.861 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:44.861 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:44.861 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:44.861 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:44.862 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:44.862 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:44.862 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:44.862 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:44.862 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:44.863 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:44.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:44.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:44.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:44.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:44.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:44.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:45.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:45.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:45.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:45.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:46.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:46.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:46.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:46.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:46.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:46.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:46.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:46.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=22714832 2021-01-15T18:41:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 22714832 '!=' null 2021-01-15T18:41:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:46.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:46.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:46.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:46.793 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:46.793 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:46.794 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:46.794 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:46.795 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:46.795 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:46.795 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:46.796 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:46.796 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:46.796 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:46.797 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:46.797 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:46.798 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:46.798 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:46.798 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:46.799 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:46.799 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:46.800 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:46.800 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:46.800 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:46.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:46.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:46.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:46.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:46.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:46.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:46.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:46.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:46.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:47.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:47.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:47.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:47.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:47.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:47.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:47.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:47.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:48.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725606 2021-01-15T18:41:48.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725606 '!=' null 2021-01-15T18:41:48.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:48.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:48.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:48.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:48.492 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:48.492 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:48.492 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:48.493 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:48.493 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:48.493 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:48.493 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:48.493 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:48.494 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:48.494 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:48.494 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:48.495 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:48.495 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:48.495 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:48.495 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:48.495 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:48.496 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:48.496 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:48.496 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:48.496 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:48.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:48.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:48.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:48.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:48.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:48.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:48.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:48.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:48.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:49.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:49.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:49.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:49.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:49.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:49.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:49.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:49.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:49.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:49.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:49.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:49.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:49.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:50.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=21841139 2021-01-15T18:41:50.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 21841139 '!=' null 2021-01-15T18:41:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:50.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:50.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:50.868 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:50.869 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:50.869 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:50.869 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:50.869 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:50.870 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:50.870 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:50.870 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:50.870 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:50.870 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:50.871 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:50.871 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:50.871 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:50.871 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:50.871 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:50.872 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:50.872 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:50.872 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:50.872 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:50.872 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:50.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:50.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:50.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:50.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:50.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:50.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:50.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:50.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:50.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:51.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:51.423 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:51.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:51.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:52.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:52.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:52.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:52.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:52.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:52.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978303 2021-01-15T18:41:52.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978303 '!=' null 2021-01-15T18:41:52.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:52.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:52.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:52.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:52.968 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:52.968 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:52.969 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:52.969 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:52.969 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:52.969 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:52.970 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:52.970 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:52.970 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:52.970 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:52.970 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:52.971 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:52.971 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:52.971 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:52.971 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:52.971 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:52.972 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:52.972 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:52.972 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:52.972 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:52.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:53.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:53.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:53.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:53.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:53.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:53.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:53.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:53.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:53.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:53.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:54.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:54.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:54.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:54.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:54.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:54.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:54.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:54.012 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13978326 2021-01-15T18:41:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13978326 '!=' null 2021-01-15T18:41:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:54.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:54.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:54.632 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:54.633 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:54.633 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:54.633 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:54.633 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:54.633 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:54.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:54.634 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:54.634 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:54.634 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:54.634 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:54.635 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:54.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:54.635 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:54.635 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:54.635 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:54.636 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:54.636 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:54.636 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:54.636 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:54.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:54.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:54.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:54.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:54.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:54.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:54.744 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:54.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:54.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:55.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:55.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:55.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:55.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:55.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:55.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:55.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:55.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:55.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:55.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:55.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:55.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:55.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:56.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220211 2021-01-15T18:41:56.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220211 '!=' null 2021-01-15T18:41:56.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:56.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:56.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:56.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:56.272 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:56.273 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:56.273 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:56.273 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:56.273 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:56.274 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:56.274 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:56.274 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:56.274 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:56.274 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:56.275 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:56.275 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:56.275 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:56.275 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:56.275 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:56.276 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:56.276 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:56.276 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:56.276 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:56.276 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:56.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:56.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:56.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:56.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:56.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:56.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:56.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:56.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:56.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:56.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:56.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:56.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:57.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:57.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:57.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:57.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:57.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:57.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:57.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:57.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:57.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:58.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=11357433 2021-01-15T18:41:58.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 11357433 '!=' null 2021-01-15T18:41:58.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:58.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:58.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:58.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:58.349 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:41:58.350 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:41:58.350 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:58.350 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:58.350 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:58.350 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:41:58.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:41:58.351 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:41:58.351 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:41:58.351 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:41:58.351 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:58.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:41:58.352 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:41:58.352 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:41:58.352 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:41:58.352 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:41:58.353 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:41:58.353 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:41:58.353 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:41:58.353 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:41:58.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:41:58.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:41:58.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:41:58.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:41:58.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:41:58.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:41:58.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:41:58.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:41:58.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:41:58.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:41:58.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:41:58.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:41:58.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:41:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:41:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:41:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:41:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:41:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:41:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:41:59.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:41:59.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:41:59.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:41:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220405 2021-01-15T18:41:59.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220405 '!=' null 2021-01-15T18:41:59.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:41:59.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:41:59.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:41:59.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:00.020 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:00.020 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:00.021 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:00.021 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:00.021 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:00.022 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:00.022 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:00.023 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:00.023 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:00.023 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:00.024 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:00.024 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:00.025 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:00.025 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:00.025 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:00.026 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:00.026 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:00.027 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:00.027 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:00.027 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:00.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:00.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:00.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:00.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:00.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:00.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:00.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:00.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:00.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:00.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:01.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:01.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:01.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:01.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:01.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:01.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=19220405 2021-01-15T18:42:01.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 19220405 '!=' null 2021-01-15T18:42:01.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:01.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:01.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:01.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:01.897 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:01.897 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:01.898 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:01.898 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:01.898 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:01.898 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:01.898 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:01.899 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:01.899 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:01.899 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:01.899 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:01.899 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:01.900 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:01.900 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:01.900 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:01.901 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:01.901 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:01.901 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:01.901 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:01.901 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:01.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:02.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:02.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:02.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:02.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:02.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:02.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:02.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:02.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:02.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:02.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:02.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:02.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:02.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:02.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:02.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:02.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:02.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:02.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:02.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:02.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:02.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104822 2021-01-15T18:42:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104822 '!=' null 2021-01-15T18:42:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:03.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:03.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:03.554 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:03.555 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:03.555 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:03.555 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:03.555 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:03.555 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:03.556 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:03.556 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:03.556 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:03.556 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:03.556 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:03.557 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:03.557 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:03.557 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:03.557 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:03.557 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:03.558 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:03.558 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:03.558 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:03.558 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:03.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:03.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:03.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:03.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:03.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:04.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:04.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:04.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:04.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:04.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:04.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:04.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:04.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:04.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:04.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:04.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:04.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:04.837 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:05.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104996 2021-01-15T18:42:05.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104996 '!=' null 2021-01-15T18:42:05.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:05.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:05.513 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:05.513 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:05.514 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:05.514 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:05.514 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:05.515 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:05.515 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:05.515 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:05.515 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:05.516 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:05.516 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:05.516 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:05.516 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:05.516 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:05.517 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:05.517 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:05.517 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:05.517 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:05.518 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:05.518 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:05.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:05.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:05.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:05.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:05.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:05.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:05.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:05.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:05.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:06.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:06.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:06.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:06.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:06.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:06.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:06.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:06.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:06.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:06.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:06.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:06.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:06.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:07.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=20094219 2021-01-15T18:42:07.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 20094219 '!=' null 2021-01-15T18:42:07.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:07.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:07.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:07.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:07.363 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:07.363 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:07.363 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:07.364 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:07.364 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:07.364 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:07.364 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:07.365 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:07.365 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:07.365 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:07.365 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:07.365 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:07.366 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:07.366 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:07.366 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:07.366 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:07.366 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:07.367 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:07.367 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:07.367 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:07.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:07.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:07.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:07.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:07.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:07.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:07.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:07.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:07.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:07.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:07.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:07.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:07.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:08.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:08.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:08.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:08.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:08.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:08.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:08.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:08.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:08.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:09.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=14852243 2021-01-15T18:42:09.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 14852243 '!=' null 2021-01-15T18:42:09.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:09.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:09.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:09.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:09.142 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:09.142 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:09.143 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:09.143 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:09.143 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:09.143 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:09.144 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:09.144 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:09.144 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:09.145 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:09.145 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:09.145 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:09.146 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:09.146 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:09.146 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:09.147 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:09.147 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:09.147 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:09.147 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:09.148 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:09.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:09.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:09.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:09.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:09.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:09.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:09.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:09.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:09.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:09.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:09.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:09.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:09.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:10.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:10.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:10.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:10.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:10.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=23589048 2021-01-15T18:42:10.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 23589048 '!=' null 2021-01-15T18:42:10.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:10.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:10.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:10.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:11.074 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:11.075 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:11.075 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:11.075 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:11.075 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:11.076 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [ 2021-01-15T18:42:11.076 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:11.076 INFO:tasks.workunit.client.0.smithi145.stdout: "item": "3.0", 2021-01-15T18:42:11.076 INFO:tasks.workunit.client.0.smithi145.stdout: "prio": 190, 2021-01-15T18:42:11.077 INFO:tasks.workunit.client.0.smithi145.stdout: "can_preempt": true 2021-01-15T18:42:11.077 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:11.077 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:11.077 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:11.078 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:11.078 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:11.078 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:11.078 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:11.079 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:11.079 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:11.079 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:11.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:11.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:11.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:11.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:11.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:11.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:11.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:11.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:11.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:11.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:11.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:11.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:11.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:12.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:12.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:12.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:12.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:12.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=15725947 2021-01-15T18:42:12.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 15725947 '!=' null 2021-01-15T18:42:12.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:12.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:12.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:12.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:12.849 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:12.850 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:12.850 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:12.850 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:12.850 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:12.851 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:12.851 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:12.851 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:12.851 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:12.851 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:12.852 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:12.852 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:12.852 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:12.852 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:12.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:12.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:12.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:12.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:12.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:12.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:12.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:12.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:13.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:13.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:13.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:13.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:13.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:13.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:13.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:13.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:13.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:13.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:13.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:13.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:13.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:14.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=13104967 2021-01-15T18:42:14.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 13104967 '!=' null 2021-01-15T18:42:14.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:14.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:14.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:14.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:14.465 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:14.466 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:14.466 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:14.466 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:14.466 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:14.467 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:14.467 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:14.467 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:14.467 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:14.467 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:14.468 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:14.468 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:14.468 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:14.468 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:14.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:14.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:14.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:14.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:14.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:14.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:14.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:14.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:14.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:14.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T18:42:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:14.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:15.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 3 2021-01-15T18:42:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 2 '!=' 2 2021-01-15T18:42:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:42:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:42:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:42:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:42:15.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:42:15.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:42:15.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:42:15.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=18347048 2021-01-15T18:42:15.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 18347048 '!=' null 2021-01-15T18:42:15.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:42:15.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:15.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: CEPH_ARGS= 2021-01-15T18:42:15.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_recovery_reservations 2021-01-15T18:42:16.081 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:16.081 INFO:tasks.workunit.client.0.smithi145.stdout: "local_reservations": { 2021-01-15T18:42:16.081 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:16.082 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:16.082 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:16.082 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:16.082 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:16.082 INFO:tasks.workunit.client.0.smithi145.stdout: "remote_reservations": { 2021-01-15T18:42:16.083 INFO:tasks.workunit.client.0.smithi145.stdout: "max_allowed": 1, 2021-01-15T18:42:16.083 INFO:tasks.workunit.client.0.smithi145.stdout: "min_priority": 0, 2021-01-15T18:42:16.083 INFO:tasks.workunit.client.0.smithi145.stdout: "queues": [], 2021-01-15T18:42:16.083 INFO:tasks.workunit.client.0.smithi145.stdout: "in_progress": [] 2021-01-15T18:42:16.084 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:16.084 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:16.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:42:16.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:42:16.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:16.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:16.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:16.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:16.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:16.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:16.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:16.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=3 2021-01-15T18:42:16.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:16.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:16.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 3 = 3 2021-01-15T18:42:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:42:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:42:17.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:295: TEST_recovery_priority: ceph pg dump pgs 2021-01-15T18:42:17.453 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:42:17.453 INFO:tasks.workunit.client.0.smithi145.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:38:19.636348+0000 96'600 122:1086 [1,4] 1 [1,4] 1 0'0 2021-01-15T18:29:13.919535+0000 0'0 2021-01-15T18:29:13.919535+0000 0 2021-01-15T18:42:17.453 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:40:16.813047+0000 96'600 122:1102 [1,2] 1 [1,2] 1 0'0 2021-01-15T18:28:58.110760+0000 0'0 2021-01-15T18:28:58.110760+0000 0 2021-01-15T18:42:17.453 INFO:tasks.workunit.client.0.smithi145.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 600 active+clean 2021-01-15T18:42:12.537932+0000 96'600 123:1106 [1,2] 1 [1,2] 1 0'0 2021-01-15T18:29:03.219749+0000 0'0 2021-01-15T18:29:03.219749+0000 0 2021-01-15T18:42:17.454 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:42:17.454 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:42:17.454 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:42:17.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: get_asok_path osd.1 2021-01-15T18:42:17.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T18:42:17.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T18:42:17.465 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:42:17.465 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:17.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:42:17.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.763429/ceph-osd.1.asok 2021-01-15T18:42:17.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: CEPH_ARGS= 2021-01-15T18:42:17.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:297: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.763429/ceph-osd.1.asok dump_pgstate_history 2021-01-15T18:42:17.565 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T18:42:17.566 INFO:tasks.workunit.client.0.smithi145.stdout: "pgs": [ 2021-01-15T18:42:17.566 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.566 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "7.0", 2021-01-15T18:42:17.566 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.567 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T18:42:17.567 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.567 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "62", 2021-01-15T18:42:17.567 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.567 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.568 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.568 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653308+0000", 2021-01-15T18:42:17.568 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:14.653317+0000" 2021-01-15T18:42:17.568 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.568 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.569 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.569 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653323+0000", 2021-01-15T18:42:17.569 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:14.653886+0000" 2021-01-15T18:42:17.569 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.569 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.570 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.570 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653887+0000", 2021-01-15T18:42:17.570 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:14.653942+0000" 2021-01-15T18:42:17.570 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.570 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.571 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.571 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653943+0000", 2021-01-15T18:42:17.571 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:14.653953+0000" 2021-01-15T18:42:17.571 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.572 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.572 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.572 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653953+0000", 2021-01-15T18:42:17.572 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.014152+0000" 2021-01-15T18:42:17.573 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.573 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.574 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.574 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653320+0000", 2021-01-15T18:42:17.574 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.014159+0000" 2021-01-15T18:42:17.574 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.574 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.575 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.575 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.014323+0000", 2021-01-15T18:42:17.575 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958635+0000" 2021-01-15T18:42:17.575 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.576 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.576 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.576 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.014160+0000", 2021-01-15T18:42:17.576 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958661+0000" 2021-01-15T18:42:17.576 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.577 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.577 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.577 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653317+0000", 2021-01-15T18:42:17.577 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958691+0000" 2021-01-15T18:42:17.578 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.578 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.578 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.578 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:14.653305+0000", 2021-01-15T18:42:17.578 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958700+0000" 2021-01-15T18:42:17.579 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.579 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.579 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.579 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.580 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "62", 2021-01-15T18:42:17.580 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.580 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.580 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.581 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958701+0000", 2021-01-15T18:42:17.581 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958963+0000" 2021-01-15T18:42:17.581 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.581 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.582 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.582 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.582 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "94", 2021-01-15T18:42:17.582 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.583 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.583 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.583 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958966+0000", 2021-01-15T18:42:17.584 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:15.958982+0000" 2021-01-15T18:42:17.584 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.584 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.584 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.585 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958989+0000", 2021-01-15T18:42:17.585 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.184967+0000" 2021-01-15T18:42:17.585 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.585 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.585 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.586 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.184968+0000", 2021-01-15T18:42:17.586 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.185038+0000" 2021-01-15T18:42:17.586 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.586 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.587 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.587 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.185038+0000", 2021-01-15T18:42:17.587 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.185049+0000" 2021-01-15T18:42:17.587 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.587 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.588 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.588 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.185050+0000", 2021-01-15T18:42:17.588 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.235290+0000" 2021-01-15T18:42:17.588 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.588 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.589 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.589 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958985+0000", 2021-01-15T18:42:17.589 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.235297+0000" 2021-01-15T18:42:17.589 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.589 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.590 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.590 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.235444+0000", 2021-01-15T18:42:17.590 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.897454+0000" 2021-01-15T18:42:17.590 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.590 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.591 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.591 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.897455+0000", 2021-01-15T18:42:17.591 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:16.897534+0000" 2021-01-15T18:42:17.591 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.591 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.592 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.592 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.897534+0000", 2021-01-15T18:42:17.592 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602160+0000" 2021-01-15T18:42:17.592 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.592 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.593 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.593 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:16.235298+0000", 2021-01-15T18:42:17.593 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602178+0000" 2021-01-15T18:42:17.593 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.593 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.594 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.594 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958983+0000", 2021-01-15T18:42:17.595 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602194+0000" 2021-01-15T18:42:17.595 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.595 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.595 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.595 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:15.958963+0000", 2021-01-15T18:42:17.596 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602197+0000" 2021-01-15T18:42:17.596 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.596 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.596 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.596 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.597 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "94", 2021-01-15T18:42:17.597 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.597 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.597 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.597 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602197+0000", 2021-01-15T18:42:17.598 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602442+0000" 2021-01-15T18:42:17.598 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.598 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.598 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.599 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2021-01-15T18:42:17.599 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.599 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.599 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.600 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602446+0000", 2021-01-15T18:42:17.600 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602458+0000" 2021-01-15T18:42:17.600 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.600 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.600 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.601 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602467+0000", 2021-01-15T18:42:17.601 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602592+0000" 2021-01-15T18:42:17.601 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.601 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.601 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.602 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602593+0000", 2021-01-15T18:42:17.602 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602643+0000" 2021-01-15T18:42:17.602 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.602 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.602 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.603 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602643+0000", 2021-01-15T18:42:17.603 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.602653+0000" 2021-01-15T18:42:17.603 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.603 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.603 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.604 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602654+0000", 2021-01-15T18:42:17.604 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:44.113099+0000" 2021-01-15T18:42:17.605 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.605 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.605 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.606 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602462+0000", 2021-01-15T18:42:17.606 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:44.113112+0000" 2021-01-15T18:42:17.606 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.606 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.607 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.607 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:44.113220+0000", 2021-01-15T18:42:17.607 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:44.180792+0000" 2021-01-15T18:42:17.607 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.607 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.608 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.608 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:44.180792+0000", 2021-01-15T18:42:17.608 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:44.180869+0000" 2021-01-15T18:42:17.608 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.608 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.609 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.609 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:44.180869+0000", 2021-01-15T18:42:17.609 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619296+0000" 2021-01-15T18:42:17.609 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.609 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.610 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.610 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:44.113112+0000", 2021-01-15T18:42:17.610 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619316+0000" 2021-01-15T18:42:17.611 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.611 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.611 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.611 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602459+0000", 2021-01-15T18:42:17.612 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619464+0000" 2021-01-15T18:42:17.612 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.612 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.612 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.613 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.602442+0000", 2021-01-15T18:42:17.613 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619468+0000" 2021-01-15T18:42:17.613 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.613 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.613 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.614 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.614 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "101", 2021-01-15T18:42:17.614 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.614 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.614 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.615 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619469+0000", 2021-01-15T18:42:17.615 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619847+0000" 2021-01-15T18:42:17.615 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.616 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.616 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.616 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.616 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "103", 2021-01-15T18:42:17.616 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.617 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.617 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.617 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619852+0000", 2021-01-15T18:42:17.617 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.619865+0000" 2021-01-15T18:42:17.617 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.618 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.618 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.618 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619873+0000", 2021-01-15T18:42:17.618 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.753476+0000" 2021-01-15T18:42:17.619 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.619 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.619 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.619 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.753477+0000", 2021-01-15T18:42:17.619 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.753535+0000" 2021-01-15T18:42:17.620 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.620 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.620 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.620 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619870+0000", 2021-01-15T18:42:17.620 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:36.753545+0000" 2021-01-15T18:42:17.621 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.621 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.621 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T18:42:17.621 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.753545+0000", 2021-01-15T18:42:17.622 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.172102+0000" 2021-01-15T18:42:17.622 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.622 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.622 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.623 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619865+0000", 2021-01-15T18:42:17.623 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.172142+0000" 2021-01-15T18:42:17.623 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.623 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.623 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.624 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:36.619847+0000", 2021-01-15T18:42:17.624 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.172150+0000" 2021-01-15T18:42:17.624 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.624 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.625 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.625 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.625 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "103", 2021-01-15T18:42:17.626 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.626 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.626 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.626 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172151+0000", 2021-01-15T18:42:17.626 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.172654+0000" 2021-01-15T18:42:17.627 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.627 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.627 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.627 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.627 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "118", 2021-01-15T18:42:17.628 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.628 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.628 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.628 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172663+0000", 2021-01-15T18:42:17.628 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.172688+0000" 2021-01-15T18:42:17.629 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.629 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.629 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.629 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172705+0000", 2021-01-15T18:42:17.629 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.173271+0000" 2021-01-15T18:42:17.630 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.630 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.630 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.630 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.173272+0000", 2021-01-15T18:42:17.630 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.173394+0000" 2021-01-15T18:42:17.631 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.631 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.631 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.631 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.173394+0000", 2021-01-15T18:42:17.631 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.173422+0000" 2021-01-15T18:42:17.632 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.632 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.632 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.632 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.173422+0000", 2021-01-15T18:42:17.633 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.557775+0000" 2021-01-15T18:42:17.633 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.633 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.633 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.633 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172697+0000", 2021-01-15T18:42:17.634 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.557785+0000" 2021-01-15T18:42:17.634 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.634 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.634 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.634 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.559221+0000", 2021-01-15T18:42:17.635 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.759546+0000" 2021-01-15T18:42:17.635 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.635 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.635 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-01-15T18:42:17.636 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.759547+0000", 2021-01-15T18:42:17.636 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.759720+0000" 2021-01-15T18:42:17.636 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.636 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.637 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-01-15T18:42:17.637 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.759720+0000", 2021-01-15T18:42:17.637 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:37.760005+0000" 2021-01-15T18:42:17.637 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.638 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.638 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovering", 2021-01-15T18:42:17.638 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.760006+0000", 2021-01-15T18:42:17.638 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:08.841346+0000" 2021-01-15T18:42:17.638 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.639 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.639 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotRecovering", 2021-01-15T18:42:17.639 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:08.841346+0000", 2021-01-15T18:42:17.639 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:08.841495+0000" 2021-01-15T18:42:17.639 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.640 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.640 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-01-15T18:42:17.640 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:08.841495+0000", 2021-01-15T18:42:17.640 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:21.445874+0000" 2021-01-15T18:42:17.640 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:21.445875+0000", 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:21.446197+0000" 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.641 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.642 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovering", 2021-01-15T18:42:17.642 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:21.446197+0000", 2021-01-15T18:42:17.642 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:17.621346+0000" 2021-01-15T18:42:17.642 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.643 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.643 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.643 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:38:17.621346+0000", 2021-01-15T18:42:17.643 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:18.123036+0000" 2021-01-15T18:42:17.643 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.644 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.644 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.644 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.557785+0000", 2021-01-15T18:42:17.644 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:18.123091+0000" 2021-01-15T18:42:17.644 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.645 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.645 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.645 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172688+0000", 2021-01-15T18:42:17.645 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:18.123140+0000" 2021-01-15T18:42:17.645 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.646 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.646 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.646 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:37.172655+0000", 2021-01-15T18:42:17.647 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:18.123153+0000" 2021-01-15T18:42:17.647 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.647 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.647 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.647 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.648 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "118", 2021-01-15T18:42:17.648 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.648 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.648 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.648 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:38:18.123154+0000", 2021-01-15T18:42:17.649 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:18.123861+0000" 2021-01-15T18:42:17.649 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.649 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.649 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.650 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.650 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.650 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.650 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "3.0", 2021-01-15T18:42:17.650 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.651 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T18:42:17.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.651 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "40", 2021-01-15T18:42:17.651 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.651 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.652 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.652 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.738608+0000", 2021-01-15T18:42:17.652 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:03.738626+0000" 2021-01-15T18:42:17.652 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.652 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.653 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.653 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.738636+0000", 2021-01-15T18:42:17.653 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:03.914445+0000" 2021-01-15T18:42:17.653 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.653 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.654 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.654 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.914445+0000", 2021-01-15T18:42:17.654 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:03.914510+0000" 2021-01-15T18:42:17.654 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.654 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.655 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.655 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.914510+0000", 2021-01-15T18:42:17.655 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:03.914522+0000" 2021-01-15T18:42:17.655 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.655 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.656 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.656 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.914522+0000", 2021-01-15T18:42:17.656 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:04.208661+0000" 2021-01-15T18:42:17.657 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.657 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.657 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.657 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.738631+0000", 2021-01-15T18:42:17.657 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:04.208668+0000" 2021-01-15T18:42:17.658 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.658 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.658 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.658 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:04.208864+0000", 2021-01-15T18:42:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:04.743030+0000" 2021-01-15T18:42:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.659 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:04.743031+0000", 2021-01-15T18:42:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:04.743121+0000" 2021-01-15T18:42:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.660 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:04.743121+0000", 2021-01-15T18:42:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119652+0000" 2021-01-15T18:42:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.661 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:04.208668+0000", 2021-01-15T18:42:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119668+0000" 2021-01-15T18:42:17.662 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.738626+0000", 2021-01-15T18:42:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119685+0000" 2021-01-15T18:42:17.663 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.664 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.664 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.664 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:03.738603+0000", 2021-01-15T18:42:17.664 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119689+0000" 2021-01-15T18:42:17.664 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.665 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.665 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.665 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.665 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "40", 2021-01-15T18:42:17.665 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.666 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.666 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.666 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119689+0000", 2021-01-15T18:42:17.666 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119943+0000" 2021-01-15T18:42:17.666 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.667 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.667 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.667 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.667 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "91", 2021-01-15T18:42:17.667 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.668 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.668 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.668 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119948+0000", 2021-01-15T18:42:17.668 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.119959+0000" 2021-01-15T18:42:17.668 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.669 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.669 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.669 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119965+0000", 2021-01-15T18:42:17.669 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.236171+0000" 2021-01-15T18:42:17.669 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.670 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.670 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.670 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.236171+0000", 2021-01-15T18:42:17.670 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.236234+0000" 2021-01-15T18:42:17.670 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.671 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.671 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.671 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.236236+0000", 2021-01-15T18:42:17.671 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.236249+0000" 2021-01-15T18:42:17.672 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.672 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.672 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.672 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.236250+0000", 2021-01-15T18:42:17.672 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.504852+0000" 2021-01-15T18:42:17.673 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.673 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.673 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.673 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119962+0000", 2021-01-15T18:42:17.673 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.504861+0000" 2021-01-15T18:42:17.674 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.674 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.674 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.674 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.504976+0000", 2021-01-15T18:42:17.674 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.765625+0000" 2021-01-15T18:42:17.675 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.675 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.675 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.675 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.765626+0000", 2021-01-15T18:42:17.675 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:05.765674+0000" 2021-01-15T18:42:17.676 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.676 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.676 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.676 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.765674+0000", 2021-01-15T18:42:17.676 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832200+0000" 2021-01-15T18:42:17.677 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.677 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.677 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.677 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.504861+0000", 2021-01-15T18:42:17.678 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832226+0000" 2021-01-15T18:42:17.678 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.678 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.678 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.678 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119959+0000", 2021-01-15T18:42:17.679 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832250+0000" 2021-01-15T18:42:17.679 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.679 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.679 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.679 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:05.119943+0000", 2021-01-15T18:42:17.680 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832257+0000" 2021-01-15T18:42:17.680 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.680 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.680 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.680 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.681 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "92", 2021-01-15T18:42:17.681 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.681 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.681 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.681 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832257+0000", 2021-01-15T18:42:17.682 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832572+0000" 2021-01-15T18:42:17.682 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.682 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.682 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.683 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "109", 2021-01-15T18:42:17.683 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.683 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.683 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.684 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832575+0000", 2021-01-15T18:42:17.684 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.832585+0000" 2021-01-15T18:42:17.684 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.684 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.684 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.685 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832591+0000", 2021-01-15T18:42:17.685 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.941150+0000" 2021-01-15T18:42:17.685 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.685 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.685 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.686 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.941150+0000", 2021-01-15T18:42:17.686 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.941242+0000" 2021-01-15T18:42:17.686 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.686 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.687 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.687 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.941243+0000", 2021-01-15T18:42:17.687 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:42.941264+0000" 2021-01-15T18:42:17.687 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.687 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.688 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.688 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.941264+0000", 2021-01-15T18:42:17.688 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.242748+0000" 2021-01-15T18:42:17.688 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.688 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.689 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.689 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832588+0000", 2021-01-15T18:42:17.689 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.242755+0000" 2021-01-15T18:42:17.689 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.689 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.690 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.690 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.242829+0000", 2021-01-15T18:42:17.690 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.351190+0000" 2021-01-15T18:42:17.690 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.690 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.691 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.691 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.351193+0000", 2021-01-15T18:42:17.691 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:43.351235+0000" 2021-01-15T18:42:17.691 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.691 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.692 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.692 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.351235+0000", 2021-01-15T18:42:17.692 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648040+0000" 2021-01-15T18:42:17.692 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.692 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.693 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.693 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:43.242756+0000", 2021-01-15T18:42:17.693 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648067+0000" 2021-01-15T18:42:17.693 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.694 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.694 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.694 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832585+0000", 2021-01-15T18:42:17.694 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648212+0000" 2021-01-15T18:42:17.694 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.695 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.695 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.695 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:42.832572+0000", 2021-01-15T18:42:17.695 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648236+0000" 2021-01-15T18:42:17.695 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.696 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.696 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.696 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.696 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "110", 2021-01-15T18:42:17.696 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.697 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.697 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.697 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648237+0000", 2021-01-15T18:42:17.697 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648630+0000" 2021-01-15T18:42:17.697 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.698 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.698 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.698 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.698 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "112", 2021-01-15T18:42:17.698 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.699 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.699 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.699 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648634+0000", 2021-01-15T18:42:17.699 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.648647+0000" 2021-01-15T18:42:17.699 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.700 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.700 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.700 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648656+0000", 2021-01-15T18:42:17.700 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.765116+0000" 2021-01-15T18:42:17.700 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.701 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.701 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.701 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.765117+0000", 2021-01-15T18:42:17.701 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.765198+0000" 2021-01-15T18:42:17.701 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.702 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.702 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.702 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648652+0000", 2021-01-15T18:42:17.702 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:54.765214+0000" 2021-01-15T18:42:17.702 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.703 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.703 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T18:42:17.703 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.765215+0000", 2021-01-15T18:42:17.703 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.125733+0000" 2021-01-15T18:42:17.703 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.704 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.704 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.704 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648647+0000", 2021-01-15T18:42:17.704 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.125770+0000" 2021-01-15T18:42:17.704 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.705 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.705 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.705 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:54.648630+0000", 2021-01-15T18:42:17.705 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.125778+0000" 2021-01-15T18:42:17.706 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.706 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.706 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.706 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.706 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "112", 2021-01-15T18:42:17.707 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.707 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.707 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.707 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.125779+0000", 2021-01-15T18:42:17.707 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.126269+0000" 2021-01-15T18:42:17.708 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.708 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.708 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.708 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.708 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "122", 2021-01-15T18:42:17.709 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.709 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.709 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.709 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.126278+0000", 2021-01-15T18:42:17.709 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.126304+0000" 2021-01-15T18:42:17.710 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.710 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.710 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.710 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.126321+0000", 2021-01-15T18:42:17.710 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.127025+0000" 2021-01-15T18:42:17.711 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.711 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.711 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.711 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.127025+0000", 2021-01-15T18:42:17.711 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.127095+0000" 2021-01-15T18:42:17.712 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.712 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.712 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.712 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.127096+0000", 2021-01-15T18:42:17.712 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.127112+0000" 2021-01-15T18:42:17.713 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.713 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.713 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.713 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.127112+0000", 2021-01-15T18:42:17.714 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.535619+0000" 2021-01-15T18:42:17.714 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.714 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.714 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.714 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.126313+0000", 2021-01-15T18:42:17.715 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.535631+0000" 2021-01-15T18:42:17.715 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.715 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.715 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.715 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.536917+0000", 2021-01-15T18:42:17.716 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:55.806811+0000" 2021-01-15T18:42:17.716 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.716 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.716 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-01-15T18:42:17.716 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.806811+0000", 2021-01-15T18:42:17.717 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:08.841460+0000" 2021-01-15T18:42:17.717 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.717 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.717 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-01-15T18:42:17.717 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:08.841461+0000", 2021-01-15T18:42:17.718 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:08.841747+0000" 2021-01-15T18:42:17.718 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.718 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.718 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovering", 2021-01-15T18:42:17.718 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:08.841748+0000", 2021-01-15T18:42:17.719 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:21.445696+0000" 2021-01-15T18:42:17.719 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.719 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.719 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/NotRecovering", 2021-01-15T18:42:17.719 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:21.445697+0000", 2021-01-15T18:42:17.720 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:36:21.445915+0000" 2021-01-15T18:42:17.720 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.720 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.720 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-01-15T18:42:17.720 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:36:21.445915+0000", 2021-01-15T18:42:17.721 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:14.770781+0000" 2021-01-15T18:42:17.721 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.721 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.721 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-01-15T18:42:17.721 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:40:14.770781+0000", 2021-01-15T18:42:17.722 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:14.771041+0000" 2021-01-15T18:42:17.722 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.722 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.722 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovering", 2021-01-15T18:42:17.722 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:40:14.771041+0000", 2021-01-15T18:42:17.723 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.659363+0000" 2021-01-15T18:42:17.723 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.723 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.723 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.723 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:42:11.659364+0000", 2021-01-15T18:42:17.724 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.960079+0000" 2021-01-15T18:42:17.724 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.724 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.724 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.724 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.535632+0000", 2021-01-15T18:42:17.725 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.960100+0000" 2021-01-15T18:42:17.725 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.725 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.725 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.725 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.126305+0000", 2021-01-15T18:42:17.726 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.960125+0000" 2021-01-15T18:42:17.726 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.726 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.726 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:55.126269+0000", 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.960132+0000" 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.727 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.728 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "122", 2021-01-15T18:42:17.728 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.728 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.728 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.728 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:42:11.960134+0000", 2021-01-15T18:42:17.729 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:42:11.960520+0000" 2021-01-15T18:42:17.729 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.729 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.729 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.729 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.730 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.730 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.730 INFO:tasks.workunit.client.0.smithi145.stdout: "pg": "1.0", 2021-01-15T18:42:17.730 INFO:tasks.workunit.client.0.smithi145.stdout: "currently": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.730 INFO:tasks.workunit.client.0.smithi145.stdout: "history": [ 2021-01-15T18:42:17.731 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.731 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "26", 2021-01-15T18:42:17.731 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.731 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.731 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Initial", 2021-01-15T18:42:17.732 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462490+0000", 2021-01-15T18:42:17.732 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.462700+0000" 2021-01-15T18:42:17.732 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.732 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.732 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.733 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "26", 2021-01-15T18:42:17.733 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.733 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.734 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.734 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462701+0000", 2021-01-15T18:42:17.734 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.462747+0000" 2021-01-15T18:42:17.734 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.734 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.735 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.735 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "29", 2021-01-15T18:42:17.735 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.735 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.736 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.736 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462750+0000", 2021-01-15T18:42:17.736 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.462761+0000" 2021-01-15T18:42:17.736 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.736 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.737 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.737 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462768+0000", 2021-01-15T18:42:17.737 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.463310+0000" 2021-01-15T18:42:17.737 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.737 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.738 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.738 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.463311+0000", 2021-01-15T18:42:17.738 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.463378+0000" 2021-01-15T18:42:17.738 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.738 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.739 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.739 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.463379+0000", 2021-01-15T18:42:17.739 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.463391+0000" 2021-01-15T18:42:17.739 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.739 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.740 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.740 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.463392+0000", 2021-01-15T18:42:17.740 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.780975+0000" 2021-01-15T18:42:17.740 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.740 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.741 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.741 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462765+0000", 2021-01-15T18:42:17.741 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:58.780988+0000" 2021-01-15T18:42:17.741 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.742 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.742 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.742 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.781189+0000", 2021-01-15T18:42:17.742 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:59.508273+0000" 2021-01-15T18:42:17.742 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.743 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.743 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.743 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:59.508273+0000", 2021-01-15T18:42:17.743 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:28:59.508322+0000" 2021-01-15T18:42:17.743 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.744 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.744 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.744 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:59.508323+0000", 2021-01-15T18:42:17.744 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.098895+0000" 2021-01-15T18:42:17.744 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.745 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.745 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.745 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.780988+0000", 2021-01-15T18:42:17.745 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.098914+0000" 2021-01-15T18:42:17.745 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.746 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.746 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.746 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462761+0000", 2021-01-15T18:42:17.746 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.098932+0000" 2021-01-15T18:42:17.746 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.747 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.747 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.747 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:28:58.462747+0000", 2021-01-15T18:42:17.747 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.098935+0000" 2021-01-15T18:42:17.747 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.748 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.748 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.748 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.748 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "29", 2021-01-15T18:42:17.748 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.749 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.749 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.749 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.098936+0000", 2021-01-15T18:42:17.749 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.099214+0000" 2021-01-15T18:42:17.749 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.750 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.750 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.750 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.750 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "105", 2021-01-15T18:42:17.750 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.099218+0000", 2021-01-15T18:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.099230+0000" 2021-01-15T18:42:17.751 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.752 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.752 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.752 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.099254+0000", 2021-01-15T18:42:17.752 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.228672+0000" 2021-01-15T18:42:17.752 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.753 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.753 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.753 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.228672+0000", 2021-01-15T18:42:17.753 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.228736+0000" 2021-01-15T18:42:17.753 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.754 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.754 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.754 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.228737+0000", 2021-01-15T18:42:17.754 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.228748+0000" 2021-01-15T18:42:17.754 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.755 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.755 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.755 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.228748+0000", 2021-01-15T18:42:17.755 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.639251+0000" 2021-01-15T18:42:17.755 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.756 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.756 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.756 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.099250+0000", 2021-01-15T18:42:17.756 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.639261+0000" 2021-01-15T18:42:17.756 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.757 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.757 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.757 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.639383+0000", 2021-01-15T18:42:17.757 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.966850+0000" 2021-01-15T18:42:17.757 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.758 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.758 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.758 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.966851+0000", 2021-01-15T18:42:17.758 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:29:00.966897+0000" 2021-01-15T18:42:17.758 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.759 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.759 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Clean", 2021-01-15T18:42:17.759 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.966897+0000", 2021-01-15T18:42:17.759 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942198+0000" 2021-01-15T18:42:17.759 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.760 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.760 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.760 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.639262+0000", 2021-01-15T18:42:17.760 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942223+0000" 2021-01-15T18:42:17.761 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.761 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.761 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.761 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.099230+0000", 2021-01-15T18:42:17.761 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942370+0000" 2021-01-15T18:42:17.762 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.762 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.762 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.762 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:29:00.099214+0000", 2021-01-15T18:42:17.762 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942378+0000" 2021-01-15T18:42:17.763 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.763 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.763 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.763 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.763 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "105", 2021-01-15T18:42:17.764 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.764 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.764 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.764 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942378+0000", 2021-01-15T18:42:17.765 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942779+0000" 2021-01-15T18:42:17.765 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.765 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.765 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.765 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.766 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "107", 2021-01-15T18:42:17.766 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.766 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.766 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.766 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942785+0000", 2021-01-15T18:42:17.767 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:46.942798+0000" 2021-01-15T18:42:17.767 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.767 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.767 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.767 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942807+0000", 2021-01-15T18:42:17.768 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.051039+0000" 2021-01-15T18:42:17.768 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.768 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.768 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.768 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.051040+0000", 2021-01-15T18:42:17.769 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.051103+0000" 2021-01-15T18:42:17.769 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.769 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.769 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.769 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942803+0000", 2021-01-15T18:42:17.770 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.051111+0000" 2021-01-15T18:42:17.770 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.770 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.770 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/WaitActingChange", 2021-01-15T18:42:17.770 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.051111+0000", 2021-01-15T18:42:17.771 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.394589+0000" 2021-01-15T18:42:17.771 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.771 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.771 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.771 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942799+0000", 2021-01-15T18:42:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.394611+0000" 2021-01-15T18:42:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.772 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:46.942780+0000", 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.394616+0000" 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.773 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "107", 2021-01-15T18:42:17.774 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.774 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.774 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.774 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394616+0000", 2021-01-15T18:42:17.774 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.394894+0000" 2021-01-15T18:42:17.775 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.775 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.775 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.775 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.775 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "120", 2021-01-15T18:42:17.776 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.776 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.776 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Start", 2021-01-15T18:42:17.776 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394899+0000", 2021-01-15T18:42:17.776 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.394913+0000" 2021-01-15T18:42:17.777 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.777 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.777 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-01-15T18:42:17.777 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394921+0000", 2021-01-15T18:42:17.777 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.520960+0000" 2021-01-15T18:42:17.778 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.778 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.778 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetLog", 2021-01-15T18:42:17.778 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.520960+0000", 2021-01-15T18:42:17.778 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.521047+0000" 2021-01-15T18:42:17.779 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.779 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.779 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-01-15T18:42:17.779 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.521047+0000", 2021-01-15T18:42:17.779 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.521064+0000" 2021-01-15T18:42:17.780 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.780 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.780 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-01-15T18:42:17.780 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.521065+0000", 2021-01-15T18:42:17.780 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.751824+0000" 2021-01-15T18:42:17.781 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.781 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.781 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Peering", 2021-01-15T18:42:17.781 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394918+0000", 2021-01-15T18:42:17.781 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:47.751842+0000" 2021-01-15T18:42:17.782 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.782 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.782 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Activating", 2021-01-15T18:42:17.782 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.753484+0000", 2021-01-15T18:42:17.782 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:35:48.274951+0000" 2021-01-15T18:42:17.783 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.783 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.783 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2021-01-15T18:42:17.783 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:48.274952+0000", 2021-01-15T18:42:17.783 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:17.621611+0000" 2021-01-15T18:42:17.784 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.784 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.784 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2021-01-15T18:42:17.784 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:38:17.621611+0000", 2021-01-15T18:42:17.784 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:38:17.621939+0000" 2021-01-15T18:42:17.785 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.785 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.785 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovering", 2021-01-15T18:42:17.785 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:38:17.621943+0000", 2021-01-15T18:42:17.785 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:14.770588+0000" 2021-01-15T18:42:17.786 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.786 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.786 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active/Recovered", 2021-01-15T18:42:17.786 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:40:14.770588+0000", 2021-01-15T18:42:17.786 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:15.129955+0000" 2021-01-15T18:42:17.787 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.787 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.787 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary/Active", 2021-01-15T18:42:17.787 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.751843+0000", 2021-01-15T18:42:17.787 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:15.129986+0000" 2021-01-15T18:42:17.788 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.788 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.788 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started/Primary", 2021-01-15T18:42:17.788 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394913+0000", 2021-01-15T18:42:17.788 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:15.130010+0000" 2021-01-15T18:42:17.789 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.789 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.789 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Started", 2021-01-15T18:42:17.789 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:35:47.394894+0000", 2021-01-15T18:42:17.789 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:15.130016+0000" 2021-01-15T18:42:17.790 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.790 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.790 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T18:42:17.790 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.790 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": "120", 2021-01-15T18:42:17.791 INFO:tasks.workunit.client.0.smithi145.stdout: "states": [ 2021-01-15T18:42:17.791 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T18:42:17.791 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "Reset", 2021-01-15T18:42:17.791 INFO:tasks.workunit.client.0.smithi145.stdout: "enter": "2021-01-15T18:40:15.130016+0000", 2021-01-15T18:42:17.791 INFO:tasks.workunit.client.0.smithi145.stdout: "exit": "2021-01-15T18:40:15.130464+0000" 2021-01-15T18:42:17.792 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.792 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.792 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.792 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.792 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T18:42:17.793 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T18:42:17.793 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T18:42:17.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:299: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2021-01-15T18:42:17.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:303: TEST_recovery_priority: echo TEST PASSED 2021-01-15T18:42:17.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: delete_pool test1 2021-01-15T18:42:17.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T18:42:17.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T18:42:17.795 INFO:tasks.workunit.client.0.smithi145.stdout:TEST PASSED 2021-01-15T18:42:18.158 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T18:42:18.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:307: TEST_recovery_priority: delete_pool test3 2021-01-15T18:42:18.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test3 2021-01-15T18:42:18.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-01-15T18:42:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test3' does not exist 2021-01-15T18:42:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:308: TEST_recovery_priority: delete_pool test7 2021-01-15T18:42:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test7 2021-01-15T18:42:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-01-15T18:42:19.252 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test7' does not exist 2021-01-15T18:42:19.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2021-01-15T18:42:19.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:42:19.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:42:19.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:42:19.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:42:19.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:42:19.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:42:19.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: return 0 2021-01-15T18:42:19.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: teardown td/osd-recovery-prio 2021-01-15T18:42:19.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-01-15T18:42:19.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:42:19.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-01-15T18:42:19.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:42:19.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:42:19.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:42:19.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:42:19.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:42:19.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:42:19.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:42:19.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:42:19.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:42:19.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:42:19.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:42:19.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:42:19.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:42:19.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:42:19.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:42:19.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:42:19.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-01-15T18:42:19.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:42:19.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:19.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:42:19.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.763429 2021-01-15T18:42:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:42:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:42:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:42:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-recovery-prio 0 2021-01-15T18:42:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2021-01-15T18:42:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:42:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2021-01-15T18:42:19.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:42:19.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:42:19.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:42:19.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:42:19.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:42:19.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:42:19.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:42:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:42:19.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:42:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:42:19.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:42:19.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:42:19.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:42:19.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:42:19.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:42:19.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:42:19.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2021-01-15T18:42:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:42:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:19.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.763429 2021-01-15T18:42:19.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.763429 2021-01-15T18:42:19.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:42:19.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:42:19.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:42:19.655 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:42:19.656 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:42:19.689 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2021-01-15T18:42:19.689 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2021-01-15T18:42:19.750 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:42:19.761 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:42:19.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:42:19.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:42:19.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:42:19.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:42:19.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:42:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:42:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:42:19.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:42:19.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-recovery-space 2021-01-15T18:42:19.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2021-01-15T18:42:19.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2021-01-15T18:42:19.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2021-01-15T18:42:19.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2021-01-15T18:42:19.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2021-01-15T18:42:19.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2021-01-15T18:42:19.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none ' 2021-01-15T18:42:19.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2021-01-15T18:42:19.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2021-01-15T18:42:19.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: export objects=600 2021-01-15T18:42:19.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: objects=600 2021-01-15T18:42:19.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export poolprefix=test 2021-01-15T18:42:19.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: poolprefix=test 2021-01-15T18:42:19.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: set 2021-01-15T18:42:19.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:42:19.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:32: run: local funcs=TEST_recovery_test_simple 2021-01-15T18:42:19.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: for func in $funcs 2021-01-15T18:42:19.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: setup td/osd-recovery-space 2021-01-15T18:42:19.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2021-01-15T18:42:19.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2021-01-15T18:42:19.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-01-15T18:42:19.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:42:19.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-01-15T18:42:19.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:42:19.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:42:19.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:42:19.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:42:19.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:42:19.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:42:19.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:42:19.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:42:19.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:42:19.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:42:19.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:42:19.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:42:19.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:42:19.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:42:19.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:42:19.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:42:19.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:42:19.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-01-15T18:42:19.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:42:19.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:19.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:19.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.821914 2021-01-15T18:42:19.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:42:19.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:42:19.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2021-01-15T18:42:19.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:42:19.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:19.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:19.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.821914 2021-01-15T18:42:19.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:42:19.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:42:19.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:42:19.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:42:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2021-01-15T18:42:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: TEST_recovery_test_simple td/osd-recovery-space 2021-01-15T18:42:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:88: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2021-01-15T18:42:19.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local pools=1 2021-01-15T18:42:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local OSDS=2 2021-01-15T18:42:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:92: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2021-01-15T18:42:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2021-01-15T18:42:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:42:19.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:42:19.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:42:19.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2021-01-15T18:42:19.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2021-01-15T18:42:20.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:42:20.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:42:20.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:42:20.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:42:20.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:20.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:20.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:20.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:42:20.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:42:20.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:42:20.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:42:20.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:42:20.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:42:20.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:42:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:42:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:42:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:42:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:42:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:20.255 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.821914/ceph-mon.a.asok 2021-01-15T18:42:20.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:42:20.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.821914/ceph-mon.a.asok config get fsid 2021-01-15T18:42:20.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:42:20.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:42:20.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:42:20.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:42:20.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:42:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:42:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:42:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:42:20.355 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:42:20.356 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:20.356 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:20.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.821914/ceph-mon.a.asok 2021-01-15T18:42:20.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:42:20.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.821914/ceph-mon.a.asok config get mon_host 2021-01-15T18:42:20.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2021-01-15T18:42:20.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-space 2021-01-15T18:42:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:42:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:42:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:42:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-space/x 2021-01-15T18:42:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:42:20.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:42:20.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:42:20.954 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:42:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:42:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:42:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:42:20.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: export CEPH_ARGS 2021-01-15T18:42:20.975 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: expr 2 - 1 2021-01-15T18:42:20.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: seq 0 1 2021-01-15T18:42:20.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:42:20.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2021-01-15T18:42:20.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-space 2021-01-15T18:42:20.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:42:20.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:42:20.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:42:20.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-space/0 2021-01-15T18:42:20.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2021-01-15T18:42:20.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:42:20.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:42:20.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:42:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2021-01-15T18:42:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2021-01-15T18:42:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:42:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:42:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2021-01-15T18:42:20.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:42:20.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:42:20.984 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:42:20.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:42:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:20.985 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:20.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:42:20.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:42:20.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:42:20.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2021-01-15T18:42:20.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2021-01-15T18:42:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:42:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:42:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:42:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:42:20.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:42:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-space/0 2021-01-15T18:42:20.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:42:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=69b0aeb5-4b63-4a97-9ff8-b8b77828d06e 2021-01-15T18:42:20.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 69b0aeb5-4b63-4a97-9ff8-b8b77828d06e' 2021-01-15T18:42:20.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:42:20.989 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 69b0aeb5-4b63-4a97-9ff8-b8b77828d06e 2021-01-15T18:42:20.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAM4gFgGy0+OxAAKbzCjWUZM6QiVQkbX2mzSg== 2021-01-15T18:42:20.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAM4gFgGy0+OxAAKbzCjWUZM6QiVQkbX2mzSg=="}' 2021-01-15T18:42:20.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 69b0aeb5-4b63-4a97-9ff8-b8b77828d06e -i td/osd-recovery-space/0/new.json 2021-01-15T18:42:21.345 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:42:21.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-space/0/new.json 2021-01-15T18:42:21.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAM4gFgGy0+OxAAKbzCjWUZM6QiVQkbX2mzSg== --osd-uuid 69b0aeb5-4b63-4a97-9ff8-b8b77828d06e 2021-01-15T18:42:21.379 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.378+0000 7f2a6e571080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:21.380 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.380+0000 7f2a6e571080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:21.381 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.380+0000 7f2a6e571080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.381+0000 7f2a6e571080 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-01-15T18:42:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.381+0000 7f2a6e571080 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-01-15T18:42:21.382 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.381+0000 7f2a6e571080 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2021-01-15T18:42:21.426 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:21.425+0000 7f2a6e571080 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2021-01-15T18:42:26.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2021-01-15T18:42:26.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:42:26.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:42:26.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:42:26.054 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:42:26.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:42:26.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:42:26.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:42:26.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:42:26.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:42:26.572 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:42:26.593 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:26.592+0000 7f800df48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:26.594 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:26.594+0000 7f800df48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:26.595 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:26.594+0000 7f800df48080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:26.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:42:26.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:42:26.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:42:26.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:42:26.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:42:26.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:42:26.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:42:26.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:42:26.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:42:26.969 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:42:27.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:27.111+0000 7f800df48080 -1 Falling back to public interface 2021-01-15T18:42:27.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:42:27.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:27.902+0000 7f800df48080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:42:28.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:42:28.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:42:28.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:42:28.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:42:28.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:42:28.356 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:42:28.752 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/822956,v1:127.0.0.1:6803/822956] [v2:127.0.0.1:6804/822956,v1:127.0.0.1:6805/822956] exists,up 69b0aeb5-4b63-4a97-9ff8-b8b77828d06e 2021-01-15T18:42:28.752 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 62f7381a-ede9-4734-aadc-8e4c39b27e45 2021-01-15T18:42:28.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:42:28.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:42:28.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:42:28.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:96: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:42:28.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2021-01-15T18:42:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-space 2021-01-15T18:42:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:42:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:42:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:42:28.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-space/1 2021-01-15T18:42:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 ' 2021-01-15T18:42:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:42:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:42:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:42:28.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2021-01-15T18:42:28.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2021-01-15T18:42:28.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:42:28.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:42:28.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2021-01-15T18:42:28.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:42:28.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:42:28.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:42:28.757 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:42:28.757 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:42:28.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:42:28.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:28.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' 2021-01-15T18:42:28.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:42:28.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:42:28.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:42:28.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2021-01-15T18:42:28.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2021-01-15T18:42:28.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:42:28.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:42:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:42:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:42:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:42:28.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-space/1 2021-01-15T18:42:28.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:42:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=62f7381a-ede9-4734-aadc-8e4c39b27e45 2021-01-15T18:42:28.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 62f7381a-ede9-4734-aadc-8e4c39b27e45' 2021-01-15T18:42:28.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:42:28.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAU4gFgTD+fLRAAJ7bthAYqh/EojH96bHKHNA== 2021-01-15T18:42:28.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAU4gFgTD+fLRAAJ7bthAYqh/EojH96bHKHNA=="}' 2021-01-15T18:42:28.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 62f7381a-ede9-4734-aadc-8e4c39b27e45 -i td/osd-recovery-space/1/new.json 2021-01-15T18:42:29.300 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:42:29.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-space/1/new.json 2021-01-15T18:42:29.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAU4gFgTD+fLRAAJ7bthAYqh/EojH96bHKHNA== --osd-uuid 62f7381a-ede9-4734-aadc-8e4c39b27e45 2021-01-15T18:42:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.335+0000 7fc8e709e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:29.338 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.337+0000 7fc8e709e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:29.339 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.338+0000 7fc8e709e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.339+0000 7fc8e709e080 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-01-15T18:42:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.339+0000 7fc8e709e080 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-01-15T18:42:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.339+0000 7fc8e709e080 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2021-01-15T18:42:29.392 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:29.391+0000 7fc8e709e080 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2021-01-15T18:42:34.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2021-01-15T18:42:34.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:42:34.195 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:42:34.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:42:34.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:42:34.724 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:42:34.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:42:34.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:42:34.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=cfee71ee-f841-4191-a39b-9576992ea7f1 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.821914/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:42:34.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:42:34.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:42:34.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:34.745+0000 7fe5c53f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:34.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:34.747+0000 7fe5c53f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:34.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:34.748+0000 7fe5c53f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:42:35.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:42:35.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:42:35.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:42:35.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:42:35.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:42:35.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:42:35.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:42:35.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:42:35.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:42:35.121 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:42:35.265 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:35.264+0000 7fe5c53f2080 -1 Falling back to public interface 2021-01-15T18:42:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:42:36.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:42:36.082+0000 7fe5c53f2080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:42:36.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:42:36.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:42:36.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:42:36.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:42:36.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:42:36.512 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:42:36.936 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/824228,v1:127.0.0.1:6811/824228] [v2:127.0.0.1:6812/824228,v1:127.0.0.1:6813/824228] exists,up 62f7381a-ede9-4734-aadc-8e4c39b27e45 2021-01-15T18:42:36.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:42:36.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:42:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:42:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2021-01-15T18:42:37.725 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-nearfull-ratio 0.4 2021-01-15T18:42:37.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2021-01-15T18:42:38.460 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-backfillfull-ratio 0.45 2021-01-15T18:42:38.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2021-01-15T18:42:39.263 INFO:tasks.workunit.client.0.smithi145.stderr:osd set-full-ratio 0.5 2021-01-15T18:42:39.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: seq 1 1 2021-01-15T18:42:39.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:105: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2021-01-15T18:42:39.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:107: TEST_recovery_test_simple: create_pool test1 1 1 2021-01-15T18:42:39.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test1 1 1 2021-01-15T18:42:40.023 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' already exists 2021-01-15T18:42:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:42:41.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-01-15T18:42:41.803 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T18:42:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:111: TEST_recovery_test_simple: wait_for_clean 2021-01-15T18:42:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:42:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:42:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:42:41.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:42:41.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:42:41.815 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:42:41.815 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:42:41.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:42:41.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:42:41.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:42:41.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:42:41.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:42:41.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:42:41.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:42:41.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:42:42.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:42:42.322 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T18:42:42.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:42:42.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:42:42.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:42:42.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836485 2021-01-15T18:42:42.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836485 2021-01-15T18:42:42.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485' 2021-01-15T18:42:42.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:42:42.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:42:42.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672964 2021-01-15T18:42:42.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672964 2021-01-15T18:42:42.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2021-01-15T18:42:42.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:42:42.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836485 2021-01-15T18:42:42.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:42:42.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:42:42.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836485 2021-01-15T18:42:42.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:42:42.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836485 2021-01-15T18:42:42.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-01-15T18:42:42.598 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836485 2021-01-15T18:42:42.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:42:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836485 2021-01-15T18:42:42.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:42:42.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672964 2021-01-15T18:42:42.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:42:42.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:42:42.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672964 2021-01-15T18:42:42.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:42:42.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672964 2021-01-15T18:42:42.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2021-01-15T18:42:42.991 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672964 2021-01-15T18:42:42.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:42:43.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672964 -lt 42949672964 2021-01-15T18:42:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:42:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:43.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:43.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:42:43.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:42:43.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:42:43.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:42:43.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:42:43.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:42:43.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:42:43.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:42:44.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:42:44.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:42:44.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:42:44.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:42:44.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:42:44.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:42:44.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:42:44.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:113: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2021-01-15T18:42:44.794 INFO:tasks.workunit.client.0.smithi145.stderr:5+0 records in 2021-01-15T18:42:44.794 INFO:tasks.workunit.client.0.smithi145.stderr:5+0 records out 2021-01-15T18:42:44.794 INFO:tasks.workunit.client.0.smithi145.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 6.7254e-05 s, 76.1 MB/s 2021-01-15T18:42:44.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: seq 1 600 2021-01-15T18:42:44.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:44.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2021-01-15T18:42:44.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:44.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2021-01-15T18:42:45.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2021-01-15T18:42:45.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2021-01-15T18:42:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2021-01-15T18:42:45.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2021-01-15T18:42:45.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2021-01-15T18:42:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2021-01-15T18:42:45.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2021-01-15T18:42:45.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2021-01-15T18:42:45.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2021-01-15T18:42:45.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2021-01-15T18:42:45.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2021-01-15T18:42:45.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:45.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2021-01-15T18:42:46.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2021-01-15T18:42:46.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2021-01-15T18:42:46.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2021-01-15T18:42:46.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2021-01-15T18:42:46.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2021-01-15T18:42:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2021-01-15T18:42:46.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2021-01-15T18:42:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2021-01-15T18:42:46.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2021-01-15T18:42:46.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2021-01-15T18:42:46.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2021-01-15T18:42:46.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:46.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2021-01-15T18:42:47.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2021-01-15T18:42:47.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2021-01-15T18:42:47.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2021-01-15T18:42:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2021-01-15T18:42:47.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2021-01-15T18:42:47.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2021-01-15T18:42:47.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2021-01-15T18:42:47.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2021-01-15T18:42:47.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2021-01-15T18:42:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2021-01-15T18:42:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2021-01-15T18:42:48.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2021-01-15T18:42:48.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2021-01-15T18:42:48.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2021-01-15T18:42:48.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2021-01-15T18:42:48.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2021-01-15T18:42:48.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2021-01-15T18:42:48.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2021-01-15T18:42:48.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2021-01-15T18:42:48.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2021-01-15T18:42:48.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2021-01-15T18:42:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2021-01-15T18:42:48.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:48.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2021-01-15T18:42:49.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2021-01-15T18:42:49.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2021-01-15T18:42:49.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2021-01-15T18:42:49.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2021-01-15T18:42:49.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2021-01-15T18:42:49.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2021-01-15T18:42:49.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2021-01-15T18:42:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2021-01-15T18:42:49.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2021-01-15T18:42:49.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2021-01-15T18:42:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2021-01-15T18:42:49.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:49.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2021-01-15T18:42:49.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2021-01-15T18:42:50.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2021-01-15T18:42:50.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2021-01-15T18:42:50.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2021-01-15T18:42:50.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2021-01-15T18:42:50.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2021-01-15T18:42:50.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2021-01-15T18:42:50.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2021-01-15T18:42:50.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2021-01-15T18:42:50.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2021-01-15T18:42:50.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:50.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2021-01-15T18:42:51.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2021-01-15T18:42:51.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2021-01-15T18:42:51.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2021-01-15T18:42:51.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2021-01-15T18:42:51.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2021-01-15T18:42:51.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2021-01-15T18:42:51.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2021-01-15T18:42:51.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2021-01-15T18:42:51.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2021-01-15T18:42:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2021-01-15T18:42:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2021-01-15T18:42:51.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:51.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2021-01-15T18:42:52.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2021-01-15T18:42:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2021-01-15T18:42:52.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2021-01-15T18:42:52.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2021-01-15T18:42:52.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2021-01-15T18:42:52.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2021-01-15T18:42:52.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2021-01-15T18:42:52.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2021-01-15T18:42:52.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2021-01-15T18:42:52.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2021-01-15T18:42:52.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2021-01-15T18:42:52.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2021-01-15T18:42:52.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:52.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2021-01-15T18:42:53.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2021-01-15T18:42:53.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2021-01-15T18:42:53.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2021-01-15T18:42:53.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2021-01-15T18:42:53.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2021-01-15T18:42:53.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2021-01-15T18:42:53.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2021-01-15T18:42:53.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2021-01-15T18:42:53.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2021-01-15T18:42:53.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2021-01-15T18:42:53.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:53.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2021-01-15T18:42:54.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2021-01-15T18:42:54.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2021-01-15T18:42:54.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2021-01-15T18:42:54.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2021-01-15T18:42:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2021-01-15T18:42:54.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2021-01-15T18:42:54.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2021-01-15T18:42:54.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2021-01-15T18:42:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:54.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2021-01-15T18:42:55.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2021-01-15T18:42:55.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2021-01-15T18:42:55.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2021-01-15T18:42:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2021-01-15T18:42:55.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2021-01-15T18:42:55.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2021-01-15T18:42:55.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2021-01-15T18:42:55.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2021-01-15T18:42:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2021-01-15T18:42:55.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2021-01-15T18:42:55.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2021-01-15T18:42:55.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:55.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2021-01-15T18:42:56.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2021-01-15T18:42:56.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2021-01-15T18:42:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2021-01-15T18:42:56.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2021-01-15T18:42:56.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2021-01-15T18:42:56.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2021-01-15T18:42:56.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2021-01-15T18:42:56.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2021-01-15T18:42:56.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2021-01-15T18:42:56.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2021-01-15T18:42:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2021-01-15T18:42:56.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:56.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2021-01-15T18:42:57.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2021-01-15T18:42:57.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2021-01-15T18:42:57.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2021-01-15T18:42:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2021-01-15T18:42:57.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2021-01-15T18:42:57.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2021-01-15T18:42:57.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2021-01-15T18:42:57.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2021-01-15T18:42:57.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2021-01-15T18:42:57.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2021-01-15T18:42:57.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2021-01-15T18:42:57.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:57.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2021-01-15T18:42:58.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2021-01-15T18:42:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2021-01-15T18:42:58.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2021-01-15T18:42:58.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2021-01-15T18:42:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2021-01-15T18:42:58.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2021-01-15T18:42:58.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2021-01-15T18:42:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2021-01-15T18:42:58.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2021-01-15T18:42:58.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2021-01-15T18:42:58.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2021-01-15T18:42:58.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:58.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2021-01-15T18:42:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2021-01-15T18:42:59.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2021-01-15T18:42:59.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2021-01-15T18:42:59.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2021-01-15T18:42:59.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2021-01-15T18:42:59.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2021-01-15T18:42:59.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2021-01-15T18:42:59.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2021-01-15T18:42:59.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2021-01-15T18:42:59.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2021-01-15T18:42:59.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2021-01-15T18:42:59.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:42:59.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2021-01-15T18:43:00.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2021-01-15T18:43:00.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2021-01-15T18:43:00.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2021-01-15T18:43:00.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2021-01-15T18:43:00.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2021-01-15T18:43:00.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2021-01-15T18:43:00.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2021-01-15T18:43:00.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2021-01-15T18:43:00.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2021-01-15T18:43:00.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:00.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2021-01-15T18:43:01.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2021-01-15T18:43:01.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2021-01-15T18:43:01.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2021-01-15T18:43:01.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2021-01-15T18:43:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2021-01-15T18:43:01.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2021-01-15T18:43:01.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2021-01-15T18:43:01.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2021-01-15T18:43:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2021-01-15T18:43:01.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2021-01-15T18:43:01.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2021-01-15T18:43:01.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:01.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2021-01-15T18:43:02.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2021-01-15T18:43:02.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2021-01-15T18:43:02.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2021-01-15T18:43:02.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2021-01-15T18:43:02.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2021-01-15T18:43:02.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2021-01-15T18:43:02.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2021-01-15T18:43:02.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2021-01-15T18:43:02.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2021-01-15T18:43:02.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2021-01-15T18:43:02.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2021-01-15T18:43:02.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:02.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2021-01-15T18:43:03.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2021-01-15T18:43:03.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2021-01-15T18:43:03.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2021-01-15T18:43:03.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2021-01-15T18:43:03.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2021-01-15T18:43:03.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2021-01-15T18:43:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2021-01-15T18:43:03.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2021-01-15T18:43:03.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2021-01-15T18:43:03.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2021-01-15T18:43:03.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2021-01-15T18:43:03.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:03.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2021-01-15T18:43:04.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2021-01-15T18:43:04.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2021-01-15T18:43:04.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2021-01-15T18:43:04.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2021-01-15T18:43:04.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2021-01-15T18:43:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2021-01-15T18:43:04.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2021-01-15T18:43:04.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2021-01-15T18:43:04.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2021-01-15T18:43:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2021-01-15T18:43:04.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:04.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2021-01-15T18:43:05.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2021-01-15T18:43:05.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2021-01-15T18:43:05.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2021-01-15T18:43:05.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2021-01-15T18:43:05.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2021-01-15T18:43:05.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2021-01-15T18:43:05.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2021-01-15T18:43:05.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2021-01-15T18:43:05.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2021-01-15T18:43:05.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2021-01-15T18:43:05.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:05.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2021-01-15T18:43:06.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2021-01-15T18:43:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2021-01-15T18:43:06.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2021-01-15T18:43:06.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2021-01-15T18:43:06.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2021-01-15T18:43:06.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2021-01-15T18:43:06.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2021-01-15T18:43:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2021-01-15T18:43:06.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2021-01-15T18:43:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2021-01-15T18:43:06.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2021-01-15T18:43:06.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2021-01-15T18:43:06.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:06.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2021-01-15T18:43:07.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2021-01-15T18:43:07.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2021-01-15T18:43:07.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2021-01-15T18:43:07.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2021-01-15T18:43:07.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2021-01-15T18:43:07.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2021-01-15T18:43:07.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2021-01-15T18:43:07.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2021-01-15T18:43:07.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2021-01-15T18:43:07.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2021-01-15T18:43:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:07.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2021-01-15T18:43:08.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2021-01-15T18:43:08.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2021-01-15T18:43:08.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2021-01-15T18:43:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2021-01-15T18:43:08.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2021-01-15T18:43:08.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2021-01-15T18:43:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2021-01-15T18:43:08.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2021-01-15T18:43:08.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2021-01-15T18:43:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2021-01-15T18:43:08.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2021-01-15T18:43:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:08.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2021-01-15T18:43:09.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2021-01-15T18:43:09.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2021-01-15T18:43:09.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2021-01-15T18:43:09.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2021-01-15T18:43:09.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2021-01-15T18:43:09.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2021-01-15T18:43:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2021-01-15T18:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2021-01-15T18:43:09.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2021-01-15T18:43:09.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2021-01-15T18:43:09.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2021-01-15T18:43:09.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2021-01-15T18:43:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2021-01-15T18:43:10.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2021-01-15T18:43:10.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2021-01-15T18:43:10.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2021-01-15T18:43:10.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2021-01-15T18:43:10.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2021-01-15T18:43:10.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2021-01-15T18:43:10.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2021-01-15T18:43:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2021-01-15T18:43:10.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2021-01-15T18:43:10.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:10.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2021-01-15T18:43:11.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2021-01-15T18:43:11.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2021-01-15T18:43:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2021-01-15T18:43:11.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2021-01-15T18:43:11.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2021-01-15T18:43:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2021-01-15T18:43:11.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2021-01-15T18:43:11.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2021-01-15T18:43:11.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2021-01-15T18:43:11.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2021-01-15T18:43:11.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2021-01-15T18:43:11.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2021-01-15T18:43:11.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:11.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2021-01-15T18:43:12.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2021-01-15T18:43:12.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2021-01-15T18:43:12.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2021-01-15T18:43:12.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2021-01-15T18:43:12.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2021-01-15T18:43:12.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2021-01-15T18:43:12.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2021-01-15T18:43:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2021-01-15T18:43:12.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2021-01-15T18:43:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2021-01-15T18:43:12.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2021-01-15T18:43:12.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:12.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2021-01-15T18:43:13.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2021-01-15T18:43:13.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2021-01-15T18:43:13.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2021-01-15T18:43:13.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2021-01-15T18:43:13.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2021-01-15T18:43:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2021-01-15T18:43:13.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2021-01-15T18:43:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2021-01-15T18:43:13.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2021-01-15T18:43:13.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2021-01-15T18:43:13.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2021-01-15T18:43:13.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:13.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2021-01-15T18:43:14.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2021-01-15T18:43:14.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2021-01-15T18:43:14.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2021-01-15T18:43:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2021-01-15T18:43:14.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2021-01-15T18:43:14.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2021-01-15T18:43:14.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2021-01-15T18:43:14.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2021-01-15T18:43:14.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2021-01-15T18:43:14.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2021-01-15T18:43:14.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2021-01-15T18:43:14.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:14.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2021-01-15T18:43:15.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2021-01-15T18:43:15.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2021-01-15T18:43:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2021-01-15T18:43:15.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2021-01-15T18:43:15.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2021-01-15T18:43:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2021-01-15T18:43:15.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2021-01-15T18:43:15.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2021-01-15T18:43:15.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2021-01-15T18:43:15.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2021-01-15T18:43:15.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2021-01-15T18:43:15.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:15.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2021-01-15T18:43:16.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2021-01-15T18:43:16.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2021-01-15T18:43:16.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2021-01-15T18:43:16.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2021-01-15T18:43:16.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2021-01-15T18:43:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2021-01-15T18:43:16.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2021-01-15T18:43:16.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2021-01-15T18:43:16.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2021-01-15T18:43:16.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2021-01-15T18:43:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2021-01-15T18:43:16.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:16.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2021-01-15T18:43:17.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2021-01-15T18:43:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2021-01-15T18:43:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2021-01-15T18:43:17.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2021-01-15T18:43:17.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2021-01-15T18:43:17.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2021-01-15T18:43:17.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2021-01-15T18:43:17.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2021-01-15T18:43:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2021-01-15T18:43:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2021-01-15T18:43:17.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2021-01-15T18:43:17.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:17.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2021-01-15T18:43:18.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2021-01-15T18:43:18.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2021-01-15T18:43:18.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2021-01-15T18:43:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2021-01-15T18:43:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2021-01-15T18:43:18.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2021-01-15T18:43:18.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2021-01-15T18:43:18.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2021-01-15T18:43:18.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2021-01-15T18:43:18.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2021-01-15T18:43:18.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2021-01-15T18:43:18.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:18.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2021-01-15T18:43:19.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2021-01-15T18:43:19.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2021-01-15T18:43:19.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2021-01-15T18:43:19.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2021-01-15T18:43:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2021-01-15T18:43:19.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2021-01-15T18:43:19.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2021-01-15T18:43:19.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2021-01-15T18:43:19.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2021-01-15T18:43:19.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2021-01-15T18:43:19.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2021-01-15T18:43:19.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:19.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2021-01-15T18:43:20.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2021-01-15T18:43:20.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2021-01-15T18:43:20.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2021-01-15T18:43:20.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2021-01-15T18:43:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2021-01-15T18:43:20.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2021-01-15T18:43:20.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2021-01-15T18:43:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:20.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2021-01-15T18:43:21.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2021-01-15T18:43:21.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2021-01-15T18:43:21.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2021-01-15T18:43:21.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2021-01-15T18:43:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2021-01-15T18:43:21.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2021-01-15T18:43:21.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2021-01-15T18:43:21.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2021-01-15T18:43:21.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2021-01-15T18:43:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2021-01-15T18:43:21.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2021-01-15T18:43:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:21.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2021-01-15T18:43:22.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2021-01-15T18:43:22.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2021-01-15T18:43:22.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2021-01-15T18:43:22.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2021-01-15T18:43:22.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2021-01-15T18:43:22.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2021-01-15T18:43:22.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2021-01-15T18:43:22.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2021-01-15T18:43:22.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2021-01-15T18:43:22.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2021-01-15T18:43:22.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2021-01-15T18:43:22.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:22.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2021-01-15T18:43:23.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2021-01-15T18:43:23.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2021-01-15T18:43:23.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2021-01-15T18:43:23.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2021-01-15T18:43:23.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2021-01-15T18:43:23.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2021-01-15T18:43:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2021-01-15T18:43:23.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2021-01-15T18:43:23.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2021-01-15T18:43:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2021-01-15T18:43:23.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2021-01-15T18:43:23.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:23.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2021-01-15T18:43:24.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2021-01-15T18:43:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2021-01-15T18:43:24.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2021-01-15T18:43:24.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2021-01-15T18:43:24.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2021-01-15T18:43:24.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2021-01-15T18:43:24.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2021-01-15T18:43:24.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2021-01-15T18:43:24.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2021-01-15T18:43:24.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2021-01-15T18:43:24.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2021-01-15T18:43:24.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:24.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2021-01-15T18:43:25.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2021-01-15T18:43:25.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2021-01-15T18:43:25.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2021-01-15T18:43:25.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2021-01-15T18:43:25.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2021-01-15T18:43:25.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2021-01-15T18:43:25.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2021-01-15T18:43:25.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2021-01-15T18:43:25.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2021-01-15T18:43:25.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:25.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2021-01-15T18:43:26.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2021-01-15T18:43:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2021-01-15T18:43:26.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2021-01-15T18:43:26.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2021-01-15T18:43:26.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2021-01-15T18:43:26.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2021-01-15T18:43:26.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2021-01-15T18:43:26.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2021-01-15T18:43:26.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2021-01-15T18:43:26.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2021-01-15T18:43:26.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2021-01-15T18:43:26.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:26.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2021-01-15T18:43:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2021-01-15T18:43:27.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2021-01-15T18:43:27.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2021-01-15T18:43:27.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2021-01-15T18:43:27.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2021-01-15T18:43:27.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2021-01-15T18:43:27.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2021-01-15T18:43:27.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2021-01-15T18:43:27.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2021-01-15T18:43:27.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2021-01-15T18:43:27.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2021-01-15T18:43:27.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:27.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2021-01-15T18:43:28.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2021-01-15T18:43:28.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2021-01-15T18:43:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2021-01-15T18:43:28.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2021-01-15T18:43:28.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2021-01-15T18:43:28.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2021-01-15T18:43:28.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2021-01-15T18:43:28.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2021-01-15T18:43:28.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2021-01-15T18:43:28.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2021-01-15T18:43:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2021-01-15T18:43:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2021-01-15T18:43:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:28.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2021-01-15T18:43:29.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2021-01-15T18:43:29.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2021-01-15T18:43:29.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2021-01-15T18:43:29.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2021-01-15T18:43:29.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2021-01-15T18:43:29.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2021-01-15T18:43:29.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2021-01-15T18:43:29.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2021-01-15T18:43:29.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2021-01-15T18:43:29.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2021-01-15T18:43:29.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:29.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2021-01-15T18:43:30.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2021-01-15T18:43:30.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2021-01-15T18:43:30.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2021-01-15T18:43:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2021-01-15T18:43:30.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2021-01-15T18:43:30.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2021-01-15T18:43:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2021-01-15T18:43:30.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2021-01-15T18:43:30.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2021-01-15T18:43:30.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2021-01-15T18:43:30.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:30.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2021-01-15T18:43:31.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2021-01-15T18:43:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2021-01-15T18:43:31.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2021-01-15T18:43:31.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2021-01-15T18:43:31.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2021-01-15T18:43:31.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2021-01-15T18:43:31.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2021-01-15T18:43:31.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2021-01-15T18:43:31.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2021-01-15T18:43:31.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2021-01-15T18:43:31.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2021-01-15T18:43:31.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:31.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2021-01-15T18:43:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2021-01-15T18:43:32.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2021-01-15T18:43:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2021-01-15T18:43:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2021-01-15T18:43:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2021-01-15T18:43:32.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2021-01-15T18:43:32.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2021-01-15T18:43:32.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2021-01-15T18:43:32.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2021-01-15T18:43:32.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2021-01-15T18:43:32.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:32.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2021-01-15T18:43:33.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2021-01-15T18:43:33.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2021-01-15T18:43:33.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2021-01-15T18:43:33.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2021-01-15T18:43:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2021-01-15T18:43:33.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2021-01-15T18:43:33.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2021-01-15T18:43:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2021-01-15T18:43:33.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2021-01-15T18:43:33.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2021-01-15T18:43:33.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2021-01-15T18:43:33.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:33.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2021-01-15T18:43:34.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2021-01-15T18:43:34.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2021-01-15T18:43:34.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2021-01-15T18:43:34.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2021-01-15T18:43:34.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2021-01-15T18:43:34.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2021-01-15T18:43:34.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2021-01-15T18:43:34.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2021-01-15T18:43:34.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2021-01-15T18:43:34.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2021-01-15T18:43:34.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:34.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2021-01-15T18:43:35.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2021-01-15T18:43:35.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2021-01-15T18:43:35.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2021-01-15T18:43:35.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2021-01-15T18:43:35.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2021-01-15T18:43:35.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2021-01-15T18:43:35.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2021-01-15T18:43:35.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2021-01-15T18:43:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2021-01-15T18:43:35.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2021-01-15T18:43:35.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2021-01-15T18:43:35.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:35.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2021-01-15T18:43:36.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:36.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2021-01-15T18:43:36.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:36.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2021-01-15T18:43:36.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:36.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2021-01-15T18:43:36.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2021-01-15T18:43:36.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2021-01-15T18:43:36.371 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: expr 2 - 1 2021-01-15T18:43:36.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: seq 0 1 2021-01-15T18:43:36.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:43:36.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2021-01-15T18:43:36.499 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:43:36.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2021-01-15T18:43:36.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2021-01-15T18:43:36.635 INFO:tasks.workunit.client.0.smithi145.stdout:{} 2021-01-15T18:43:36.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:123: TEST_recovery_test_simple: sleep 5 2021-01-15T18:43:41.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:125: TEST_recovery_test_simple: ceph pg dump pgs 2021-01-15T18:43:42.019 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:43:42.020 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 0 0 0 3072000 0 0 600 600 active+clean 2021-01-15T18:42:42.209859+0000 23'600 23:617 [1] 1 [1] 1 0'0 2021-01-15T18:42:39.643174+0000 0'0 2021-01-15T18:42:39.643174+0000 0 2021-01-15T18:43:42.020 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:43:42.020 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:43:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:43:42.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:127: TEST_recovery_test_simple: seq 1 1 2021-01-15T18:43:42.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:127: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2021-01-15T18:43:42.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:129: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2021-01-15T18:43:42.718 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:43:42.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:133: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2021-01-15T18:43:42.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:76: wait_for_recovery_toofull: local timeout=30 2021-01-15T18:43:42.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2021-01-15T18:43:42.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:51: wait_for_state: local state=recovery_toofull 2021-01-15T18:43:42.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local cur_in_state 2021-01-15T18:43:42.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: delays=($(get_timeout_delays $2 5)) 2021-01-15T18:43:42.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: get_timeout_delays 30 5 2021-01-15T18:43:42.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:43:42.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:43:42.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:43:42.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:43:42.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local -a delays 2021-01-15T18:43:42.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -i loop=0 2021-01-15T18:43:42.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:56: wait_for_state: flush_pg_stats 2021-01-15T18:43:42.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:43:42.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:43:43.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:43:43.242 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T18:43:43.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:43:43.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:43:43.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:43:43.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836499 2021-01-15T18:43:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836499 2021-01-15T18:43:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499' 2021-01-15T18:43:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:43:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:43:43.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672978 2021-01-15T18:43:43.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672978 2021-01-15T18:43:43.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2021-01-15T18:43:43.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:43:43.517 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836499 2021-01-15T18:43:43.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:43:43.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:43:43.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836499 2021-01-15T18:43:43.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:43:43.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836499 2021-01-15T18:43:43.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-01-15T18:43:43.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:43:43.526 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836499 2021-01-15T18:43:43.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836499 2021-01-15T18:43:43.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:43:44.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:43:44.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:43:45.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836499 2021-01-15T18:43:45.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:43:45.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672978 2021-01-15T18:43:45.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:43:45.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:43:45.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672978 2021-01-15T18:43:45.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:43:45.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672978 2021-01-15T18:43:45.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2021-01-15T18:43:45.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:43:45.354 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672978 2021-01-15T18:43:45.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672978 2021-01-15T18:43:45.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: get_num_pgs 2021-01-15T18:43:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:43:45.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:43:46.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: test 1 == 0 2021-01-15T18:43:46.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:61: wait_for_state: true 2021-01-15T18:43:46.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: get_num_in_state recovery_toofull 2021-01-15T18:43:46.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:42: get_num_in_state: local state=recovery_toofull 2021-01-15T18:43:46.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local expression 2021-01-15T18:43:46.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2021-01-15T18:43:46.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: ceph --format json pg dump pgs 2021-01-15T18:43:46.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2021-01-15T18:43:46.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: cur_in_state=1 2021-01-15T18:43:46.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: test 1 -gt 0 2021-01-15T18:43:46.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: break 2021-01-15T18:43:46.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:71: wait_for_state: return 0 2021-01-15T18:43:46.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:135: TEST_recovery_test_simple: ERRORS=0 2021-01-15T18:43:46.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ceph pg dump pgs 2021-01-15T18:43:46.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: grep +recovery_toofull 2021-01-15T18:43:46.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: wc -l 2021-01-15T18:43:47.018 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:43:47.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2021-01-15T18:43:47.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:142: TEST_recovery_test_simple: ceph pg dump pgs 2021-01-15T18:43:47.394 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T18:43:47.395 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 600 0 600 0 0 3072000 0 0 600 600 active+recovery_toofull+undersized+degraded+remapped 2021-01-15T18:43:43.293223+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2021-01-15T18:42:39.643174+0000 0'0 2021-01-15T18:42:39.643174+0000 0 2021-01-15T18:43:47.395 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:43:47.395 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T18:43:47.396 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:43:47.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph status 2021-01-15T18:43:47.904 INFO:tasks.workunit.client.0.smithi145.stdout: cluster: 2021-01-15T18:43:47.904 INFO:tasks.workunit.client.0.smithi145.stdout: id: cfee71ee-f841-4191-a39b-9576992ea7f1 2021-01-15T18:43:47.905 INFO:tasks.workunit.client.0.smithi145.stdout: health: HEALTH_ERR 2021-01-15T18:43:47.905 INFO:tasks.workunit.client.0.smithi145.stdout: 2 full osd(s) 2021-01-15T18:43:47.906 INFO:tasks.workunit.client.0.smithi145.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2021-01-15T18:43:47.906 INFO:tasks.workunit.client.0.smithi145.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2021-01-15T18:43:47.906 INFO:tasks.workunit.client.0.smithi145.stdout: 1 pool(s) do not have an application enabled 2021-01-15T18:43:47.907 INFO:tasks.workunit.client.0.smithi145.stdout: 1 pool(s) full 2021-01-15T18:43:47.907 INFO:tasks.workunit.client.0.smithi145.stdout: OSD count 2 < osd_pool_default_size 3 2021-01-15T18:43:47.908 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:43:47.908 INFO:tasks.workunit.client.0.smithi145.stdout: services: 2021-01-15T18:43:47.908 INFO:tasks.workunit.client.0.smithi145.stdout: mon: 1 daemons, quorum a (age 87s) 2021-01-15T18:43:47.909 INFO:tasks.workunit.client.0.smithi145.stdout: mgr: x(active, since 83s) 2021-01-15T18:43:47.909 INFO:tasks.workunit.client.0.smithi145.stdout: osd: 2 osds: 2 up (since 71s), 2 in (since 71s); 1 remapped pgs 2021-01-15T18:43:47.910 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:43:47.910 INFO:tasks.workunit.client.0.smithi145.stdout: data: 2021-01-15T18:43:47.911 INFO:tasks.workunit.client.0.smithi145.stdout: pools: 1 pools, 1 pgs 2021-01-15T18:43:47.911 INFO:tasks.workunit.client.0.smithi145.stdout: objects: 600 objects, 2.9 MiB 2021-01-15T18:43:47.911 INFO:tasks.workunit.client.0.smithi145.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2021-01-15T18:43:47.912 INFO:tasks.workunit.client.0.smithi145.stdout: pgs: 600/1200 objects degraded (50.000%) 2021-01-15T18:43:47.912 INFO:tasks.workunit.client.0.smithi145.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2021-01-15T18:43:47.913 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T18:43:47.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status --format=json-pretty 2021-01-15T18:43:48.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2021-01-15T18:43:48.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2021-01-15T18:43:48.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:146: TEST_recovery_test_simple: SEV=HEALTH_ERR 2021-01-15T18:43:48.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2021-01-15T18:43:48.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2021-01-15T18:43:48.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2021-01-15T18:43:48.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2021-01-15T18:43:48.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2021-01-15T18:43:48.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:156: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2021-01-15T18:43:48.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:158: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2021-01-15T18:43:48.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:163: TEST_recovery_test_simple: seq 1 1 2021-01-15T18:43:48.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:163: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2021-01-15T18:43:48.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:165: TEST_recovery_test_simple: delete_pool test1 2021-01-15T18:43:48.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test1 2021-01-15T18:43:48.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-01-15T18:43:48.961 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test1' does not exist 2021-01-15T18:43:48.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:167: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2021-01-15T18:43:48.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:43:48.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:43:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:43:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:43:48.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:43:49.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:43:49.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: teardown td/osd-recovery-space 2021-01-15T18:43:49.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-01-15T18:43:49.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:43:49.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-01-15T18:43:49.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:43:49.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:43:49.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:43:49.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:43:49.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:43:49.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:43:49.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:43:49.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:43:49.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:43:49.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:43:49.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:43:49.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:43:49.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:43:49.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:43:49.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:43:49.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:43:49.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-01-15T18:43:49.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:43:49.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:43:49.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.821914 2021-01-15T18:43:49.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:43:49.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:43:49.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:43:49.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-recovery-space 0 2021-01-15T18:43:49.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2021-01-15T18:43:49.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:43:49.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2021-01-15T18:43:49.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:43:49.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:43:49.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:43:49.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:43:49.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:43:49.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:43:49.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:43:49.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:43:49.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:43:49.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:43:49.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:43:49.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:43:49.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:43:49.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:43:49.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:43:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:43:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2021-01-15T18:43:49.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:43:49.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.821914 2021-01-15T18:43:49.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.821914 2021-01-15T18:43:49.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:43:49.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:43:49.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:43:49.118 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:43:49.118 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:43:49.142 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2021-01-15T18:43:49.142 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2021-01-15T18:43:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:43:49.210 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:43:49.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:43:49.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:43:49.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:43:49.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:43:49.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:43:49.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:43:49.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:43:49.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:43:49.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-recovery-stats 2021-01-15T18:43:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2021-01-15T18:43:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2021-01-15T18:43:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2021-01-15T18:43:49.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2021-01-15T18:43:49.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2021-01-15T18:43:49.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none ' 2021-01-15T18:43:49.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2021-01-15T18:43:49.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:43:49.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2021-01-15T18:43:49.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2021-01-15T18:43:49.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2021-01-15T18:43:49.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2021-01-15T18:43:49.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2021-01-15T18:43:49.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2021-01-15T18:43:49.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2021-01-15T18:43:49.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:43:49.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2021-01-15T18:43:49.218 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_erasure_remapped 2021-01-15T18:43:49.218 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_multi 2021-01-15T18:43:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_replicated_out1 2021-01-15T18:43:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_sizedown 2021-01-15T18:43:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_sizeup 2021-01-15T18:43:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_recovery_undersized' 2021-01-15T18:43:49.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:43:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:43:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:43:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:43:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:43:49.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:43:49.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:43:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:43:49.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:43:49.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:43:49.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:43:49.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:43:49.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:43:49.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:43:49.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:43:49.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:43:49.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:43:49.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:43:49.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:43:49.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:43:49.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:43:49.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:43:49.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:43:49.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:43:49.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:49.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:43:49.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:43:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:43:49.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:43:49.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:43:49.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:49.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:43:49.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:43:49.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:43:49.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:43:49.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:43:49.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:43:49.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2021-01-15T18:43:49.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2021-01-15T18:43:49.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2021-01-15T18:43:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2021-01-15T18:43:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2021-01-15T18:43:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:43:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:43:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:43:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:43:49.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:43:49.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:43:49.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:43:49.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:43:49.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:43:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:49.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:49.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:43:49.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:43:49.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:43:49.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:43:49.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:43:49.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:43:49.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:43:49.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:43:49.716 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:43:49.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:43:49.717 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:43:49.717 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.717 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:49.717 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:43:49.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:43:49.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:43:49.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:43:49.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:43:49.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:43:49.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:43:49.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:43:49.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:43:49.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:43:49.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:43:49.818 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:43:49.818 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:49.818 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:49.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:43:49.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:43:49.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:43:49.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2021-01-15T18:43:49.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:43:49.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:43:49.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:43:49.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:43:49.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:43:49.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:43:50.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:43:50.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:43:50.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:43:50.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:43:50.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:50.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:50.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:50.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:43:50.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:43:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2021-01-15T18:43:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:43:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:43:50.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:43:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:43:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:43:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:43:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:43:50.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:43:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:43:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:43:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:43:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:43:50.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:43:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:43:50.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:43:50.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:43:50.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:43:50.852 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:43:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:50.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:50.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:43:50.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:43:50.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:43:50.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:43:50.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:43:50.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:43:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:43:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:43:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:43:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:43:50.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:43:50.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:43:50.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=70aeaf97-0247-4776-9542-2dddba35e0df 2021-01-15T18:43:50.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 70aeaf97-0247-4776-9542-2dddba35e0df' 2021-01-15T18:43:50.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:43:50.857 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 70aeaf97-0247-4776-9542-2dddba35e0df 2021-01-15T18:43:50.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBm4gFg6Jx2MxAAx3Ki8T2iGbLqere1VJj4DQ== 2021-01-15T18:43:50.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBm4gFg6Jx2MxAAx3Ki8T2iGbLqere1VJj4DQ=="}' 2021-01-15T18:43:50.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 70aeaf97-0247-4776-9542-2dddba35e0df -i td/osd-recovery-stats/0/new.json 2021-01-15T18:43:51.206 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:43:51.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:43:51.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBm4gFg6Jx2MxAAx3Ki8T2iGbLqere1VJj4DQ== --osd-uuid 70aeaf97-0247-4776-9542-2dddba35e0df 2021-01-15T18:43:51.240 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.238+0000 7fad80a1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:51.242 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.240+0000 7fad80a1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:51.244 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.242+0000 7fad80a1c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.243+0000 7fad80a1c080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:43:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.243+0000 7fad80a1c080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:43:51.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.243+0000 7fad80a1c080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:43:51.288 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:51.286+0000 7fad80a1c080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:43:56.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:43:56.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:43:56.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:43:56.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:43:56.025 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:43:56.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:43:56.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:43:56.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:43:56.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:43:56.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:43:56.550 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:43:56.570 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:56.569+0000 7f080245b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:56.605 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:56.604+0000 7f080245b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:56.606+0000 7f080245b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:56.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:43:56.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:43:56.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:43:56.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:43:56.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:43:56.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:43:56.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:43:56.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:43:56.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:43:56.982 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:43:57.122 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:57.121+0000 7f080245b080 -1 Falling back to public interface 2021-01-15T18:43:57.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:43:57.991 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:57.990+0000 7f080245b080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:43:58.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:43:58.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:43:58.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:43:58.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:43:58.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:43:58.368 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:43:58.776 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/838752,v1:127.0.0.1:6803/838752] [v2:127.0.0.1:6804/838752,v1:127.0.0.1:6805/838752] exists,up 70aeaf97-0247-4776-9542-2dddba35e0df 2021-01-15T18:43:58.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:43:58.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:43:58.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:43:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2021-01-15T18:43:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:43:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:43:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:43:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:43:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:43:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:43:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:43:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:43:58.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:43:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:43:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:43:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:43:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:43:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:43:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:43:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:43:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:43:58.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:43:58.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:43:58.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:43:58.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:58.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:43:58.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:43:58.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:43:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:43:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:43:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:43:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:43:58.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:43:58.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:43:58.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:43:58.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:43:58.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:43:58.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:43:58.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9883a254-61f4-4e2d-ab77-1b219b6652fd 2021-01-15T18:43:58.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 9883a254-61f4-4e2d-ab77-1b219b6652fd' 2021-01-15T18:43:58.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:43:58.786 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 9883a254-61f4-4e2d-ab77-1b219b6652fd 2021-01-15T18:43:58.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBu4gFgqf8gLxAA/5S8135n9go1DsqYFEr/cw== 2021-01-15T18:43:58.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBu4gFgqf8gLxAA/5S8135n9go1DsqYFEr/cw=="}' 2021-01-15T18:43:58.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9883a254-61f4-4e2d-ab77-1b219b6652fd -i td/osd-recovery-stats/1/new.json 2021-01-15T18:43:59.289 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:43:59.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:43:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBu4gFgqf8gLxAA/5S8135n9go1DsqYFEr/cw== --osd-uuid 9883a254-61f4-4e2d-ab77-1b219b6652fd 2021-01-15T18:43:59.323 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.322+0000 7f8973552080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:59.328 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.327+0000 7f8973552080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:59.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.329+0000 7f8973552080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:43:59.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.330+0000 7f8973552080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:43:59.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.330+0000 7f8973552080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:43:59.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.330+0000 7f8973552080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:43:59.453 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:43:59.452+0000 7f8973552080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:44:04.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:44:04.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:44:04.212 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:44:04.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:44:04.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:44:04.734 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:44:04.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:44:04.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:44:04.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:04.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:44:04.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:44:04.755 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:04.754+0000 7f26abb4a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:04.768 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:04.767+0000 7f26abb4a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:04.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:04.769+0000 7f26abb4a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:44:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:44:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:44:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:44:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:44:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:44:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:05.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:44:05.166 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:44:05.285 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:05.285+0000 7f26abb4a080 -1 Falling back to public interface 2021-01-15T18:44:05.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:44:06.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:06.085+0000 7f26abb4a080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:44:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:06.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:44:06.558 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:44:06.983 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/840016,v1:127.0.0.1:6811/840016] [v2:127.0.0.1:6812/840016,v1:127.0.0.1:6813/840016] exists,up 9883a254-61f4-4e2d-ab77-1b219b6652fd 2021-01-15T18:44:06.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:44:06.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:44:06.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:44:06.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2021-01-15T18:44:06.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:44:06.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:44:06.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:44:06.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:44:06.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:44:06.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:44:06.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:44:06.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:44:06.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:44:06.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:44:06.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:44:06.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:44:06.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:44:06.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:44:06.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:44:06.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:44:06.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:44:06.987 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:44:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:44:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:44:06.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:44:06.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:44:06.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:44:06.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:44:06.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:44:06.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:44:06.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:44:06.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:06.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:44:06.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:44:06.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:44:06.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:44:06.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8723720c-eb50-43d0-b0a3-fc46646eae98 2021-01-15T18:44:06.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 8723720c-eb50-43d0-b0a3-fc46646eae98' 2021-01-15T18:44:06.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:44:06.992 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 8723720c-eb50-43d0-b0a3-fc46646eae98 2021-01-15T18:44:06.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB24gFgzj2FOxAA38Dx5iACImavp0hbRB8cDA== 2021-01-15T18:44:06.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB24gFgzj2FOxAA38Dx5iACImavp0hbRB8cDA=="}' 2021-01-15T18:44:06.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8723720c-eb50-43d0-b0a3-fc46646eae98 -i td/osd-recovery-stats/2/new.json 2021-01-15T18:44:07.520 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:44:07.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:44:07.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB24gFgzj2FOxAA38Dx5iACImavp0hbRB8cDA== --osd-uuid 8723720c-eb50-43d0-b0a3-fc46646eae98 2021-01-15T18:44:07.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.554+0000 7f11023fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:07.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.584+0000 7f11023fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:07.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.586+0000 7f11023fb080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:07.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.586+0000 7f11023fb080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:44:07.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.587+0000 7f11023fb080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:44:07.650 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.587+0000 7f11023fb080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:44:07.651 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:07.650+0000 7f11023fb080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:44:12.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:44:12.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:44:12.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:44:12.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:44:12.707 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:44:13.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:44:13.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:13.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:44:13.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:44:13.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:44:13.240 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:44:13.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:13.259+0000 7fc14da4d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:13.284 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:13.283+0000 7fc14da4d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:13.285 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:13.284+0000 7fc14da4d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:44:13.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:44:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:44:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:44:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:44:13.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:13.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:44:13.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:13.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:44:13.631 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:44:13.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:13.800+0000 7fc14da4d080 -1 Falling back to public interface 2021-01-15T18:44:14.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:44:14.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:14.626+0000 7fc14da4d080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:44:15.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:44:15.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:15.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:44:15.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:15.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:44:15.047 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:44:15.453 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/841280,v1:127.0.0.1:6819/841280] [v2:127.0.0.1:6820/841280,v1:127.0.0.1:6821/841280] exists,up 8723720c-eb50-43d0-b0a3-fc46646eae98 2021-01-15T18:44:15.453 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 df8c5328-4cdb-4a83-9416-0993cee04054 2021-01-15T18:44:15.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:44:15.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:44:15.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:44:15.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2021-01-15T18:44:15.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:44:15.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:44:15.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:44:15.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:44:15.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:44:15.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:44:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:44:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:44:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:44:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:44:15.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:44:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:44:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:44:15.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:44:15.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:44:15.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:44:15.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:44:15.458 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:44:15.458 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:44:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:44:15.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:15.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:44:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:44:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:44:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:44:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:44:15.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:44:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:44:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:44:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:44:15.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:44:15.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:44:15.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=df8c5328-4cdb-4a83-9416-0993cee04054 2021-01-15T18:44:15.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 df8c5328-4cdb-4a83-9416-0993cee04054' 2021-01-15T18:44:15.463 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:44:15.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB/4gFgmaPXGxAAcBrvhW688Zhmro4rbcPMEQ== 2021-01-15T18:44:15.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB/4gFgmaPXGxAAcBrvhW688Zhmro4rbcPMEQ=="}' 2021-01-15T18:44:15.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new df8c5328-4cdb-4a83-9416-0993cee04054 -i td/osd-recovery-stats/3/new.json 2021-01-15T18:44:16.044 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:44:16.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:44:16.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB/4gFgmaPXGxAAcBrvhW688Zhmro4rbcPMEQ== --osd-uuid df8c5328-4cdb-4a83-9416-0993cee04054 2021-01-15T18:44:16.078 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.077+0000 7f3762856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:16.079 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.079+0000 7f3762856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:16.081 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.080+0000 7f3762856080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.081+0000 7f3762856080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:44:16.082 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.081+0000 7f3762856080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:44:16.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.081+0000 7f3762856080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:44:16.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:16.174+0000 7f3762856080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:44:20.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:44:20.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:44:20.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:44:20.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:44:20.713 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:44:21.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:44:21.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:21.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:44:21.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:44:21.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:44:21.243 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:44:21.263 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:21.262+0000 7f112ff66080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:21.266 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:21.266+0000 7f112ff66080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:21.268 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:21.267+0000 7f112ff66080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:21.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:44:21.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:44:21.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:44:21.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:44:21.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:44:21.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:21.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:44:21.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:21.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:44:21.636 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:44:21.786 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:21.785+0000 7f112ff66080 -1 Falling back to public interface 2021-01-15T18:44:22.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:44:22.828 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:22.827+0000 7f112ff66080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:44:23.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:44:23.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:23.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:44:23.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:23.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:44:23.058 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:44:23.504 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/842544,v1:127.0.0.1:6827/842544] [v2:127.0.0.1:6828/842544,v1:127.0.0.1:6829/842544] exists,up df8c5328-4cdb-4a83-9416-0993cee04054 2021-01-15T18:44:23.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:44:23.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:44:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:44:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2021-01-15T18:44:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:44:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:44:23.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:44:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:44:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/4 2021-01-15T18:44:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:44:23.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:44:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:44:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:44:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-01-15T18:44:23.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-01-15T18:44:23.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:44:23.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:44:23.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:44:23.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:44:23.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:44:23.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:44:23.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:44:23.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:44:23.509 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:44:23.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:23.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:23.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:44:23.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:44:23.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:44:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:44:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:44:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:44:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:44:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:23.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:44:23.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:44:23.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/4 2021-01-15T18:44:23.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:44:23.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=9ce97868-4230-438b-81d7-facd59e63356 2021-01-15T18:44:23.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 9ce97868-4230-438b-81d7-facd59e63356' 2021-01-15T18:44:23.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:44:23.514 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 9ce97868-4230-438b-81d7-facd59e63356 2021-01-15T18:44:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCH4gFg0pcEHxAAAkxYRR1pwEqps/VLQltjXw== 2021-01-15T18:44:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCH4gFg0pcEHxAAAkxYRR1pwEqps/VLQltjXw=="}' 2021-01-15T18:44:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 9ce97868-4230-438b-81d7-facd59e63356 -i td/osd-recovery-stats/4/new.json 2021-01-15T18:44:24.073 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:44:24.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/4/new.json 2021-01-15T18:44:24.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCH4gFg0pcEHxAAAkxYRR1pwEqps/VLQltjXw== --osd-uuid 9ce97868-4230-438b-81d7-facd59e63356 2021-01-15T18:44:24.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.107+0000 7f21fddec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:24.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.109+0000 7f21fddec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:24.112 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.110+0000 7f21fddec080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:24.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.112+0000 7f21fddec080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:44:24.113 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.112+0000 7f21fddec080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:44:24.114 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.112+0000 7f21fddec080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:44:24.183 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:24.183+0000 7f21fddec080 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-01-15T18:44:28.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-01-15T18:44:29.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:44:29.057 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:44:29.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:44:29.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:44:29.490 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:44:29.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:44:29.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:44:29.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:29.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:44:29.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:44:29.511 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:29.510+0000 7f5527bcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:29.542 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:29.540+0000 7f5527bcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:29.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:29.542+0000 7f5527bcf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:29.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:44:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:44:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:44:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:44:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:44:29.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:29.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:44:29.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:29.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:44:29.884 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:44:30.066 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:30.063+0000 7f5527bcf080 -1 Falling back to public interface 2021-01-15T18:44:30.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:44:30.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:30.901+0000 7f5527bcf080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:44:31.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:44:31.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:31.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:44:31.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:31.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:44:31.293 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:44:31.711 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/843808,v1:127.0.0.1:6835/843808] [v2:127.0.0.1:6836/843808,v1:127.0.0.1:6837/843808] exists,up 9ce97868-4230-438b-81d7-facd59e63356 2021-01-15T18:44:31.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:44:31.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:44:31.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:44:31.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2021-01-15T18:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:44:31.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:44:31.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:44:31.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:44:31.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/5 2021-01-15T18:44:31.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:44:31.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:44:31.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:44:31.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:44:31.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-01-15T18:44:31.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-01-15T18:44:31.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:44:31.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:44:31.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:44:31.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:44:31.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:44:31.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:44:31.736 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:44:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:44:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:44:31.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:31.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:44:31.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:44:31.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:44:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:44:31.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:44:31.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:44:31.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:44:31.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:44:31.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:31.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:44:31.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:44:31.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/5 2021-01-15T18:44:31.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:44:31.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2786cbf0-f98f-4fec-9825-0d04ed2d5930 2021-01-15T18:44:31.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 2786cbf0-f98f-4fec-9825-0d04ed2d5930' 2021-01-15T18:44:31.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:44:31.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCP4gFgoutDKxAACVGYR19dhS1DCtJ7GEkMxw== 2021-01-15T18:44:31.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCP4gFgoutDKxAACVGYR19dhS1DCtJ7GEkMxw=="}' 2021-01-15T18:44:31.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2786cbf0-f98f-4fec-9825-0d04ed2d5930 -i td/osd-recovery-stats/5/new.json 2021-01-15T18:44:31.747 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 2786cbf0-f98f-4fec-9825-0d04ed2d5930 2021-01-15T18:44:32.330 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:44:32.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/5/new.json 2021-01-15T18:44:32.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCP4gFgoutDKxAACVGYR19dhS1DCtJ7GEkMxw== --osd-uuid 2786cbf0-f98f-4fec-9825-0d04ed2d5930 2021-01-15T18:44:32.364 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.363+0000 7f2e240bf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:32.366 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.365+0000 7f2e240bf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:32.367 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.366+0000 7f2e240bf080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:32.368 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.367+0000 7f2e240bf080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:44:32.369 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.367+0000 7f2e240bf080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:44:32.369 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.367+0000 7f2e240bf080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:44:32.416 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:32.415+0000 7f2e240bf080 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-01-15T18:44:37.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-01-15T18:44:37.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:44:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:44:37.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:44:37.414 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:44:37.933 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:44:37.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:44:37.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:44:37.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:44:37.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:44:37.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:44:37.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:37.953+0000 7fdaf1208080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:37.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:37.955+0000 7fdaf1208080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:37.958 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:37.957+0000 7fdaf1208080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:44:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:44:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:44:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:44:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:44:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:44:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:44:38.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:38.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:44:38.326 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:44:38.474 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:38.473+0000 7fdaf1208080 -1 Falling back to public interface 2021-01-15T18:44:38.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:44:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:44:39.269+0000 7fdaf1208080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:44:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:44:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:44:39.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:44:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:44:39.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:44:39.721 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:44:40.191 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/845080,v1:127.0.0.1:6843/845080] [v2:127.0.0.1:6844/845080,v1:127.0.0.1:6845/845080] exists,up 2786cbf0-f98f-4fec-9825-0d04ed2d5930 2021-01-15T18:44:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:44:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:44:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:44:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2021-01-15T18:44:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T18:44:40.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2021-01-15T18:44:40.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T18:44:41.470 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:44:41.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:44:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2021-01-15T18:44:42.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:44:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:44:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:44:42.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:44:42.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:44:42.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:44:42.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:44:42.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:44:42.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:44:42.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:44:42.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:44:42.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:44:42.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:44:42.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:44:42.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:44:43.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:44:43.055 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:44:43.055 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:44:43.056 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:44:43.056 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:44:43.056 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:44:43.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:44:43.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:44:43.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T18:44:43.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T18:44:43.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T18:44:43.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:44:43.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705674 2021-01-15T18:44:43.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705674 2021-01-15T18:44:43.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2021-01-15T18:44:43.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:44:43.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542152 2021-01-15T18:44:43.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542152 2021-01-15T18:44:43.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542152' 2021-01-15T18:44:43.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:44:43.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378631 2021-01-15T18:44:43.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378631 2021-01-15T18:44:43.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542152 3-81604378631' 2021-01-15T18:44:43.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:44:43.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247814 2021-01-15T18:44:43.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247814 2021-01-15T18:44:43.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542152 3-81604378631 4-98784247814' 2021-01-15T18:44:43.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:44:43.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:44:43.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084292 2021-01-15T18:44:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084292 2021-01-15T18:44:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-60129542152 3-81604378631 4-98784247814 5-120259084292' 2021-01-15T18:44:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:43.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T18:44:43.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:43.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:44:43.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T18:44:43.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T18:44:43.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T18:44:43.881 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T18:44:43.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:44:44.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836491 2021-01-15T18:44:44.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:44:45.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:44:45.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:44:45.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836491 2021-01-15T18:44:45.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:45.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705674 2021-01-15T18:44:45.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:45.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:44:45.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705674 2021-01-15T18:44:45.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:45.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705674 2021-01-15T18:44:45.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2021-01-15T18:44:45.675 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 38654705674 2021-01-15T18:44:45.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:44:46.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705674 -lt 38654705674 2021-01-15T18:44:46.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:46.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542152 2021-01-15T18:44:46.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:46.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:44:46.080 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542152 2021-01-15T18:44:46.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:46.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542152 2021-01-15T18:44:46.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2021-01-15T18:44:46.082 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 60129542152 2021-01-15T18:44:46.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:44:46.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542152 -lt 60129542152 2021-01-15T18:44:46.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:46.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378631 2021-01-15T18:44:46.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:46.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:44:46.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378631 2021-01-15T18:44:46.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:46.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378631 2021-01-15T18:44:46.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378631' 2021-01-15T18:44:46.500 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378631 2021-01-15T18:44:46.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:44:46.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378632 -lt 81604378631 2021-01-15T18:44:46.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:46.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247814 2021-01-15T18:44:46.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:46.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:44:46.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247814 2021-01-15T18:44:46.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:46.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247814 2021-01-15T18:44:46.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2021-01-15T18:44:46.925 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 98784247814 2021-01-15T18:44:46.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:44:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247814 -lt 98784247814 2021-01-15T18:44:47.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:44:47.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084292 2021-01-15T18:44:47.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:44:47.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:44:47.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084292 2021-01-15T18:44:47.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:44:47.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084292 2021-01-15T18:44:47.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084292' 2021-01-15T18:44:47.322 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 120259084292 2021-01-15T18:44:47.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:44:47.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084293 -lt 120259084292 2021-01-15T18:44:47.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:44:47.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:44:47.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:44:48.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:44:48.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:44:48.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:44:48.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:44:48.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:44:48.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:44:48.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:44:48.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:44:48.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:44:48.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:44:48.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:44:48.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:44:49.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:44:49.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:44:49.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:44:49.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2021-01-15T18:44:49.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:49.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2021-01-15T18:44:49.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:49.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2021-01-15T18:44:49.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:49.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2021-01-15T18:44:49.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:49.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2021-01-15T18:44:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:49.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2021-01-15T18:44:50.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2021-01-15T18:44:50.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2021-01-15T18:44:50.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2021-01-15T18:44:50.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2021-01-15T18:44:50.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2021-01-15T18:44:50.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:50.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2021-01-15T18:44:51.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:51.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2021-01-15T18:44:51.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:51.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2021-01-15T18:44:51.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:51.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2021-01-15T18:44:51.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:51.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2021-01-15T18:44:51.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:51.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2021-01-15T18:44:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2021-01-15T18:44:52.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:52.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2021-01-15T18:44:52.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:52.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2021-01-15T18:44:52.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:52.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2021-01-15T18:44:52.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:52.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2021-01-15T18:44:53.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:53.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2021-01-15T18:44:53.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2021-01-15T18:44:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:53.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2021-01-15T18:44:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2021-01-15T18:44:53.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:53.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2021-01-15T18:44:54.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:54.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2021-01-15T18:44:54.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:54.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2021-01-15T18:44:54.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:54.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2021-01-15T18:44:54.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:54.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2021-01-15T18:44:54.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2021-01-15T18:44:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2021-01-15T18:44:55.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2021-01-15T18:44:55.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2021-01-15T18:44:55.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2021-01-15T18:44:55.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:55.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2021-01-15T18:44:56.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:56.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2021-01-15T18:44:56.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:56.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2021-01-15T18:44:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:56.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2021-01-15T18:44:56.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:56.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2021-01-15T18:44:56.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:56.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2021-01-15T18:44:57.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2021-01-15T18:44:57.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2021-01-15T18:44:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2021-01-15T18:44:57.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2021-01-15T18:44:57.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2021-01-15T18:44:57.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:57.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2021-01-15T18:44:58.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2021-01-15T18:44:58.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2021-01-15T18:44:58.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2021-01-15T18:44:58.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2021-01-15T18:44:58.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2021-01-15T18:44:58.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:58.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2021-01-15T18:44:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:59.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2021-01-15T18:44:59.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:59.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2021-01-15T18:44:59.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:59.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2021-01-15T18:44:59.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:44:59.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2021-01-15T18:45:00.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2021-01-15T18:45:00.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2021-01-15T18:45:00.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2021-01-15T18:45:00.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2021-01-15T18:45:00.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2021-01-15T18:45:00.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:00.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2021-01-15T18:45:01.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:01.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2021-01-15T18:45:01.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:01.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2021-01-15T18:45:01.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:01.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2021-01-15T18:45:01.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:01.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2021-01-15T18:45:01.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:01.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2021-01-15T18:45:02.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2021-01-15T18:45:02.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2021-01-15T18:45:02.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2021-01-15T18:45:02.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2021-01-15T18:45:02.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2021-01-15T18:45:02.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:02.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2021-01-15T18:45:03.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:03.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2021-01-15T18:45:03.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:03.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2021-01-15T18:45:03.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:03.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2021-01-15T18:45:03.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:03.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2021-01-15T18:45:03.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:03.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2021-01-15T18:45:04.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:04.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2021-01-15T18:45:04.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:04.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2021-01-15T18:45:04.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:04.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2021-01-15T18:45:04.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:04.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2021-01-15T18:45:04.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:04.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2021-01-15T18:45:05.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:05.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2021-01-15T18:45:05.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:05.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2021-01-15T18:45:05.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:05.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2021-01-15T18:45:05.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:05.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2021-01-15T18:45:05.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:05.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2021-01-15T18:45:06.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2021-01-15T18:45:06.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2021-01-15T18:45:06.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2021-01-15T18:45:06.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2021-01-15T18:45:06.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2021-01-15T18:45:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:06.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2021-01-15T18:45:07.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:07.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2021-01-15T18:45:07.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:07.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2021-01-15T18:45:07.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:07.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2021-01-15T18:45:07.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:07.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2021-01-15T18:45:07.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:07.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2021-01-15T18:45:08.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:08.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2021-01-15T18:45:08.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:08.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2021-01-15T18:45:08.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:08.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2021-01-15T18:45:08.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:08.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2021-01-15T18:45:08.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:08.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2021-01-15T18:45:09.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:09.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2021-01-15T18:45:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:09.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2021-01-15T18:45:09.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:09.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2021-01-15T18:45:09.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:09.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2021-01-15T18:45:09.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:09.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2021-01-15T18:45:10.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:10.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2021-01-15T18:45:10.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:10.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2021-01-15T18:45:10.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:10.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2021-01-15T18:45:10.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:10.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2021-01-15T18:45:10.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:10.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2021-01-15T18:45:11.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:11.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2021-01-15T18:45:11.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:11.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2021-01-15T18:45:11.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:11.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2021-01-15T18:45:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2021-01-15T18:45:11.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:11.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2021-01-15T18:45:12.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:12.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2021-01-15T18:45:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:12.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2021-01-15T18:45:12.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:12.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2021-01-15T18:45:12.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:12.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2021-01-15T18:45:12.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:12.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2021-01-15T18:45:13.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2021-01-15T18:45:13.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2021-01-15T18:45:13.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2021-01-15T18:45:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2021-01-15T18:45:13.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2021-01-15T18:45:13.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:13.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2021-01-15T18:45:14.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:14.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2021-01-15T18:45:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:14.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2021-01-15T18:45:14.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:14.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2021-01-15T18:45:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:14.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2021-01-15T18:45:14.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:14.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2021-01-15T18:45:15.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2021-01-15T18:45:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2021-01-15T18:45:15.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:15.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2021-01-15T18:45:15.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:15.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2021-01-15T18:45:15.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:15.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2021-01-15T18:45:16.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2021-01-15T18:45:16.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2021-01-15T18:45:16.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2021-01-15T18:45:16.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2021-01-15T18:45:16.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2021-01-15T18:45:16.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:16.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2021-01-15T18:45:17.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:17.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2021-01-15T18:45:17.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:17.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2021-01-15T18:45:17.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:17.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2021-01-15T18:45:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2021-01-15T18:45:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2021-01-15T18:45:18.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2021-01-15T18:45:18.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2021-01-15T18:45:18.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2021-01-15T18:45:18.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2021-01-15T18:45:18.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2021-01-15T18:45:18.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:18.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2021-01-15T18:45:19.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:19.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2021-01-15T18:45:19.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:19.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2021-01-15T18:45:19.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:19.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2021-01-15T18:45:19.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:19.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2021-01-15T18:45:19.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:19.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2021-01-15T18:45:20.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:20.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2021-01-15T18:45:20.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:20.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2021-01-15T18:45:20.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:20.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2021-01-15T18:45:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:20.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2021-01-15T18:45:20.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:20.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2021-01-15T18:45:21.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:21.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2021-01-15T18:45:21.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:21.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2021-01-15T18:45:21.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:21.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2021-01-15T18:45:21.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:21.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2021-01-15T18:45:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:21.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2021-01-15T18:45:22.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2021-01-15T18:45:22.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2021-01-15T18:45:22.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2021-01-15T18:45:22.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2021-01-15T18:45:22.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2021-01-15T18:45:22.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:22.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2021-01-15T18:45:23.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:23.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2021-01-15T18:45:23.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:23.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2021-01-15T18:45:23.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:23.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2021-01-15T18:45:23.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2021-01-15T18:45:23.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:23.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2021-01-15T18:45:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:24.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2021-01-15T18:45:24.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:24.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2021-01-15T18:45:24.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:24.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2021-01-15T18:45:24.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:24.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2021-01-15T18:45:24.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:24.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2021-01-15T18:45:25.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:25.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2021-01-15T18:45:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2021-01-15T18:45:25.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:25.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2021-01-15T18:45:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:25.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2021-01-15T18:45:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2021-01-15T18:45:26.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2021-01-15T18:45:26.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2021-01-15T18:45:26.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2021-01-15T18:45:26.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2021-01-15T18:45:26.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:26.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2021-01-15T18:45:27.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:45:27.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2021-01-15T18:45:27.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2021-01-15T18:45:27.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:45:27.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:45:27.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:45:27.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:45:27.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2021-01-15T18:45:27.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2021-01-15T18:45:27.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:45:27.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:45:27.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:45:27.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:45:28.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2021-01-15T18:45:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2021-01-15T18:45:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:45:28.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:45:28.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:45:28.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:45:28.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:45:28.151 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:45:28.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:45:28.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:45:28.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:45:28.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:45:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2021-01-15T18:45:28.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2021-01-15T18:45:29.671 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:45:29.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2021-01-15T18:45:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 838752 2021-01-15T18:45:29.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2021-01-15T18:45:30.208 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:45:30.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2021-01-15T18:45:30.835 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:45:30.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2021-01-15T18:45:31.627 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:45:31.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2021-01-15T18:45:31.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:45:31.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:45:31.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:45:31.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:45:32.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T18:45:32.210 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:45:32.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2021-01-15T18:45:34.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2021-01-15T18:45:34.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:45:34.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:45:34.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:45:34.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:45:34.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:45:34.223 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:45:34.223 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:45:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:45:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:45:34.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:45:34.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:45:34.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:45:34.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:45:34.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:45:34.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:45:34.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:45:34.721 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:45:34.721 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:45:34.722 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:45:34.722 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:45:34.722 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:45:34.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:45:34.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:34.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:45:34.818 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T18:45:34.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T18:45:34.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T18:45:34.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T18:45:34.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:34.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:45:34.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=38654705686 2021-01-15T18:45:34.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 38654705686 2021-01-15T18:45:34.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-38654705686' 2021-01-15T18:45:34.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:34.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:45:35.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542164 2021-01-15T18:45:35.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542164 2021-01-15T18:45:35.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-38654705686 2-60129542164' 2021-01-15T18:45:35.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:35.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:45:35.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378644 2021-01-15T18:45:35.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378644 2021-01-15T18:45:35.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-38654705686 2-60129542164 3-81604378644' 2021-01-15T18:45:35.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:35.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:45:35.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=98784247826 2021-01-15T18:45:35.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 98784247826 2021-01-15T18:45:35.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-38654705686 2-60129542164 3-81604378644 4-98784247826' 2021-01-15T18:45:35.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:45:35.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:45:35.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=120259084304 2021-01-15T18:45:35.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 120259084304 2021-01-15T18:45:35.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-38654705686 2-60129542164 3-81604378644 4-98784247826 5-120259084304' 2021-01-15T18:45:35.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:45:35.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705686 2021-01-15T18:45:35.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:45:35.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:45:35.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-38654705686 2021-01-15T18:45:35.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:45:35.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=38654705686 2021-01-15T18:45:35.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 38654705686' 2021-01-15T18:45:35.514 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 38654705686 2021-01-15T18:45:35.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:45:35.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705684 -lt 38654705686 2021-01-15T18:45:35.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:45:36.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:45:36.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:45:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 38654705686 -lt 38654705686 2021-01-15T18:45:37.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:45:37.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542164 2021-01-15T18:45:37.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:45:37.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:45:37.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-60129542164 2021-01-15T18:45:37.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:45:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542164 2021-01-15T18:45:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 60129542164' 2021-01-15T18:45:37.316 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 60129542164 2021-01-15T18:45:37.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:45:37.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542165 -lt 60129542164 2021-01-15T18:45:37.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:45:37.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378644 2021-01-15T18:45:37.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:45:37.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:45:37.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-81604378644 2021-01-15T18:45:37.729 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:45:37.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378644 2021-01-15T18:45:37.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 81604378644' 2021-01-15T18:45:37.731 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 81604378644 2021-01-15T18:45:37.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:45:38.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378644 -lt 81604378644 2021-01-15T18:45:38.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:45:38.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247826 2021-01-15T18:45:38.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:45:38.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:45:38.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-98784247826 2021-01-15T18:45:38.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:45:38.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=98784247826 2021-01-15T18:45:38.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 98784247826' 2021-01-15T18:45:38.127 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 98784247826 2021-01-15T18:45:38.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:45:38.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 98784247826 -lt 98784247826 2021-01-15T18:45:38.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:45:38.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-120259084304 2021-01-15T18:45:38.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:45:38.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:45:38.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-120259084304 2021-01-15T18:45:38.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:45:38.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=120259084304 2021-01-15T18:45:38.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 120259084304' 2021-01-15T18:45:38.521 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 120259084304 2021-01-15T18:45:38.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:45:38.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 120259084305 -lt 120259084304 2021-01-15T18:45:38.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:45:38.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:38.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:39.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:45:39.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:39.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:39.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:39.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:39.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:39.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:39.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:39.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:39.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:39.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:39.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:40.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:40.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:45:40.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:45:40.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:45:40.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:40.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:40.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:40.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:40.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:40.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:40.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:40.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:40.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:40.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:40.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:41.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:41.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:41.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:41.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:41.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:41.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:41.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:41.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:41.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:41.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:45:41.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:45:41.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:41.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:41.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:41.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:41.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:41.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:41.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:41.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:41.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:41.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:42.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:42.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:42.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:42.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:42.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:42.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:42.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:42.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:42.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:42.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:42.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:42.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:42.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:43.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:43.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:43.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:43.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:43.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:43.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:43.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:43.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:43.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:43.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:43.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:43.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:43.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:43.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:43.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:43.891 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:44.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:44.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:44.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:44.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:44.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:44.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:44.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:44.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:44.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:44.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:45:44.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:45:44.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:44.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:44.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:45.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:45.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:45.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:45.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:45.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:45.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:45.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:45.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:45.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:45.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:45.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:45.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:45.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:45.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:45.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:45.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:45.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:45.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:45.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:45.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:45.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:46.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:45:46.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:45:46.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:46.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:46.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:46.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:46.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:46.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:46.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:46.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:46.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:46.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:46.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:46.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:46.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:46.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:46.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:47.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:47.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:47.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:47.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:47.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:47.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:47.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:47.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:47.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:47.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:45:47.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:45:47.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:47.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:47.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:48.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:48.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:48.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:48.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:48.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:48.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:48.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:48.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:48.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:48.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:48.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:48.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:48.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:48.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:48.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:48.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:48.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:48.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:48.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:49.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:45:49.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:45:49.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:49.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:49.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:49.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:49.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:49.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:49.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:49.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:49.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:49.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:49.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:49.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:49.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:49.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:49.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:50.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:50.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:50.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:50.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:50.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:50.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:50.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:50.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:50.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:50.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:50.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:50.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:50.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:50.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:51.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:51.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:51.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:51.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:51.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:51.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:51.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:51.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:51.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:51.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:51.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:51.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:51.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:51.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:51.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:51.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:51.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:51.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:52.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:52.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:52.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:52.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:52.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:52.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:52.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:52.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:52.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:52.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:52.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:52.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:52.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:52.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:52.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:52.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:52.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:53.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:53.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:53.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:53.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:53.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:54.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2021-01-15T18:45:54.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2021-01-15T18:45:54.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:54.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:54.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:54.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:54.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:54.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:54.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:54.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:54.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:54.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:54.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:54.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:54.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:54.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:55.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:55.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:55.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:55.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:55.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:55.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:55.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:55.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:55.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:55.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:55.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:55.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:55.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:55.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:55.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:55.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:55.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:55.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:55.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:55.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:56.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:56.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:56.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:56.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:56.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:56.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:56.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:56.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:57.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:57.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:57.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:57.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:57.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:57.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:57.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:57.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:57.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:57.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:57.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:57.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:57.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:57.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:58.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:58.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:58.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:58.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:58.098 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:58.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:45:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:45:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:45:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:45:58.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:45:58.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:45:58.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:45:58.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:45:58.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:45:58.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:45:58.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:45:58.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:45:58.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:45:58.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:45:59.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:45:59.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:45:59.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:45:59.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:45:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:45:59.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:45:59.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:45:59.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:45:59.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:45:59.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:45:59.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:45:59.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:45:59.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=7 2021-01-15T18:46:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 7 '!=' null 2021-01-15T18:46:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:00.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:00.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:00.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:00.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:00.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:00.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:00.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:00.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:00.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:00.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:00.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:00.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:01.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:01.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:01.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:01.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:01.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:01.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:01.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:01.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:01.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:01.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:46:01.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:46:01.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:01.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:01.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:01.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:01.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:01.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:01.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:01.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:01.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:02.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:02.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:02.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:02.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:02.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:02.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:02.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:02.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:02.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:02.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:02.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:02.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:02.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:03.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:46:03.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:46:03.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:03.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:03.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:03.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:03.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:03.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:03.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:03.296 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:03.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:03.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:03.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:03.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:03.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:04.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:04.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:04.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:04.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:04.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:04.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:04.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:46:04.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:46:04.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:04.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:04.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:04.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:04.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:04.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:04.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:04.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:04.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:04.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:04.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:05.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:05.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:05.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:05.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:05.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:05.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:05.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:05.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:05.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:05.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:06.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:46:06.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:46:06.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:06.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:06.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:06.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:06.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:06.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:06.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:06.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:06.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:06.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:06.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:06.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:06.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:06.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:06.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:07.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:07.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:07.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:07.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:07.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:46:07.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:46:07.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:07.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:07.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:07.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:08.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:08.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:08.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:08.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:08.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:08.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:08.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:08.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:46:08.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:08.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:08.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:08.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:46:08.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:46:08.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:46:08.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:46:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:46:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:46:08.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:46:08.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:46:08.879 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:46:09.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:46:09.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:46:09.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:46:09.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:46:09.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:46:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:46:09.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:09.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:09.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:09.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:09.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:09.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:09.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:09.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:46:09.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:09.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:10.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:46:10.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:46:10.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:46:10.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2021-01-15T18:46:10.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:46:10.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:46:10.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-01-15T18:46:10.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2021-01-15T18:46:10.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-01-15T18:46:10.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:46:10.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2021-01-15T18:46:10.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:46:10.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-01-15T18:46:10.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-01-15T18:46:10.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:46:10.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2021-01-15T18:46:10.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2021-01-15T18:46:10.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:46:10.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:46:10.387 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:46:10.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2021-01-15T18:46:10.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:46:10.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:46:10.402 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-01-15T18:46:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:46:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:46:10.403 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:46:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-01-15T18:46:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-01-15T18:46:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:46:10.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:46:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:46:10.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:46:10.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:46:10.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:46:10.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:46:10.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:46:10.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:46:10.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:46:10.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:46:10.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:46:10.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:46:10.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:46:10.424 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:46:10.425 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.425 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2021-01-15T18:46:10.425 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:46:10.426 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:46:10.426 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:46:10.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2021-01-15T18:46:10.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2021-01-15T18:46:10.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-01-15T18:46:10.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:46:10.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-01-15T18:46:10.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:46:10.432 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:46:10.432 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:46:10.433 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:46:10.433 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:46:10.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:46:10.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:46:10.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:46:10.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:46:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:46:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:46:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-01-15T18:46:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2021-01-15T18:46:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:46:10.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:46:11.040 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:46:11.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2021-01-15T18:46:11.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:46:11.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:46:11.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:46:11.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:46:11.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:46:11.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:46:11.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:46:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:46:11.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:46:11.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:46:11.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:46:11.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:46:11.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:46:11.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:46:11.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:46:11.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:46:11.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:46:11.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:46:11.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:46:11.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:46:11.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:46:11.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:46:11.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:46:11.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:46:11.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:46:11.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:46:11.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:46:11.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:46:11.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:46:11.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:46:11.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:46:11.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:46:11.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:46:11.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:46:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:46:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:46:11.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:46:11.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:46:11.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:46:11.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:46:11.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:46:11.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:46:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:46:11.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:46:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:46:11.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:46:11.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:46:11.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:46:11.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:46:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:46:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:46:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:46:11.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:46:11.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:46:11.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:46:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:46:11.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:46:11.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:46:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:46:11.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:46:11.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:46:11.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:46:11.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:46:11.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:46:11.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:46:11.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:46:11.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:46:11.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2021-01-15T18:46:11.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2021-01-15T18:46:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:46:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:46:11.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:46:11.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:46:11.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:46:11.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:46:11.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:11.533 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:11.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:11.534 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.534 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:11.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:46:11.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:46:11.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:46:11.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:46:11.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:46:11.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:46:11.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:46:11.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:46:11.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:46:11.711 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:46:11.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:46:11.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.712 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.712 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:46:11.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:46:11.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:46:11.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:46:11.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:46:11.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:46:11.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:46:11.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:46:11.813 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:46:11.813 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:46:11.813 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:46:11.813 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:46:11.814 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:11.814 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:11.814 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:46:11.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:46:11.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:46:11.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2021-01-15T18:46:11.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:46:11.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:46:11.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:46:11.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:46:11.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:46:11.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:46:12.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:46:12.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:12.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:12.828 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:12.828 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:12.828 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:12.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:12.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:46:12.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:46:12.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2021-01-15T18:46:12.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:46:12.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:46:12.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:46:12.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:46:12.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:46:12.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:46:12.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:46:12.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:46:12.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:46:12.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:46:12.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:46:12.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:46:12.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:46:12.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:46:12.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:46:12.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:12.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:12.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:12.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:12.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:12.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:46:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:46:12.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:46:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:46:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:46:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:46:12.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:46:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:46:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:46:12.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:46:12.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:46:12.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d98886db-2880-45c1-8adb-44381bb0ad49 2021-01-15T18:46:12.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d98886db-2880-45c1-8adb-44381bb0ad49' 2021-01-15T18:46:12.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:46:12.861 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d98886db-2880-45c1-8adb-44381bb0ad49 2021-01-15T18:46:12.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD04gFgS26tMxAASxtd4buvELFFt3t1n0UoAw== 2021-01-15T18:46:12.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD04gFgS26tMxAASxtd4buvELFFt3t1n0UoAw=="}' 2021-01-15T18:46:12.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d98886db-2880-45c1-8adb-44381bb0ad49 -i td/osd-recovery-stats/0/new.json 2021-01-15T18:46:13.235 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:46:13.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:46:13.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD04gFgS26tMxAASxtd4buvELFFt3t1n0UoAw== --osd-uuid d98886db-2880-45c1-8adb-44381bb0ad49 2021-01-15T18:46:13.269 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.269+0000 7ffa71da6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:13.271 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.270+0000 7ffa71da6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:13.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.272+0000 7ffa71da6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:13.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.273+0000 7ffa71da6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:46:13.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.273+0000 7ffa71da6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:46:13.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.273+0000 7ffa71da6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:46:13.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:13.315+0000 7ffa71da6080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:46:17.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:46:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:46:17.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:46:17.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:46:17.879 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:46:18.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:46:18.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:46:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:46:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:46:18.402 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:46:18.422 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:18.420+0000 7febb7dd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:18.424 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:18.422+0000 7febb7dd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:18.426 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:18.424+0000 7febb7dd1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:46:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:46:18.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:46:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:46:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:46:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:46:18.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:18.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:46:18.798 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:46:18.949 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:18.948+0000 7febb7dd1080 -1 Falling back to public interface 2021-01-15T18:46:19.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:46:19.800 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:19.799+0000 7febb7dd1080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:46:20.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:46:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:46:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:20.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:46:20.219 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:46:20.617 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/853656,v1:127.0.0.1:6803/853656] [v2:127.0.0.1:6804/853656,v1:127.0.0.1:6805/853656] exists,up d98886db-2880-45c1-8adb-44381bb0ad49 2021-01-15T18:46:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:46:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:46:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:46:20.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2021-01-15T18:46:20.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:46:20.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:46:20.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:46:20.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:46:20.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:46:20.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:46:20.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:46:20.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:46:20.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:46:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:46:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:46:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:46:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:46:20.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:46:20.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:46:20.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:20.622 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:20.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:20.622 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:20.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:46:20.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:46:20.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:46:20.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:46:20.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:46:20.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:46:20.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:46:20.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=5ec55ef0-9f4c-434c-bc97-f6cedb040753 2021-01-15T18:46:20.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 5ec55ef0-9f4c-434c-bc97-f6cedb040753' 2021-01-15T18:46:20.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:46:20.627 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 5ec55ef0-9f4c-434c-bc97-f6cedb040753 2021-01-15T18:46:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQD84gFgoHqiJRAAKUWoCeW15YWJNkdrWzD+FA== 2021-01-15T18:46:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQD84gFgoHqiJRAAKUWoCeW15YWJNkdrWzD+FA=="}' 2021-01-15T18:46:20.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 5ec55ef0-9f4c-434c-bc97-f6cedb040753 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:46:21.163 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:46:21.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:46:21.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD84gFgoHqiJRAAKUWoCeW15YWJNkdrWzD+FA== --osd-uuid 5ec55ef0-9f4c-434c-bc97-f6cedb040753 2021-01-15T18:46:21.198 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.197+0000 7f6786f43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:21.199 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.199+0000 7f6786f43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:21.200 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.200+0000 7f6786f43080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:21.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.200+0000 7f6786f43080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:46:21.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.200+0000 7f6786f43080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:46:21.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.200+0000 7f6786f43080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:46:21.246 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:21.245+0000 7f6786f43080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:46:25.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:46:25.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:46:25.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:46:25.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:46:25.830 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:46:26.356 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:46:26.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:46:26.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:46:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:46:26.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:46:26.377 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:26.376+0000 7fd853100080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:26.428 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:26.427+0000 7fd853100080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:26.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:26.428+0000 7fd853100080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:26.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:46:26.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:46:26.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:46:26.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:46:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:46:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:46:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:46:26.750 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:46:26.945 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:26.944+0000 7fd853100080 -1 Falling back to public interface 2021-01-15T18:46:27.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:46:27.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:27.729+0000 7fd853100080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:46:28.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:46:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:46:28.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:46:28.142 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:46:28.538 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/854920,v1:127.0.0.1:6811/854920] [v2:127.0.0.1:6812/854920,v1:127.0.0.1:6813/854920] exists,up 5ec55ef0-9f4c-434c-bc97-f6cedb040753 2021-01-15T18:46:28.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:46:28.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:46:28.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:46:28.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2021-01-15T18:46:28.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:46:28.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:46:28.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:46:28.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:46:28.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:46:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:46:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:46:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:46:28.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:46:28.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:46:28.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:46:28.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:46:28.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:46:28.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:46:28.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:46:28.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:28.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:28.543 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:28.543 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:28.544 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:28.544 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:28.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:28.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:46:28.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:46:28.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:46:28.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:46:28.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:46:28.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:46:28.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:46:28.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:28.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:46:28.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:46:28.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:46:28.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:46:28.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4c387e21-2687-4bda-8530-0bc2377ca6ac 2021-01-15T18:46:28.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 4c387e21-2687-4bda-8530-0bc2377ca6ac' 2021-01-15T18:46:28.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:46:28.548 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 4c387e21-2687-4bda-8530-0bc2377ca6ac 2021-01-15T18:46:28.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAE4wFg6hDyIBAA60nHn7S/qTvCuKFUsEtjNQ== 2021-01-15T18:46:28.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAE4wFg6hDyIBAA60nHn7S/qTvCuKFUsEtjNQ=="}' 2021-01-15T18:46:28.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4c387e21-2687-4bda-8530-0bc2377ca6ac -i td/osd-recovery-stats/2/new.json 2021-01-15T18:46:29.195 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:46:29.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:46:29.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAE4wFg6hDyIBAA60nHn7S/qTvCuKFUsEtjNQ== --osd-uuid 4c387e21-2687-4bda-8530-0bc2377ca6ac 2021-01-15T18:46:29.231 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.230+0000 7fb8c6abd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:29.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.231+0000 7fb8c6abd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:29.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.233+0000 7fb8c6abd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:29.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.233+0000 7fb8c6abd080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:46:29.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.233+0000 7fb8c6abd080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:46:29.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.233+0000 7fb8c6abd080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:46:29.287 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:29.286+0000 7fb8c6abd080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:46:34.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:46:34.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:46:34.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:46:34.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:46:34.111 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:46:34.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:46:34.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:34.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:46:34.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:46:34.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:46:34.711 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:46:34.712 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:34.661+0000 7fc898fa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:34.712 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:34.683+0000 7fc898fa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:34.712 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:34.684+0000 7fc898fa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:35.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:46:35.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:46:35.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:46:35.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:46:35.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:46:35.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:35.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:46:35.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:35.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:46:35.124 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:46:35.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:35.200+0000 7fc898fa9080 -1 Falling back to public interface 2021-01-15T18:46:35.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:46:36.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:36.009+0000 7fc898fa9080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:46:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:46:36.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:36.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:46:36.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:36.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:46:36.450 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:46:36.915 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/856192,v1:127.0.0.1:6819/856192] [v2:127.0.0.1:6820/856192,v1:127.0.0.1:6821/856192] exists,up 4c387e21-2687-4bda-8530-0bc2377ca6ac 2021-01-15T18:46:36.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:46:36.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:46:36.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:46:36.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2021-01-15T18:46:36.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:46:36.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:46:36.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:46:36.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:46:36.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:46:36.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:46:36.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:46:36.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:46:36.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:46:36.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:46:36.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:46:36.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:46:36.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:46:36.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:46:36.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:46:36.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:46:36.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:46:36.928 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:46:36.929 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:46:36.929 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:46:36.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:36.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:46:36.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:46:36.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:46:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:46:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:46:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:46:36.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:46:36.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:46:36.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:46:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:46:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:46:36.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:46:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e1a0336d-065e-4967-9d4f-00504eeb64a6 2021-01-15T18:46:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 e1a0336d-065e-4967-9d4f-00504eeb64a6' 2021-01-15T18:46:36.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:46:36.935 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 e1a0336d-065e-4967-9d4f-00504eeb64a6 2021-01-15T18:46:36.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAM4wFgWoiHNxAATe0/qHWVEqjfSIModY/QOQ== 2021-01-15T18:46:36.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAM4wFgWoiHNxAATe0/qHWVEqjfSIModY/QOQ=="}' 2021-01-15T18:46:36.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e1a0336d-065e-4967-9d4f-00504eeb64a6 -i td/osd-recovery-stats/3/new.json 2021-01-15T18:46:37.429 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:46:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:46:37.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAM4wFgWoiHNxAATe0/qHWVEqjfSIModY/QOQ== --osd-uuid e1a0336d-065e-4967-9d4f-00504eeb64a6 2021-01-15T18:46:37.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.464+0000 7ff729369080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:37.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.465+0000 7ff729369080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:37.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.467+0000 7ff729369080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:37.468 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.467+0000 7ff729369080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:46:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.467+0000 7ff729369080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:46:37.469 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.467+0000 7ff729369080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:46:37.514 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:37.513+0000 7ff729369080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:46:42.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:46:42.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:46:42.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:46:42.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:46:42.036 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:46:42.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:46:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:46:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:46:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:46:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:46:42.557 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:46:42.576 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:42.575+0000 7f7a6fea9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:42.578 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:42.577+0000 7f7a6fea9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:42.579 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:42.578+0000 7f7a6fea9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:46:42.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:46:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:46:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:46:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:46:42.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:46:42.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:42.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:46:42.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:42.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:46:42.956 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:46:43.100 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:43.099+0000 7f7a6fea9080 -1 Falling back to public interface 2021-01-15T18:46:43.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:46:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:46:43.917+0000 7f7a6fea9080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:46:44.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:46:44.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:46:44.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:46:44.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:46:44.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:46:44.364 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:46:44.759 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/857456,v1:127.0.0.1:6827/857456] [v2:127.0.0.1:6828/857456,v1:127.0.0.1:6829/857456] exists,up e1a0336d-065e-4967-9d4f-00504eeb64a6 2021-01-15T18:46:44.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:46:44.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:46:44.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:46:44.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2021-01-15T18:46:45.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2021-01-15T18:46:45.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 erasure myprofile 2021-01-15T18:46:46.167 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:46:46.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:46:47.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2021-01-15T18:46:47.962 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 2 2021-01-15T18:46:47.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2021-01-15T18:46:47.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:46:47.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:46:47.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:46:47.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:46:47.973 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:46:47.973 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:46:47.974 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:46:47.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:46:47.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:46:47.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:46:48.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:46:48.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:46:48.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:46:48.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:46:48.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:46:48.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:46:48.486 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:46:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:46:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:46:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:46:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:46:48.487 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:46:48.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836488 2021-01-15T18:46:48.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836488 2021-01-15T18:46:48.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488' 2021-01-15T18:46:48.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:46:48.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:46:48.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672967 2021-01-15T18:46:48.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672967 2021-01-15T18:46:48.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2021-01-15T18:46:48.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:46:48.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:46:48.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509445 2021-01-15T18:46:48.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509445 2021-01-15T18:46:48.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509445' 2021-01-15T18:46:48.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:46:48.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:46:49.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345923 2021-01-15T18:46:49.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345923 2021-01-15T18:46:49.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509445 3-85899345923' 2021-01-15T18:46:49.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:46:49.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836488 2021-01-15T18:46:49.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:46:49.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:46:49.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836488 2021-01-15T18:46:49.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:46:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836488 2021-01-15T18:46:49.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-01-15T18:46:49.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:46:49.062 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836488 2021-01-15T18:46:49.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836488 2021-01-15T18:46:49.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:46:50.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:46:50.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:46:50.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836488 -lt 21474836488 2021-01-15T18:46:50.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:46:50.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672967 2021-01-15T18:46:50.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:46:50.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:46:50.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672967 2021-01-15T18:46:50.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:46:50.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672967 2021-01-15T18:46:50.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2021-01-15T18:46:50.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:46:50.848 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672967 2021-01-15T18:46:51.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672967 2021-01-15T18:46:51.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:46:51.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509445 2021-01-15T18:46:51.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:46:51.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:46:51.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509445 2021-01-15T18:46:51.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:46:51.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509445 2021-01-15T18:46:51.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2021-01-15T18:46:51.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:46:51.242 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509445 2021-01-15T18:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509445 -lt 64424509445 2021-01-15T18:46:51.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:46:51.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345923 2021-01-15T18:46:51.655 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:46:51.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:46:51.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345923 2021-01-15T18:46:51.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:46:51.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345923 2021-01-15T18:46:51.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345923' 2021-01-15T18:46:51.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:46:51.657 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345923 2021-01-15T18:46:52.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345924 -lt 85899345923 2021-01-15T18:46:52.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:46:52.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:52.061 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:52.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:46:52.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:46:52.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:46:52.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:46:52.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:46:52.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:46:52.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:46:52.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:46:52.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:46:52.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:46:52.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:46:52.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:46:53.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:46:53.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:46:53.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:46:53.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2021-01-15T18:46:53.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:53.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2021-01-15T18:46:53.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:53.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2021-01-15T18:46:53.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:53.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2021-01-15T18:46:54.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:54.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2021-01-15T18:46:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2021-01-15T18:46:54.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:54.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2021-01-15T18:46:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:54.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2021-01-15T18:46:54.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:54.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2021-01-15T18:46:55.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:55.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2021-01-15T18:46:55.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:55.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2021-01-15T18:46:55.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:55.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2021-01-15T18:46:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:55.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2021-01-15T18:46:55.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:55.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2021-01-15T18:46:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:56.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2021-01-15T18:46:56.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:56.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2021-01-15T18:46:56.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:56.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2021-01-15T18:46:56.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:56.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2021-01-15T18:46:56.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:56.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2021-01-15T18:46:57.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:57.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2021-01-15T18:46:57.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:57.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2021-01-15T18:46:57.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:57.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2021-01-15T18:46:57.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:57.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2021-01-15T18:46:57.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:57.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2021-01-15T18:46:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2021-01-15T18:46:58.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:58.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2021-01-15T18:46:58.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:58.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2021-01-15T18:46:58.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:58.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2021-01-15T18:46:58.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:58.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2021-01-15T18:46:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:59.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2021-01-15T18:46:59.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:59.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2021-01-15T18:46:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2021-01-15T18:46:59.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:59.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2021-01-15T18:46:59.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:46:59.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2021-01-15T18:47:00.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2021-01-15T18:47:00.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:00.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2021-01-15T18:47:00.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:00.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2021-01-15T18:47:00.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:00.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2021-01-15T18:47:00.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:00.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2021-01-15T18:47:01.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:01.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2021-01-15T18:47:01.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:01.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2021-01-15T18:47:01.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:01.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2021-01-15T18:47:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:01.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2021-01-15T18:47:01.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:01.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2021-01-15T18:47:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:02.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2021-01-15T18:47:02.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:02.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2021-01-15T18:47:02.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:02.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2021-01-15T18:47:02.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:02.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2021-01-15T18:47:02.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:02.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2021-01-15T18:47:03.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:03.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2021-01-15T18:47:03.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:03.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2021-01-15T18:47:03.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:03.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2021-01-15T18:47:03.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:03.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2021-01-15T18:47:03.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:03.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2021-01-15T18:47:04.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:04.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2021-01-15T18:47:04.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:04.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2021-01-15T18:47:04.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:04.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2021-01-15T18:47:04.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:04.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2021-01-15T18:47:04.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:04.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2021-01-15T18:47:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2021-01-15T18:47:05.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:05.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2021-01-15T18:47:05.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:05.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2021-01-15T18:47:05.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:05.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2021-01-15T18:47:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:05.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2021-01-15T18:47:06.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:06.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2021-01-15T18:47:06.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:06.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2021-01-15T18:47:06.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:06.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2021-01-15T18:47:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:06.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2021-01-15T18:47:06.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:06.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2021-01-15T18:47:07.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:07.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2021-01-15T18:47:07.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:07.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2021-01-15T18:47:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:07.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2021-01-15T18:47:07.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:07.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2021-01-15T18:47:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:07.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2021-01-15T18:47:08.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:08.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2021-01-15T18:47:08.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:08.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2021-01-15T18:47:08.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:08.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2021-01-15T18:47:08.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:08.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2021-01-15T18:47:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:08.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2021-01-15T18:47:09.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:09.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2021-01-15T18:47:09.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:09.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2021-01-15T18:47:09.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:09.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2021-01-15T18:47:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:09.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2021-01-15T18:47:09.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:09.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2021-01-15T18:47:10.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:10.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2021-01-15T18:47:10.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:10.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2021-01-15T18:47:10.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:10.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2021-01-15T18:47:10.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:10.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2021-01-15T18:47:10.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:10.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2021-01-15T18:47:11.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:11.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2021-01-15T18:47:11.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:11.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2021-01-15T18:47:11.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:11.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2021-01-15T18:47:11.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:11.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2021-01-15T18:47:11.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:11.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2021-01-15T18:47:12.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:12.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2021-01-15T18:47:12.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:12.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2021-01-15T18:47:12.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:12.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2021-01-15T18:47:12.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:12.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2021-01-15T18:47:13.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:13.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2021-01-15T18:47:13.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:13.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2021-01-15T18:47:13.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:13.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2021-01-15T18:47:13.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:13.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2021-01-15T18:47:13.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:13.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2021-01-15T18:47:14.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:14.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2021-01-15T18:47:14.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:14.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2021-01-15T18:47:14.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2021-01-15T18:47:14.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:14.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2021-01-15T18:47:14.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:14.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2021-01-15T18:47:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2021-01-15T18:47:15.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:15.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2021-01-15T18:47:15.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:15.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2021-01-15T18:47:15.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:15.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2021-01-15T18:47:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:15.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2021-01-15T18:47:16.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:16.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2021-01-15T18:47:16.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2021-01-15T18:47:16.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:16.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2021-01-15T18:47:16.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:16.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2021-01-15T18:47:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2021-01-15T18:47:17.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:17.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2021-01-15T18:47:17.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:17.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2021-01-15T18:47:17.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:17.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2021-01-15T18:47:17.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:17.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2021-01-15T18:47:17.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:17.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2021-01-15T18:47:18.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:18.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2021-01-15T18:47:18.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:18.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2021-01-15T18:47:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:18.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2021-01-15T18:47:18.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:18.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2021-01-15T18:47:18.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:18.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2021-01-15T18:47:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:19.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2021-01-15T18:47:19.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:19.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2021-01-15T18:47:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:19.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2021-01-15T18:47:19.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:19.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2021-01-15T18:47:19.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:19.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2021-01-15T18:47:20.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:20.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2021-01-15T18:47:20.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:20.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2021-01-15T18:47:20.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:20.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2021-01-15T18:47:20.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:20.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2021-01-15T18:47:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:20.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2021-01-15T18:47:21.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:21.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2021-01-15T18:47:21.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:21.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2021-01-15T18:47:21.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:21.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2021-01-15T18:47:21.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:21.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2021-01-15T18:47:21.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:21.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2021-01-15T18:47:22.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:22.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2021-01-15T18:47:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2021-01-15T18:47:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2021-01-15T18:47:22.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:22.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2021-01-15T18:47:22.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:22.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2021-01-15T18:47:23.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:23.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2021-01-15T18:47:23.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:23.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2021-01-15T18:47:23.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:23.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2021-01-15T18:47:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2021-01-15T18:47:23.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:23.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2021-01-15T18:47:24.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:24.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2021-01-15T18:47:24.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:24.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2021-01-15T18:47:24.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:24.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2021-01-15T18:47:24.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:24.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2021-01-15T18:47:24.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:24.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2021-01-15T18:47:25.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:25.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2021-01-15T18:47:25.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:25.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2021-01-15T18:47:25.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:25.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2021-01-15T18:47:25.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:25.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2021-01-15T18:47:25.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:25.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2021-01-15T18:47:26.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:26.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2021-01-15T18:47:26.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:26.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2021-01-15T18:47:26.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:26.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2021-01-15T18:47:26.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:26.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2021-01-15T18:47:26.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:26.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2021-01-15T18:47:27.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:27.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2021-01-15T18:47:27.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:27.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2021-01-15T18:47:27.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:27.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2021-01-15T18:47:27.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:27.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2021-01-15T18:47:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:27.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2021-01-15T18:47:28.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:28.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2021-01-15T18:47:28.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:28.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2021-01-15T18:47:28.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:28.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2021-01-15T18:47:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:28.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2021-01-15T18:47:28.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:28.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2021-01-15T18:47:29.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:29.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2021-01-15T18:47:29.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:29.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2021-01-15T18:47:29.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:29.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2021-01-15T18:47:29.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:29.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2021-01-15T18:47:29.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:29.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2021-01-15T18:47:30.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2021-01-15T18:47:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:30.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2021-01-15T18:47:30.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:30.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2021-01-15T18:47:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:30.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2021-01-15T18:47:30.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:30.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2021-01-15T18:47:31.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:31.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2021-01-15T18:47:31.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:31.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2021-01-15T18:47:31.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:31.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2021-01-15T18:47:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:31.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2021-01-15T18:47:31.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:31.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2021-01-15T18:47:32.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:32.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2021-01-15T18:47:32.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:32.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2021-01-15T18:47:32.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:32.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2021-01-15T18:47:32.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:32.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2021-01-15T18:47:32.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:32.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2021-01-15T18:47:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2021-01-15T18:47:33.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:33.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2021-01-15T18:47:33.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2021-01-15T18:47:33.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2021-01-15T18:47:33.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2021-01-15T18:47:33.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:47:33.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:47:33.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:47:33.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:47:34.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2021-01-15T18:47:34.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2021-01-15T18:47:34.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:47:34.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:47:34.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:47:34.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:47:34.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2021-01-15T18:47:34.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2021-01-15T18:47:34.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:47:34.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:47:34.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:47:34.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:47:34.570 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:47:34.571 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:47:34.571 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:47:34.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:47:34.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:47:34.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:47:35.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2021-01-15T18:47:35.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2021-01-15T18:47:36.132 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:47:36.142 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2021-01-15T18:47:36.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 853656 2021-01-15T18:47:36.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2021-01-15T18:47:36.626 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:47:36.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2021-01-15T18:47:37.235 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:47:37.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-01-15T18:47:37.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2021-01-15T18:47:37.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2021-01-15T18:47:37.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2021-01-15T18:47:37.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-01-15T18:47:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2021-01-15T18:47:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2021-01-15T18:47:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2021-01-15T18:47:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2021-01-15T18:47:37.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2021-01-15T18:47:37.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2021-01-15T18:47:37.796 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T18:47:37.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2021-01-15T18:47:37.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2021-01-15T18:47:38.561 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:47:38.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2021-01-15T18:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:47:38.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:47:39.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T18:47:39.171 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:47:39.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2021-01-15T18:47:41.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2021-01-15T18:47:41.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:47:41.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:47:41.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:47:41.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:47:41.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:47:41.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:47:41.184 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:47:41.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:47:41.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:47:41.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:47:41.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:47:41.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:47:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:47:41.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:47:41.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:47:41.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:47:41.677 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:47:41.678 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:47:41.678 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:47:41.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:47:41.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:47:41.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:47:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T18:47:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T18:47:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T18:47:41.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T18:47:41.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:47:41.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:47:41.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672980 2021-01-15T18:47:41.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672980 2021-01-15T18:47:41.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980' 2021-01-15T18:47:41.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:47:41.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:47:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T18:47:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T18:47:42.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457' 2021-01-15T18:47:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:47:42.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:47:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2021-01-15T18:47:42.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2021-01-15T18:47:42.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672980 2-64424509457 3-85899345936' 2021-01-15T18:47:42.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:47:42.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672980 2021-01-15T18:47:42.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:47:42.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:47:42.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672980 2021-01-15T18:47:42.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:47:42.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672980 2021-01-15T18:47:42.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-01-15T18:47:42.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:47:42.222 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672980 2021-01-15T18:47:42.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672978 -lt 42949672980 2021-01-15T18:47:42.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:47:43.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:47:43.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:47:44.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672980 2021-01-15T18:47:44.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:47:44.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T18:47:44.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:47:44.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:47:44.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T18:47:44.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:47:44.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T18:47:44.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T18:47:44.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:47:44.053 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T18:47:44.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509458 -lt 64424509457 2021-01-15T18:47:44.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:47:44.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2021-01-15T18:47:44.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:47:44.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:47:44.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2021-01-15T18:47:44.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:47:44.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2021-01-15T18:47:44.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-01-15T18:47:44.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:47:44.463 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345936 2021-01-15T18:47:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345936 2021-01-15T18:47:44.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:47:44.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:44.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:47:45.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:45.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:45.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:45.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:45.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:45.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:45.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:45.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:45.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:46.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:46.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:47:46.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:47:46.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:47:46.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:46.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:46.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:46.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:46.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:46.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:46.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:46.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:46.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:46.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:46.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:46.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:46.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:46.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:47.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:47.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:47:47.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:47:47.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:47.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:47.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:47.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:47.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:47.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:47.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:47.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:47.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:47.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:47.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:48.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:48.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:48.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:48.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:48.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:48.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:48.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:48.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:48.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:47:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:47:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:49.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:49.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:49.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:49.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:49.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:49.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:49.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:49.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:49.469 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:49.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:49.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:49.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:49.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:50.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:50.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:50.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:50.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:50.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:50.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:47:50.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:47:50.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:50.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:50.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:51.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:51.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:51.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:51.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:51.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:51.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:51.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:51.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:51.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:51.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:51.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:51.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:51.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:52.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:52.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:52.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:47:52.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:47:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:52.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:52.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:52.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:52.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:52.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:52.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:52.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:52.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:52.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:52.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:52.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:52.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:52.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:53.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:53.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:53.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:53.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:53.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:53.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:53.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:53.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:47:53.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:47:53.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:53.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:54.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:54.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:54.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:54.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:54.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:54.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:54.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:54.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:54.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:54.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:54.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:54.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:54.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:54.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:54.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:54.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:54.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:54.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:54.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:54.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:54.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:55.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:47:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:47:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:55.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:55.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:55.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:55.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:55.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:55.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:55.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:55.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:55.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:55.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:55.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:55.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:55.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:55.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:56.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:56.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:56.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:56.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:56.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:56.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:56.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:56.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:56.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:57.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:47:57.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:47:57.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:57.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:57.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:57.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:57.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:57.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:57.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:57.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:57.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:57.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:57.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:57.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:57.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:57.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:57.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:58.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:58.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:58.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:58.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:58.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:58.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:58.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:58.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:58.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:47:58.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:47:58.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:47:58.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:47:58.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:47:58.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:47:58.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:47:58.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:47:58.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:47:58.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:47:58.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:47:58.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:47:58.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:47:58.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:47:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:47:59.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:47:59.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:47:59.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:47:59.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:47:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:47:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:47:59.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:47:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:47:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:47:59.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:47:59.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:47:59.553 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:00.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:00.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:00.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:00.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:00.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:00.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:00.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:00.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:00.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:00.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:00.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:00.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:00.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:00.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:00.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:00.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:00.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:01.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:01.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:01.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:01.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:01.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:01.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:01.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:48:01.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:48:01.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:01.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:01.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:01.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:01.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:01.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:01.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:01.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:01.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:01.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:01.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:02.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:02.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:02.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:02.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:02.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:02.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:02.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:02.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:02.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:02.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:02.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:02.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:02.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:03.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:03.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:03.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:03.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:03.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:03.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:03.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:03.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:03.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:03.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:03.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:03.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:03.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:03.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:04.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:04.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:04.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:04.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:04.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:04.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:04.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:04.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:04.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:04.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:04.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:04.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:04.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:04.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:04.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:04.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:04.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:04.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:04.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:05.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:05.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:05.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:05.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:05.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:05.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:05.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:05.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:05.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:06.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:48:06.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:48:06.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:06.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:06.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:06.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:06.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:06.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:06.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:06.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:06.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:06.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:07.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:07.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:07.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:07.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:07.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:07.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:07.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:07.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:07.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:07.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:07.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:07.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:07.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:07.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:07.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:07.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:07.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:07.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:07.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:07.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:07.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:07.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:08.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:08.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:08.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:08.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:08.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:08.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:08.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:08.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:09.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:09.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:09.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:09.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:09.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:09.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:09.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:09.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:09.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:09.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:10.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:10.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:10.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:10.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:10.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:10.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:10.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:10.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:10.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:10.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:10.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:10.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:10.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:10.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:10.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:10.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:10.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:10.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:10.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:11.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:11.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:11.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:11.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:11.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:11.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:11.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:11.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:11.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:11.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:12.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:48:12.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:48:12.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:12.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:12.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:12.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:12.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:12.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:12.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:12.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:12.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:12.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:12.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:12.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:12.749 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:12.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:12.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:13.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:13.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:13.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:13.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:13.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:13.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:13.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:13.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:13.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:48:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:48:13.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:13.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:13.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:13.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:13.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:13.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:13.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:13.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:13.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:13.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:14.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:48:14.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:14.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:14.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:14.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:48:14.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:48:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:48:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:48:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:48:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:48:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:48:14.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:48:14.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:48:15.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:48:15.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:48:15.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:48:15.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:48:15.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:48:15.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:48:15.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:48:15.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:48:15.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:48:15.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:48:15.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:48:15.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:48:15.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:48:15.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:48:15.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:48:15.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:48:15.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:48:16.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:48:16.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:48:16.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:48:16.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2021-01-15T18:48:16.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:48:16.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:48:16.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-01-15T18:48:16.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2021-01-15T18:48:16.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-01-15T18:48:16.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:48:16.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2021-01-15T18:48:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2021-01-15T18:48:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-01-15T18:48:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-01-15T18:48:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:48:16.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2021-01-15T18:48:16.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2021-01-15T18:48:16.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:48:16.276 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.276 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:48:16.277 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:48:16.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2021-01-15T18:48:16.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:48:16.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:48:16.290 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.290 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2021-01-15T18:48:16.290 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:48:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:48:16.291 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:48:16.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-01-15T18:48:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-01-15T18:48:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:48:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:48:16.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:48:16.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:48:16.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:48:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:48:16.299 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:48:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:48:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:48:16.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:48:16.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:48:16.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:48:16.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:48:16.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:48:16.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:48:16.311 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.311 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2021-01-15T18:48:16.311 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:48:16.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:48:16.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:48:16.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2021-01-15T18:48:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2021-01-15T18:48:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:48:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:48:16.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:48:16.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:48:16.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:48:16.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:48:16.318 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:48:16.318 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:48:16.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2021-01-15T18:48:16.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2021-01-15T18:48:16.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2021-01-15T18:48:16.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:48:16.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2021-01-15T18:48:16.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:48:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-01-15T18:48:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2021-01-15T18:48:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:48:16.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:48:16.875 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:48:16.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2021-01-15T18:48:16.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:48:16.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:48:16.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:48:16.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:48:16.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:48:17.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:48:17.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:48:17.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:48:17.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:48:17.194 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:48:17.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:48:17.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:48:17.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:48:17.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:48:17.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:48:17.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:48:17.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:48:17.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:48:17.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:48:17.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:48:17.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:48:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:48:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:48:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:48:17.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:48:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:48:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:48:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:48:17.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:48:17.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:48:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:48:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:48:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:48:17.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:48:17.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:48:17.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:48:17.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:48:17.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:48:17.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:48:17.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:48:17.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:48:17.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:48:17.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:48:17.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:48:17.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:48:17.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:48:17.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:48:17.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:48:17.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:48:17.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:48:17.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:48:17.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:48:17.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:48:17.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:48:17.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:48:17.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:48:17.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:48:17.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:48:17.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:48:17.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:48:17.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:48:17.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:48:17.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:48:17.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:48:17.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:48:17.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:48:17.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:48:17.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2021-01-15T18:48:17.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2021-01-15T18:48:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2021-01-15T18:48:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:48:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:48:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:48:17.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:48:17.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:48:17.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:48:17.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:17.535 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:17.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:17.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:17.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:48:17.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:48:17.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:48:17.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:48:17.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:48:17.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:48:17.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:48:17.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:48:17.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:48:17.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:48:17.721 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:48:17.722 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.722 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.722 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:48:17.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:48:17.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:48:17.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:48:17.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:48:17.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:48:17.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:48:17.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:48:17.821 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:48:17.821 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:48:17.821 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:48:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:48:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:17.822 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:48:17.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:48:17.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:48:17.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2021-01-15T18:48:17.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:48:17.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:48:17.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:48:17.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:48:17.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:48:17.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:48:18.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:48:18.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:18.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:18.830 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:18.830 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:18.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:48:18.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:48:18.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2021-01-15T18:48:18.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2021-01-15T18:48:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2021-01-15T18:48:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:48:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:48:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:48:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:48:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:18.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:18.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:18.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:18.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:18.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:48:18.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:18.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=2d66b988-6770-4dc0-a465-698d59aa3212 2021-01-15T18:48:18.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 2d66b988-6770-4dc0-a465-698d59aa3212' 2021-01-15T18:48:18.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:18.865 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 2d66b988-6770-4dc0-a465-698d59aa3212 2021-01-15T18:48:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBy4wFgqWnfMxAAEYl5FarmSCeN0Tp7HLiyJQ== 2021-01-15T18:48:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBy4wFgqWnfMxAAEYl5FarmSCeN0Tp7HLiyJQ=="}' 2021-01-15T18:48:18.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 2d66b988-6770-4dc0-a465-698d59aa3212 -i td/osd-recovery-stats/0/new.json 2021-01-15T18:48:19.244 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:19.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:48:19.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBy4wFgqWnfMxAAEYl5FarmSCeN0Tp7HLiyJQ== --osd-uuid 2d66b988-6770-4dc0-a465-698d59aa3212 2021-01-15T18:48:19.278 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.277+0000 7f9bc8562080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:19.280 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.279+0000 7f9bc8562080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:19.281 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.281+0000 7f9bc8562080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:19.282 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.281+0000 7f9bc8562080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:48:19.282 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.281+0000 7f9bc8562080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:48:19.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.281+0000 7f9bc8562080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:48:19.326 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:19.325+0000 7f9bc8562080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:48:23.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:48:23.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:48:23.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:48:23.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:48:23.953 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:48:24.477 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:48:24.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:48:24.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:24.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:48:24.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:48:24.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:48:24.498 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:24.497+0000 7fae9f2d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:24.499 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:24.499+0000 7fae9f2d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:24.500 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:24.500+0000 7fae9f2d2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:24.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:48:24.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:48:24.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:48:24.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:48:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:48:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:48:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:24.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:48:24.874 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:25.020 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:25.019+0000 7fae9f2d2080 -1 Falling back to public interface 2021-01-15T18:48:25.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:48:25.867 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:25.866+0000 7fae9f2d2080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:48:26.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:48:26.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:26.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:48:26.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:26.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:48:26.272 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:26.680 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/865850,v1:127.0.0.1:6803/865850] [v2:127.0.0.1:6804/865850,v1:127.0.0.1:6805/865850] exists,up 2d66b988-6770-4dc0-a465-698d59aa3212 2021-01-15T18:48:26.680 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 6af58a85-847f-4aed-a690-73479b6e52f6 2021-01-15T18:48:26.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:48:26.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:48:26.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:48:26.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:26.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2021-01-15T18:48:26.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:26.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:26.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:48:26.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:26.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:48:26.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:26.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:26.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:26.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:26.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:48:26.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:48:26.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:26.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:26.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:26.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:26.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:26.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:26.685 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:26.685 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:26.685 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:26.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:26.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:26.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:26.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:26.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:26.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:26.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:26.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:26.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:26.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:26.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:26.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:26.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:48:26.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:26.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6af58a85-847f-4aed-a690-73479b6e52f6 2021-01-15T18:48:26.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 6af58a85-847f-4aed-a690-73479b6e52f6' 2021-01-15T18:48:26.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:26.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB64wFgqa5VKRAAgut20gz/ht7eX8FmAlz3QQ== 2021-01-15T18:48:26.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB64wFgqa5VKRAAgut20gz/ht7eX8FmAlz3QQ=="}' 2021-01-15T18:48:26.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6af58a85-847f-4aed-a690-73479b6e52f6 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:48:27.201 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:27.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:48:27.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB64wFgqa5VKRAAgut20gz/ht7eX8FmAlz3QQ== --osd-uuid 6af58a85-847f-4aed-a690-73479b6e52f6 2021-01-15T18:48:27.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.235+0000 7f86274f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:27.239 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.239+0000 7f86274f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:27.241 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.240+0000 7f86274f9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.241+0000 7f86274f9080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:48:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.241+0000 7f86274f9080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:48:27.242 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.241+0000 7f86274f9080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:48:27.340 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:27.339+0000 7f86274f9080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:48:31.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:48:31.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:48:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:48:31.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:48:31.915 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:48:32.443 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:48:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:48:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:32.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:48:32.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:48:32.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:48:32.464 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:32.463+0000 7f8e68203080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:32.487 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:32.486+0000 7f8e68203080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:32.489 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:32.488+0000 7f8e68203080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:32.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:48:32.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:48:32.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:48:32.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:48:32.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:48:32.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:32.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:48:32.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:32.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:48:32.852 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:33.008 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:33.007+0000 7f8e68203080 -1 Falling back to public interface 2021-01-15T18:48:33.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:48:33.789 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:33.788+0000 7f8e68203080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:48:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:48:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:34.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:48:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:34.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:48:34.246 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:34.645 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/867115,v1:127.0.0.1:6811/867115] [v2:127.0.0.1:6812/867115,v1:127.0.0.1:6813/867115] exists,up 6af58a85-847f-4aed-a690-73479b6e52f6 2021-01-15T18:48:34.646 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 b01db13b-7122-439c-813c-f98b26ea830e 2021-01-15T18:48:34.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:48:34.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:48:34.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:48:34.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:34.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2021-01-15T18:48:34.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:34.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:34.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:48:34.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:34.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:48:34.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:34.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:34.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:34.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:34.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:48:34.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:48:34.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:34.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:34.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:34.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:34.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:34.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:34.652 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:34.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:34.653 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:34.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:34.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:34.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:34.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:48:34.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:34.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b01db13b-7122-439c-813c-f98b26ea830e 2021-01-15T18:48:34.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 b01db13b-7122-439c-813c-f98b26ea830e' 2021-01-15T18:48:34.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:34.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCC4wFgTcM1JxAAPKN1+hnL1RvPidp6EreMOQ== 2021-01-15T18:48:34.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCC4wFgTcM1JxAAPKN1+hnL1RvPidp6EreMOQ=="}' 2021-01-15T18:48:34.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b01db13b-7122-439c-813c-f98b26ea830e -i td/osd-recovery-stats/2/new.json 2021-01-15T18:48:35.274 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:48:35.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:48:35.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCC4wFgTcM1JxAAPKN1+hnL1RvPidp6EreMOQ== --osd-uuid b01db13b-7122-439c-813c-f98b26ea830e 2021-01-15T18:48:35.310 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.309+0000 7f421533d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:35.311 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.311+0000 7f421533d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:35.313 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.312+0000 7f421533d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:35.314 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.313+0000 7f421533d080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:48:35.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.313+0000 7f421533d080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:48:35.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.313+0000 7f421533d080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:48:35.353 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:35.352+0000 7f421533d080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:48:40.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:48:40.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:48:40.181 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:48:40.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:48:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:48:40.707 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:48:40.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:48:40.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:40.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:48:40.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:48:40.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:48:40.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:40.726+0000 7f717696f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:40.754 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:40.753+0000 7f717696f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:40.755 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:40.754+0000 7f717696f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:41.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:48:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:48:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:48:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:48:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:48:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:48:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:48:41.104 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:41.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:41.273+0000 7f717696f080 -1 Falling back to public interface 2021-01-15T18:48:41.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:48:42.122 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:42.121+0000 7f717696f080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:48:42.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:48:42.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:42.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:48:42.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:42.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:48:42.497 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:42.910 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/868387,v1:127.0.0.1:6819/868387] [v2:127.0.0.1:6820/868387,v1:127.0.0.1:6821/868387] exists,up b01db13b-7122-439c-813c-f98b26ea830e 2021-01-15T18:48:42.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2021-01-15T18:48:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:42.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:42.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:48:42.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:42.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:48:42.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:42.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:42.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:42.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:42.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:48:42.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:48:42.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:42.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:42.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:42.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:42.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:42.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:42.915 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:42.915 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:42.916 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:42.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:42.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:42.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:42.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:42.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:42.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:42.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:42.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:42.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:48:42.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:42.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=07771047-dd7e-4690-9219-bf29786cf5d6 2021-01-15T18:48:42.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 07771047-dd7e-4690-9219-bf29786cf5d6' 2021-01-15T18:48:42.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:42.920 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 07771047-dd7e-4690-9219-bf29786cf5d6 2021-01-15T18:48:42.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCK4wFgN9MUNxAASfuQLEeTU+3Cu6ZCVLcOOg== 2021-01-15T18:48:42.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCK4wFgN9MUNxAASfuQLEeTU+3Cu6ZCVLcOOg=="}' 2021-01-15T18:48:42.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 07771047-dd7e-4690-9219-bf29786cf5d6 -i td/osd-recovery-stats/3/new.json 2021-01-15T18:48:43.488 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:48:43.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:48:43.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCK4wFgN9MUNxAASfuQLEeTU+3Cu6ZCVLcOOg== --osd-uuid 07771047-dd7e-4690-9219-bf29786cf5d6 2021-01-15T18:48:43.523 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.522+0000 7fa51d338080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:43.524 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.524+0000 7fa51d338080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:43.525 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.524+0000 7fa51d338080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:43.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.525+0000 7fa51d338080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:48:43.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.525+0000 7fa51d338080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:48:43.527 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.525+0000 7fa51d338080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:48:43.569 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:43.568+0000 7fa51d338080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:48:48.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:48:48.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:48:48.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:48:48.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:48:48.441 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:48:48.966 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:48:48.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:48:48.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:48.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:48:48.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:48:48.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:48:48.987 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:48.986+0000 7f5ae718d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:48.989 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:48.988+0000 7f5ae718d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:48.991 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:48.990+0000 7f5ae718d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:49.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:48:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:48:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:48:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:48:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:48:49.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:49.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:48:49.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:49.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:48:49.356 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:49.506 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:49.506+0000 7f5ae718d080 -1 Falling back to public interface 2021-01-15T18:48:49.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:48:50.395 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:50.394+0000 7f5ae718d080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:48:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:48:50.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:50.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:48:50.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:50.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:48:50.770 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:51.180 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/869651,v1:127.0.0.1:6827/869651] [v2:127.0.0.1:6828/869651,v1:127.0.0.1:6829/869651] exists,up 07771047-dd7e-4690-9219-bf29786cf5d6 2021-01-15T18:48:51.181 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 53788dd7-4757-4bef-b013-3c91ed49c69a 2021-01-15T18:48:51.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:48:51.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:48:51.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:48:51.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:51.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2021-01-15T18:48:51.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:51.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:51.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:48:51.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:51.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/4 2021-01-15T18:48:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:51.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:51.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:51.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:51.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-01-15T18:48:51.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-01-15T18:48:51.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:51.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:51.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:51.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:51.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:51.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:51.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:51.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:51.198 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:51.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:51.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:51.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:51.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:51.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:51.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:51.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:51.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:51.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/4 2021-01-15T18:48:51.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:51.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=53788dd7-4757-4bef-b013-3c91ed49c69a 2021-01-15T18:48:51.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 53788dd7-4757-4bef-b013-3c91ed49c69a' 2021-01-15T18:48:51.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:51.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCT4wFgJm8VCxAA10dOHdMT867+kORtPjow1A== 2021-01-15T18:48:51.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCT4wFgJm8VCxAA10dOHdMT867+kORtPjow1A=="}' 2021-01-15T18:48:51.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 53788dd7-4757-4bef-b013-3c91ed49c69a -i td/osd-recovery-stats/4/new.json 2021-01-15T18:48:51.713 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:48:51.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/4/new.json 2021-01-15T18:48:51.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCT4wFgJm8VCxAA10dOHdMT867+kORtPjow1A== --osd-uuid 53788dd7-4757-4bef-b013-3c91ed49c69a 2021-01-15T18:48:51.748 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.747+0000 7f3be2a6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:51.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.749+0000 7f3be2a6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:51.750 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.750+0000 7f3be2a6a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:51.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.750+0000 7f3be2a6a080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:48:51.751 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.750+0000 7f3be2a6a080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:48:51.752 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.750+0000 7f3be2a6a080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:48:51.861 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:51.860+0000 7f3be2a6a080 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-01-15T18:48:56.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-01-15T18:48:56.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:48:56.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:48:56.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:48:56.562 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:48:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:48:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:48:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:48:57.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:48:57.118 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:48:57.138 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:57.137+0000 7fafce955080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:57.139 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:57.138+0000 7fafce955080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:57.141 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:57.139+0000 7fafce955080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:48:57.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:48:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:48:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:48:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:48:57.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:48:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:48:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:57.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:48:57.536 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:48:57.664 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:57.664+0000 7fafce955080 -1 Falling back to public interface 2021-01-15T18:48:57.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:48:58.512 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:48:58.511+0000 7fafce955080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:48:58.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:48:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:48:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:48:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:48:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:48:58.960 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:48:59.377 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/870915,v1:127.0.0.1:6835/870915] [v2:127.0.0.1:6836/870915,v1:127.0.0.1:6837/870915] exists,up 53788dd7-4757-4bef-b013-3c91ed49c69a 2021-01-15T18:48:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:48:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:48:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:48:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:48:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2021-01-15T18:48:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:48:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:48:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:48:59.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:48:59.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/5 2021-01-15T18:48:59.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:48:59.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:48:59.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:48:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:48:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-01-15T18:48:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-01-15T18:48:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:48:59.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:48:59.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:48:59.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:48:59.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:48:59.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:48:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:48:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:48:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:48:59.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:48:59.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:48:59.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:48:59.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:48:59.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:48:59.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:48:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:48:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:48:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:48:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:48:59.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:48:59.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/5 2021-01-15T18:48:59.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:48:59.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=adac28cd-5f7a-4cea-842f-b696ce168387 2021-01-15T18:48:59.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 adac28cd-5f7a-4cea-842f-b696ce168387' 2021-01-15T18:48:59.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:48:59.387 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 adac28cd-5f7a-4cea-842f-b696ce168387 2021-01-15T18:48:59.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCb4wFgu9VbFxAAHaKpyq5TGkzwcxvUqOSh3Q== 2021-01-15T18:48:59.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCb4wFgu9VbFxAAHaKpyq5TGkzwcxvUqOSh3Q=="}' 2021-01-15T18:48:59.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new adac28cd-5f7a-4cea-842f-b696ce168387 -i td/osd-recovery-stats/5/new.json 2021-01-15T18:48:59.988 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:48:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/5/new.json 2021-01-15T18:49:00.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCb4wFgu9VbFxAAHaKpyq5TGkzwcxvUqOSh3Q== --osd-uuid adac28cd-5f7a-4cea-842f-b696ce168387 2021-01-15T18:49:00.023 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.022+0000 7f69d5dea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:00.024 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.024+0000 7f69d5dea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:00.025 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.024+0000 7f69d5dea080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:00.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.025+0000 7f69d5dea080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:49:00.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.025+0000 7f69d5dea080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:49:00.026 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.025+0000 7f69d5dea080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:49:00.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:00.084+0000 7f69d5dea080 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-01-15T18:49:05.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-01-15T18:49:05.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:49:05.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:49:05.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:49:05.076 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:49:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:49:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:49:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:49:05.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:49:05.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:49:05.597 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:49:05.617 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:05.616+0000 7fe50f77b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:05.645 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:05.644+0000 7fe50f77b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:05.646 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:05.645+0000 7fe50f77b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:05.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:49:05.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:49:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:49:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:49:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:49:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:05.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:49:05.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:05.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:49:05.990 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:49:06.163 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:06.162+0000 7fe50f77b080 -1 Falling back to public interface 2021-01-15T18:49:06.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:49:07.064 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:07.063+0000 7fe50f77b080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:49:07.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:49:07.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:07.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:49:07.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:07.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:49:07.388 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:49:07.795 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/872179,v1:127.0.0.1:6843/872179] [v2:127.0.0.1:6844/872179,v1:127.0.0.1:6845/872179] exists,up adac28cd-5f7a-4cea-842f-b696ce168387 2021-01-15T18:49:07.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:49:07.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:49:07.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:49:07.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2021-01-15T18:49:07.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:49:08.383 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:49:08.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:49:09.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2021-01-15T18:49:10.256 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T18:49:10.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2021-01-15T18:49:10.983 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 min_size to 1 2021-01-15T18:49:10.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2021-01-15T18:49:10.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:49:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:49:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:49:10.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:49:10.995 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:49:10.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:49:10.996 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:49:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:49:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:49:10.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:49:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:49:11.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:49:11.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:49:11.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:49:11.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:49:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:49:11.519 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:49:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:49:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:49:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:49:11.520 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:49:11.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:49:11.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:11.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:49:11.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T18:49:11.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T18:49:11.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T18:49:11.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:11.656 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:49:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T18:49:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T18:49:11.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2021-01-15T18:49:11.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:11.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:49:11.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T18:49:11.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T18:49:11.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448' 2021-01-15T18:49:11.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:11.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:49:12.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T18:49:12.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T18:49:12.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927' 2021-01-15T18:49:12.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:12.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:49:12.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215109 2021-01-15T18:49:12.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215109 2021-01-15T18:49:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927 4-103079215109' 2021-01-15T18:49:12.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:49:12.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:49:12.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051587 2021-01-15T18:49:12.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051587 2021-01-15T18:49:12.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927 4-103079215109 5-124554051587' 2021-01-15T18:49:12.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:12.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T18:49:12.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:12.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:49:12.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T18:49:12.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:12.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T18:49:12.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T18:49:12.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:49:12.337 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T18:49:12.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836489 -lt 21474836491 2021-01-15T18:49:12.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:49:13.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:49:13.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:49:14.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836491 2021-01-15T18:49:14.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:14.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T18:49:14.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:14.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:49:14.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T18:49:14.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:14.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T18:49:14.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T18:49:14.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:49:14.140 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T18:49:14.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T18:49:14.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:14.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T18:49:14.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:14.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:49:14.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T18:49:14.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:14.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T18:49:14.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T18:49:14.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:49:14.569 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T18:49:14.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509449 -lt 64424509448 2021-01-15T18:49:15.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:15.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T18:49:15.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:15.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:49:15.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T18:49:15.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:15.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T18:49:15.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T18:49:15.134 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:49:15.134 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T18:49:15.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345927 2021-01-15T18:49:15.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:15.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215109 2021-01-15T18:49:15.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:15.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:49:15.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215109 2021-01-15T18:49:15.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:15.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215109 2021-01-15T18:49:15.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215109' 2021-01-15T18:49:15.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:49:15.413 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215109 2021-01-15T18:49:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215109 -lt 103079215109 2021-01-15T18:49:15.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:49:15.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051587 2021-01-15T18:49:15.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:49:15.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:49:15.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051587 2021-01-15T18:49:15.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:49:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051587 2021-01-15T18:49:15.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051587' 2021-01-15T18:49:15.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:49:15.834 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 124554051587 2021-01-15T18:49:16.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051588 -lt 124554051587 2021-01-15T18:49:16.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:49:16.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:49:16.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:49:16.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:49:16.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:49:16.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:49:16.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:49:16.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:49:16.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:49:16.746 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:49:16.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:49:17.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:49:17.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:49:17.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:49:17.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:49:17.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:49:17.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:49:17.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:49:17.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2021-01-15T18:49:17.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2021-01-15T18:49:17.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:49:17.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:49:17.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:49:17.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:49:18.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2021-01-15T18:49:18.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2021-01-15T18:49:18.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:49:18.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:49:18.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:49:18.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:49:18.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:49:18.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:49:18.181 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:49:18.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:49:18.609 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:49:18.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:49:19.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2021-01-15T18:49:19.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2021-01-15T18:49:19.716 INFO:tasks.workunit.client.0.smithi145.stderr:noout is set 2021-01-15T18:49:19.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2021-01-15T18:49:20.486 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:49:20.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2021-01-15T18:49:20.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 865850 2021-01-15T18:49:20.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2021-01-15T18:49:20.948 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:49:20.955 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2021-01-15T18:49:20.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2021-01-15T18:49:20.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2021-01-15T18:49:20.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:20.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2021-01-15T18:49:21.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:21.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2021-01-15T18:49:21.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:21.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2021-01-15T18:49:21.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:21.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2021-01-15T18:49:21.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:21.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2021-01-15T18:49:22.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2021-01-15T18:49:22.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2021-01-15T18:49:22.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2021-01-15T18:49:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2021-01-15T18:49:22.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2021-01-15T18:49:22.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2021-01-15T18:49:22.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2021-01-15T18:49:22.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:22.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2021-01-15T18:49:23.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2021-01-15T18:49:23.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2021-01-15T18:49:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2021-01-15T18:49:23.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2021-01-15T18:49:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2021-01-15T18:49:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2021-01-15T18:49:23.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2021-01-15T18:49:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2021-01-15T18:49:23.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:23.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2021-01-15T18:49:24.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2021-01-15T18:49:24.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2021-01-15T18:49:24.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2021-01-15T18:49:24.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2021-01-15T18:49:24.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2021-01-15T18:49:24.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2021-01-15T18:49:24.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2021-01-15T18:49:24.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2021-01-15T18:49:24.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:24.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2021-01-15T18:49:25.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2021-01-15T18:49:25.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2021-01-15T18:49:25.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2021-01-15T18:49:25.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2021-01-15T18:49:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2021-01-15T18:49:25.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2021-01-15T18:49:25.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2021-01-15T18:49:25.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2021-01-15T18:49:25.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:25.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2021-01-15T18:49:26.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2021-01-15T18:49:26.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2021-01-15T18:49:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2021-01-15T18:49:26.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2021-01-15T18:49:26.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2021-01-15T18:49:26.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2021-01-15T18:49:26.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2021-01-15T18:49:26.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2021-01-15T18:49:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2021-01-15T18:49:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2021-01-15T18:49:27.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2021-01-15T18:49:27.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2021-01-15T18:49:27.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2021-01-15T18:49:27.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2021-01-15T18:49:27.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2021-01-15T18:49:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2021-01-15T18:49:27.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2021-01-15T18:49:27.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2021-01-15T18:49:27.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:27.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2021-01-15T18:49:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2021-01-15T18:49:28.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2021-01-15T18:49:28.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2021-01-15T18:49:28.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2021-01-15T18:49:28.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2021-01-15T18:49:28.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2021-01-15T18:49:28.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2021-01-15T18:49:28.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:28.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2021-01-15T18:49:29.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2021-01-15T18:49:29.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2021-01-15T18:49:29.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2021-01-15T18:49:29.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2021-01-15T18:49:29.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2021-01-15T18:49:29.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2021-01-15T18:49:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2021-01-15T18:49:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2021-01-15T18:49:29.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2021-01-15T18:49:29.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:29.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2021-01-15T18:49:30.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2021-01-15T18:49:30.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2021-01-15T18:49:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2021-01-15T18:49:30.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2021-01-15T18:49:30.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2021-01-15T18:49:30.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2021-01-15T18:49:30.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2021-01-15T18:49:30.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2021-01-15T18:49:30.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:30.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2021-01-15T18:49:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2021-01-15T18:49:31.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2021-01-15T18:49:31.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2021-01-15T18:49:31.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2021-01-15T18:49:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2021-01-15T18:49:31.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2021-01-15T18:49:31.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2021-01-15T18:49:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2021-01-15T18:49:31.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:31.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2021-01-15T18:49:32.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:32.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2021-01-15T18:49:32.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:32.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2021-01-15T18:49:32.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:32.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2021-01-15T18:49:32.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2021-01-15T18:49:32.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2021-01-15T18:49:32.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2021-01-15T18:49:32.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 867115 2021-01-15T18:49:32.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2021-01-15T18:49:33.022 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T18:49:33.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2021-01-15T18:49:33.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-recovery-stats 2021-01-15T18:49:33.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:49:33.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T18:49:33.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:49:33.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:49:33.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:49:33.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:49:33.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:49:33.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:49:33.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:49:33.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:49:33.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:49:33.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:49:33.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:49:33.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:49:33.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:49:33.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:49:33.038 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:49:33.038 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:49:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:49:33.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:49:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:49:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:49:33.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:49:33.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:49:33.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:49:33.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:49:33.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:49:33.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:49:33.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:49:33.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:49:33.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T18:49:33.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:49:33.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-recovery-stats/0/whoami 2021-01-15T18:49:33.042 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:49:33.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T18:49:33.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:49:33.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:49:33.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:49:33.055 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:33.054+0000 7f6d56341080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:33.057 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:33.056+0000 7f6d56341080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:33.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:33.057+0000 7f6d56341080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:33.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T18:49:33.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:49:33.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:49:33.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:49:33.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:49:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:49:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:49:33.448 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:49:33.573 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:33.571+0000 7f6d56341080 -1 Falling back to public interface 2021-01-15T18:49:33.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:49:34.835 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:34.834+0000 7f6d56341080 -1 osd.0 39 log_to_monitors {default=true} 2021-01-15T18:49:34.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:49:34.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:34.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:49:34.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:34.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:49:34.862 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:49:35.300 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 46 up_thru 46 down_at 42 last_clean_interval [5,39) [v2:127.0.0.1:6802/875477,v1:127.0.0.1:6803/875477] [v2:127.0.0.1:6804/875477,v1:127.0.0.1:6805/875477] exists,up 2d66b988-6770-4dc0-a465-698d59aa3212 2021-01-15T18:49:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:49:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:49:35.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:49:35.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2021-01-15T18:49:38.301 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2021-01-15T18:49:38.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2021-01-15T18:49:38.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2021-01-15T18:49:38.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2021-01-15T18:49:38.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2021-01-15T18:49:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2021-01-15T18:49:38.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2021-01-15T18:49:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2021-01-15T18:49:39.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2021-01-15T18:49:39.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2021-01-15T18:49:39.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2021-01-15T18:49:39.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2021-01-15T18:49:39.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2021-01-15T18:49:39.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2021-01-15T18:49:39.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2021-01-15T18:49:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2021-01-15T18:49:39.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2021-01-15T18:49:39.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:39.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2021-01-15T18:49:40.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2021-01-15T18:49:40.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2021-01-15T18:49:40.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2021-01-15T18:49:40.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2021-01-15T18:49:40.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2021-01-15T18:49:40.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2021-01-15T18:49:40.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2021-01-15T18:49:40.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2021-01-15T18:49:40.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2021-01-15T18:49:40.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:40.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2021-01-15T18:49:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2021-01-15T18:49:41.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2021-01-15T18:49:41.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2021-01-15T18:49:41.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2021-01-15T18:49:41.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2021-01-15T18:49:41.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2021-01-15T18:49:41.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2021-01-15T18:49:41.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2021-01-15T18:49:41.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:41.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2021-01-15T18:49:42.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2021-01-15T18:49:42.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2021-01-15T18:49:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2021-01-15T18:49:42.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2021-01-15T18:49:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2021-01-15T18:49:42.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2021-01-15T18:49:42.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2021-01-15T18:49:42.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2021-01-15T18:49:42.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:42.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2021-01-15T18:49:43.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2021-01-15T18:49:43.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2021-01-15T18:49:43.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2021-01-15T18:49:43.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2021-01-15T18:49:43.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2021-01-15T18:49:43.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2021-01-15T18:49:43.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2021-01-15T18:49:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2021-01-15T18:49:43.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:43.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2021-01-15T18:49:44.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2021-01-15T18:49:44.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2021-01-15T18:49:44.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2021-01-15T18:49:44.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2021-01-15T18:49:44.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2021-01-15T18:49:44.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2021-01-15T18:49:44.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2021-01-15T18:49:44.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2021-01-15T18:49:44.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:44.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2021-01-15T18:49:45.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2021-01-15T18:49:45.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2021-01-15T18:49:45.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2021-01-15T18:49:45.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2021-01-15T18:49:45.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2021-01-15T18:49:45.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2021-01-15T18:49:45.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2021-01-15T18:49:45.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2021-01-15T18:49:45.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:45.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2021-01-15T18:49:46.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2021-01-15T18:49:46.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2021-01-15T18:49:46.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2021-01-15T18:49:46.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2021-01-15T18:49:46.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2021-01-15T18:49:46.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2021-01-15T18:49:46.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2021-01-15T18:49:46.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2021-01-15T18:49:46.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2021-01-15T18:49:46.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:46.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2021-01-15T18:49:47.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2021-01-15T18:49:47.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2021-01-15T18:49:47.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2021-01-15T18:49:47.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2021-01-15T18:49:47.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2021-01-15T18:49:47.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2021-01-15T18:49:47.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2021-01-15T18:49:47.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2021-01-15T18:49:47.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2021-01-15T18:49:48.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2021-01-15T18:49:48.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2021-01-15T18:49:48.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2021-01-15T18:49:48.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2021-01-15T18:49:48.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2021-01-15T18:49:48.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2021-01-15T18:49:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2021-01-15T18:49:48.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2021-01-15T18:49:48.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:48.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2021-01-15T18:49:49.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2021-01-15T18:49:49.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2021-01-15T18:49:49.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2021-01-15T18:49:49.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:49:49.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:49:49.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:49:49.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:49:49.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2021-01-15T18:49:49.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2021-01-15T18:49:49.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:49:49.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj200 2021-01-15T18:49:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj200 2021-01-15T18:49:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:49:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj200 2021-01-15T18:49:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj200 2021-01-15T18:49:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:49:49.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=0 2021-01-15T18:49:49.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj200 2021-01-15T18:49:49.964 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2021-01-15T18:49:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2021-01-15T18:49:50.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2021-01-15T18:49:51.136 INFO:tasks.workunit.client.0.smithi145.stderr:noout is unset 2021-01-15T18:49:51.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2021-01-15T18:49:52.075 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.2 is already out. 2021-01-15T18:49:52.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2021-01-15T18:49:52.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-recovery-stats 2021-01-15T18:49:52.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T18:49:52.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T18:49:52.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T18:49:52.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:49:52.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:49:52.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:49:52.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:49:52.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:49:52.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:49:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:49:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T18:49:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T18:49:52.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:49:52.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T18:49:52.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:49:52.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:49:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:49:52.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:49:52.092 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:49:52.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:49:52.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:49:52.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:49:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:49:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:49:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:49:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:49:52.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:49:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T18:49:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T18:49:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:49:52.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T18:49:52.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:49:52.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-recovery-stats/1/whoami 2021-01-15T18:49:52.095 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:49:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T18:49:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T18:49:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T18:49:52.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T18:49:52.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:52.108+0000 7f43484a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:52.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:52.109+0000 7f43484a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:52.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:52.110+0000 7f43484a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:49:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T18:49:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:49:52.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:49:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:49:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:49:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:49:52.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:52.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:49:52.520 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:49:52.630 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:52.629+0000 7f43484a2080 -1 Falling back to public interface 2021-01-15T18:49:52.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:49:53.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:49:53.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:53.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:49:53.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:53.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:49:53.980 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:49:54.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:49:54.410 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:49:54.409+0000 7f43484a2080 -1 osd.1 43 log_to_monitors {default=true} 2021-01-15T18:49:55.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:49:55.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:49:55.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:49:55.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:49:55.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:49:55.377 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:49:55.777 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up out weight 0 up_from 54 up_thru 42 down_at 44 last_clean_interval [10,43) [v2:127.0.0.1:6810/878104,v1:127.0.0.1:6811/878104] [v2:127.0.0.1:6812/878104,v1:127.0.0.1:6813/878104] exists,up 6af58a85-847f-4aed-a690-73479b6e52f6 2021-01-15T18:49:55.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:49:55.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:49:55.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:49:55.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2021-01-15T18:49:58.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2021-01-15T18:49:59.450 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 4 2021-01-15T18:49:59.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2021-01-15T18:50:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:50:00.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2021-01-15T18:50:00.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:50:00.542 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:50:00.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:50:00.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:50:00.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2021-01-15T18:50:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:50:01.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2021-01-15T18:50:03.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2021-01-15T18:50:03.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:50:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:50:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:50:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:50:03.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:50:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:50:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:50:03.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:50:03.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:50:03.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:50:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:50:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:50:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:50:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:50:03.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:50:03.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:50:03.643 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:50:03.643 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:50:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:50:03.644 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:50:03.645 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:50:03.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:50:03.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:03.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:50:03.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=197568495624 2021-01-15T18:50:03.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 197568495624 2021-01-15T18:50:03.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624' 2021-01-15T18:50:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:03.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:50:03.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=231928233988 2021-01-15T18:50:03.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 231928233988 2021-01-15T18:50:03.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624 1-231928233988' 2021-01-15T18:50:03.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:03.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:50:04.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509461 2021-01-15T18:50:04.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509461 2021-01-15T18:50:04.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624 1-231928233988 2-64424509461' 2021-01-15T18:50:04.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:04.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:50:04.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345939 2021-01-15T18:50:04.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345939 2021-01-15T18:50:04.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624 1-231928233988 2-64424509461 3-85899345939' 2021-01-15T18:50:04.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:04.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:50:04.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=103079215121 2021-01-15T18:50:04.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 103079215121 2021-01-15T18:50:04.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624 1-231928233988 2-64424509461 3-85899345939 4-103079215121' 2021-01-15T18:50:04.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:50:04.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:50:04.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051600 2021-01-15T18:50:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051600 2021-01-15T18:50:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-197568495624 1-231928233988 2-64424509461 3-85899345939 4-103079215121 5-124554051600' 2021-01-15T18:50:04.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:04.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-197568495624 2021-01-15T18:50:04.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:04.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:50:04.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-197568495624 2021-01-15T18:50:04.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:04.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=197568495624 2021-01-15T18:50:04.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 197568495624' 2021-01-15T18:50:04.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:50:04.504 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 197568495624 2021-01-15T18:50:04.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 197568495624 -lt 197568495624 2021-01-15T18:50:04.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:04.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-231928233988 2021-01-15T18:50:04.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:04.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:50:04.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-231928233988 2021-01-15T18:50:04.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:04.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=231928233988 2021-01-15T18:50:04.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 231928233988' 2021-01-15T18:50:04.903 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 231928233988 2021-01-15T18:50:04.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:50:05.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 231928233989 -lt 231928233988 2021-01-15T18:50:05.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:05.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509461 2021-01-15T18:50:05.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:05.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:50:05.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509461 2021-01-15T18:50:05.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:05.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509461 2021-01-15T18:50:05.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2021-01-15T18:50:05.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:50:05.299 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509461 2021-01-15T18:50:05.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509461 -lt 64424509461 2021-01-15T18:50:05.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:05.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345939 2021-01-15T18:50:05.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:05.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:50:05.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345939 2021-01-15T18:50:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345939 2021-01-15T18:50:05.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2021-01-15T18:50:05.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:50:05.993 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345939 2021-01-15T18:50:06.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345939 -lt 85899345939 2021-01-15T18:50:06.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:06.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215121 2021-01-15T18:50:06.143 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:06.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:50:06.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-103079215121 2021-01-15T18:50:06.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:06.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=103079215121 2021-01-15T18:50:06.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 103079215121' 2021-01-15T18:50:06.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:50:06.146 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 103079215121 2021-01-15T18:50:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 103079215121 -lt 103079215121 2021-01-15T18:50:06.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:50:06.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051600 2021-01-15T18:50:06.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:50:06.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:50:06.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-124554051600 2021-01-15T18:50:06.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:50:06.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051600 2021-01-15T18:50:06.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 124554051600' 2021-01-15T18:50:06.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:50:06.561 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 124554051600 2021-01-15T18:50:06.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051600 -lt 124554051600 2021-01-15T18:50:06.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:50:06.974 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:06.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:07.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:50:07.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:07.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:07.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:07.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:07.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:07.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:07.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:07.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:07.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:07.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:07.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:50:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:50:08.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:50:08.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:08.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:08.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:08.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:08.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:08.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:08.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:08.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:08.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:08.607 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:09.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:09.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:09.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:09.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:09.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:09.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:09.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:09.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:09.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:10.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=1 2021-01-15T18:50:10.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 1 '!=' null 2021-01-15T18:50:10.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:10.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:10.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:10.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:10.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:10.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:10.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:10.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:10.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:10.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:10.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:10.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:10.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:10.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:10.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:11.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:11.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:11.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:11.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:11.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:11.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:11.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:11.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:11.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:11.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:11.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:11.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:11.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:11.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:11.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:11.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:11.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:11.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:11.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:11.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:11.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:11.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:12.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:12.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:12.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:12.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:12.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:12.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:12.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:12.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:12.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:12.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:12.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:12.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:13.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:50:13.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:50:13.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:13.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:13.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:13.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:13.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:13.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:13.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:13.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:13.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:13.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:13.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:13.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:13.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:13.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:13.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:14.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:14.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:14.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:14.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:14.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:14.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:14.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:14.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:14.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:14.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:50:14.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:50:14.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:14.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:14.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:14.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:14.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:14.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:14.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:14.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:14.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:14.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:14.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:15.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:15.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:15.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:15.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:15.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:15.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:15.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:15.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:15.670 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:16.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:50:16.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:50:16.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:16.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:16.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:16.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:16.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:16.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:16.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:16.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:16.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:16.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:16.290 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:16.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:16.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:16.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:16.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:17.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:17.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:17.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:17.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:17.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:17.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:17.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:17.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:17.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:17.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:17.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:17.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:17.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:17.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:17.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:17.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:17.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:17.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:17.829 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:18.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:18.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:18.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:18.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:18.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:18.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:18.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:18.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:18.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:18.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:19.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:19.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:19.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:19.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:19.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:19.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:19.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:19.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:19.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:19.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:19.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:19.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:19.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:19.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:19.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:19.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:19.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:20.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:20.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:20.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:20.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:20.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:20.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:20.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:20.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:20.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:20.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:20.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:20.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:20.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:20.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:20.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:20.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:20.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:21.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:21.257 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:21.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:21.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:21.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:21.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:22.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:22.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:22.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:22.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:22.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:22.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:22.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:22.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:22.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:22.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:22.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:22.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:22.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:22.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:22.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:22.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:22.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:23.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:23.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:23.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:23.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:23.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:23.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:23.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:23.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:23.298 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:23.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:23.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:23.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:23.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:23.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:23.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:23.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:23.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:23.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:23.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:23.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:23.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:23.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:24.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:24.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:24.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:24.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:24.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:24.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:24.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:24.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:24.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:24.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:24.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:24.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:24.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:25.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:25.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:25.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:25.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:25.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:25.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:25.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:25.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:25.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:25.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:25.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:25.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:25.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:25.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:26.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:26.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:26.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:26.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:26.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:26.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:26.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:26.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:26.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:26.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:26.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:26.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:26.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:26.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:27.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:27.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:27.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:27.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:27.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:27.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:27.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:27.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:27.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:27.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:27.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:27.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:27.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:28.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:28.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:28.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:28.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:28.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:28.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:28.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:28.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:28.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:28.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:28.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:28.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:28.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:28.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:28.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:28.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:28.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:29.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:29.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:29.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:29.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:29.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:29.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:29.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:29.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:29.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:29.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:29.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:30.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:30.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:30.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:30.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:30.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:30.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:30.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:30.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:30.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:30.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:30.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:30.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:30.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:30.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:30.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:30.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:30.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:30.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:30.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:30.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:30.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:31.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:31.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:31.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:31.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:31.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:31.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:31.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:31.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:31.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:31.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:31.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:31.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:31.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:31.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:31.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:31.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:31.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:32.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:32.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:32.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:32.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:32.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:32.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:32.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:32.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:33.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:33.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:33.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:33.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:33.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:33.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:33.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:33.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:33.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:33.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:33.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:33.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:33.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:33.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:33.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:33.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:33.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:33.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:33.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:33.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:33.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:34.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:34.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:34.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:34.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:34.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:34.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:34.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:34.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:35.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:35.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:35.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:35.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:35.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:35.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:35.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:35.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:35.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:35.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:35.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:35.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:35.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:36.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:36.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:36.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:36.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:36.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:36.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:36.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:36.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:36.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:36.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:36.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:36.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:36.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:36.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:36.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:37.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:37.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:37.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:37.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:37.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:50:37.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:50:37.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:37.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:37.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:37.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:37.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:37.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:37.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:37.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:37.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:37.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:37.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:38.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:38.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:38.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:38.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:38.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:38.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:38.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:38.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:38.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:38.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:38.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:38.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:38.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:39.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:39.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:39.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:39.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:39.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:39.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:39.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:39.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:39.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:39.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:40.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:40.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:40.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:40.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:40.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:40.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:40.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:40.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:40.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:40.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:40.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:40.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:40.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:40.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:41.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:41.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:41.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:41.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:41.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:41.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:41.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:41.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:41.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:42.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:50:42.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:50:42.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:42.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:42.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:42.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:42.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:42.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:42.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:42.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:42.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:42.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:42.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:43.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:43.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:43.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:43.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:43.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:43.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:43.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:43.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:43.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:43.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:43.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:43.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:43.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:43.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:43.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:43.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:43.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:43.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:43.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:43.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:44.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:44.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:44.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:44.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:44.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:44.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:44.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:44.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:44.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:44.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:45.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:45.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:45.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:45.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:45.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:45.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:45.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:45.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:45.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:45.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:45.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:45.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:45.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:45.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:46.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:46.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:46.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:46.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:46.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:46.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:46.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:46.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:46.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:46.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:46.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:46.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:46.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:46.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:46.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:46.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:46.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:47.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:47.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:47.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:47.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:47.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:47.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:47.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:47.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:47.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:47.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:47.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:47.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:47.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:48.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:50:48.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:50:48.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:48.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:48.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:48.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:48.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:48.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:48.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:48.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:48.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:48.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:48.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:48.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:48.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:48.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:48.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:49.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:49.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:49.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:49.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:49.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:49.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:49.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:49.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:49.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:50:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:50:49.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:49.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:49.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:49.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:49.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:49.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:49.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:49.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:49.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:49.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:50.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:50:50.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:50.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:50.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:50.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:50:50.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:50:50.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:50:50.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:50:50.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:50:50.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:50:50.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:50:50.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:50:50.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:50:51.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:50:51.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:50:51.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:50:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:50:51.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:50:51.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:50:51.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:50:51.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:50:51.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:50:51.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:50:51.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:50:51.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:50:51.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:50:51.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:50:51.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:50:51.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:50:51.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:50:52.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:50:52.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:50:52.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:50:52.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2021-01-15T18:50:52.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:50:52.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:50:52.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:50:52.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:50:52.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2021-01-15T18:50:52.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2021-01-15T18:50:52.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:50:52.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:50:52.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2021-01-15T18:50:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-01-15T18:50:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2021-01-15T18:50:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:50:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2021-01-15T18:50:52.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:50:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2021-01-15T18:50:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2021-01-15T18:50:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:50:52.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-01-15T18:50:52.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:50:52.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:50:52.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:50:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2021-01-15T18:50:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:50:52.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-01-15T18:50:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:50:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:50:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2021-01-15T18:50:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2021-01-15T18:50:52.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2021-01-15T18:50:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:50:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2021-01-15T18:50:52.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:50:52.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:50:52.760 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:50:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:50:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:50:52.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:50:52.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:50:52.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:50:52.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:50:52.771 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.772 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.772 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-01-15T18:50:52.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:50:52.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:50:52.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2021-01-15T18:50:52.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2021-01-15T18:50:52.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2021-01-15T18:50:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:50:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2021-01-15T18:50:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:50:52.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.776 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:50:52.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:50:52.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:50:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:50:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:50:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:50:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:50:52.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2021-01-15T18:50:52.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2021-01-15T18:50:52.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2021-01-15T18:50:52.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2021-01-15T18:50:52.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2021-01-15T18:50:52.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2021-01-15T18:50:52.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2021-01-15T18:50:52.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2021-01-15T18:50:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:50:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2021-01-15T18:50:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:50:52.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:50:52.792 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2021-01-15T18:50:52.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2021-01-15T18:50:52.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:50:52.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2021-01-15T18:50:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2021-01-15T18:50:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:50:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:50:52.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:50:52.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:50:52.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2021-01-15T18:50:52.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:50:52.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:50:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:50:53.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2021-01-15T18:50:53.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:50:53.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:50:53.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:50:53.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:50:53.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:50:53.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:50:53.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:50:53.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:50:53.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:50:53.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:50:53.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:50:53.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:50:53.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:50:53.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:50:53.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:50:53.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:50:53.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:50:53.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:50:53.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:50:53.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:50:53.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:50:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:50:53.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:50:53.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:50:53.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:50:53.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:50:53.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:50:53.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:50:53.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:50:53.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:53.560 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:53.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:50:53.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:50:53.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:50:53.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:50:53.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:50:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:50:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:50:53.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:50:53.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:50:53.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:50:53.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:50:53.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:50:53.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:50:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:50:53.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:50:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:50:53.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:50:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:50:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:50:53.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:50:53.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:50:53.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:50:53.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:50:53.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:50:53.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:50:53.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:50:53.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:50:53.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:50:53.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:53.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:53.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:50:53.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:50:53.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:50:53.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:50:53.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:50:53.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:53.573 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:53.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:50:53.574 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:50:53.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:50:53.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:50:53.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:50:53.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2021-01-15T18:50:53.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2021-01-15T18:50:53.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2021-01-15T18:50:53.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2021-01-15T18:50:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2021-01-15T18:50:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:50:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:50:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:50:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:50:53.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:50:53.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:50:53.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:50:53.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:50:53.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:50:53.870 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:50:53.870 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:53.870 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:53.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:50:53.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:50:54.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:50:54.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:50:54.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:50:54.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:50:54.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:50:54.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:50:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:50:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:50:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:50:54.073 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:50:54.074 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:54.074 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:54.074 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:50:54.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:50:54.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:50:54.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:50:54.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:50:54.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:50:54.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:50:54.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:50:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:50:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:50:54.174 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:50:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:50:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:54.175 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:54.176 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:50:54.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:50:54.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:50:54.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2021-01-15T18:50:54.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:50:54.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:50:54.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:50:54.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:50:54.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:50:54.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:50:54.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:50:54.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:50:54.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:50:54.781 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:50:54.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:54.782 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:54.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:50:54.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:50:54.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:50:54.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2021-01-15T18:50:54.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:50:54.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:50:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:50:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:50:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:50:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:50:54.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:50:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:50:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:50:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:50:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:50:54.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:50:54.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:50:54.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:50:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:50:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:50:54.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:50:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:50:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:50:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:50:54.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:50:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:50:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:50:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:50:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:50:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:50:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:50:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:50:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:50:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:50:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:50:54.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:50:54.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:50:54.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:50:54.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c381aca6-bb0b-40f9-8a03-6514d62e7761 2021-01-15T18:50:54.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 c381aca6-bb0b-40f9-8a03-6514d62e7761' 2021-01-15T18:50:54.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:50:54.814 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 c381aca6-bb0b-40f9-8a03-6514d62e7761 2021-01-15T18:50:54.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAO5AFg0qfoMBAAhh4KFziDbq5/oqXj+Re23w== 2021-01-15T18:50:54.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAO5AFg0qfoMBAAhh4KFziDbq5/oqXj+Re23w=="}' 2021-01-15T18:50:54.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c381aca6-bb0b-40f9-8a03-6514d62e7761 -i td/osd-recovery-stats/0/new.json 2021-01-15T18:50:55.172 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:50:55.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:50:55.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAO5AFg0qfoMBAAhh4KFziDbq5/oqXj+Re23w== --osd-uuid c381aca6-bb0b-40f9-8a03-6514d62e7761 2021-01-15T18:50:55.206 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.205+0000 7f75a13b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:50:55.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.207+0000 7f75a13b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:50:55.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.208+0000 7f75a13b5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:50:55.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.208+0000 7f75a13b5080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:50:55.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.208+0000 7f75a13b5080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:50:55.209 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.208+0000 7f75a13b5080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:50:55.253 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:50:55.252+0000 7f75a13b5080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:50:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:50:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:50:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:50:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:50:59.980 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:51:00.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:51:00.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:00.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:00.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:00.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:00.506 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:51:00.527 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:00.526+0000 7f3ea99fd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:00.560 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:00.559+0000 7f3ea99fd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:00.562 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:00.561+0000 7f3ea99fd080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:00.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:51:00.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:00.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:51:00.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:00.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:00.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:00.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:00.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:00.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:51:00.925 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:01.077 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:01.076+0000 7f3ea99fd080 -1 Falling back to public interface 2021-01-15T18:51:01.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:01.845 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:01.844+0000 7f3ea99fd080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:51:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:02.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:02.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:51:02.317 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:02.751 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/882706,v1:127.0.0.1:6803/882706] [v2:127.0.0.1:6804/882706,v1:127.0.0.1:6805/882706] exists,up c381aca6-bb0b-40f9-8a03-6514d62e7761 2021-01-15T18:51:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2021-01-15T18:51:02.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:51:02.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:51:02.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:51:02.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:51:02.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:51:02.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:51:02.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:51:02.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:51:02.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:51:02.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:51:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:51:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:51:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:51:02.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:51:02.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:51:02.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:51:02.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:51:02.756 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:51:02.756 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:51:02.756 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:51:02.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:51:02.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:51:02.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:51:02.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:51:02.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:02.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:51:02.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:51:02.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:51:02.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:51:02.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dd873bbb-e3b0-48af-9dec-14eba7bd4551 2021-01-15T18:51:02.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 dd873bbb-e3b0-48af-9dec-14eba7bd4551' 2021-01-15T18:51:02.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:51:02.760 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 dd873bbb-e3b0-48af-9dec-14eba7bd4551 2021-01-15T18:51:02.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAW5AFgMaikLRAASYswBcJybkpPQckj3c3xMw== 2021-01-15T18:51:02.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAW5AFgMaikLRAASYswBcJybkpPQckj3c3xMw=="}' 2021-01-15T18:51:02.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dd873bbb-e3b0-48af-9dec-14eba7bd4551 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:51:03.279 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:03.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:51:03.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAW5AFgMaikLRAASYswBcJybkpPQckj3c3xMw== --osd-uuid dd873bbb-e3b0-48af-9dec-14eba7bd4551 2021-01-15T18:51:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.313+0000 7fc5a2fc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.315+0000 7fc5a2fc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:03.318 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.317+0000 7fc5a2fc4080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:03.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.318+0000 7fc5a2fc4080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:51:03.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.318+0000 7fc5a2fc4080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:51:03.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.318+0000 7fc5a2fc4080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:51:03.367 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:03.367+0000 7fc5a2fc4080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:51:08.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:51:08.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:51:08.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:51:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:51:08.143 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:51:08.677 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:51:08.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:51:08.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:08.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:08.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:08.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:08.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:08.697+0000 7fded53b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:08.700 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:08.699+0000 7fded53b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:08.702 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:08.701+0000 7fded53b1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:09.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:51:09.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:09.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:51:09.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:09.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:51:09.139 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:09.219 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:09.218+0000 7fded53b1080 -1 Falling back to public interface 2021-01-15T18:51:09.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:10.017 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:10.016+0000 7fded53b1080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:51:10.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:10.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:10.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:10.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:51:10.531 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:10.940 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/883970,v1:127.0.0.1:6811/883970] [v2:127.0.0.1:6812/883970,v1:127.0.0.1:6813/883970] exists,up dd873bbb-e3b0-48af-9dec-14eba7bd4551 2021-01-15T18:51:10.941 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 6de32982-ee82-4603-b8ea-363931eaa624 2021-01-15T18:51:10.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:10.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:10.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:10.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2021-01-15T18:51:10.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:51:10.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:51:10.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:51:10.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:51:10.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:51:10.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:51:10.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:51:10.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:51:10.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:51:10.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:51:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:51:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:51:10.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:51:10.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:51:10.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:51:10.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:51:10.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:51:10.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:51:10.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:51:10.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:51:10.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:10.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:10.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:51:10.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:51:10.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:51:10.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:51:10.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:51:10.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:51:10.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:51:10.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:10.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:51:10.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:51:10.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:51:10.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:51:10.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6de32982-ee82-4603-b8ea-363931eaa624 2021-01-15T18:51:10.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 6de32982-ee82-4603-b8ea-363931eaa624' 2021-01-15T18:51:10.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:51:10.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAe5AFgDtvWOBAA3UcVb8wLaJWIGVi/3V47Dw== 2021-01-15T18:51:10.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAe5AFgDtvWOBAA3UcVb8wLaJWIGVi/3V47Dw=="}' 2021-01-15T18:51:10.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6de32982-ee82-4603-b8ea-363931eaa624 -i td/osd-recovery-stats/2/new.json 2021-01-15T18:51:11.469 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:51:11.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:51:11.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAe5AFgDtvWOBAA3UcVb8wLaJWIGVi/3V47Dw== --osd-uuid 6de32982-ee82-4603-b8ea-363931eaa624 2021-01-15T18:51:11.505 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.502+0000 7f7b708a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:11.506 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.504+0000 7f7b708a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:11.507 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.505+0000 7f7b708a7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:11.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.506+0000 7f7b708a7080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:51:11.508 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.506+0000 7f7b708a7080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:51:11.509 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.506+0000 7f7b708a7080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:51:11.561 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:11.560+0000 7f7b708a7080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:51:16.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:51:16.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:51:16.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:51:16.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:51:16.118 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:51:16.660 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:51:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:51:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:16.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:16.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:16.682 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:16.681+0000 7ff377b69080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:16.716 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:16.715+0000 7ff377b69080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:16.716+0000 7ff377b69080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:51:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:51:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:17.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:51:17.063 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:17.231 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:17.230+0000 7ff377b69080 -1 Falling back to public interface 2021-01-15T18:51:17.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:18.033 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:18.032+0000 7ff377b69080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:51:18.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:18.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:18.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:18.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:18.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:51:18.456 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:18.855 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/885234,v1:127.0.0.1:6819/885234] [v2:127.0.0.1:6820/885234,v1:127.0.0.1:6821/885234] exists,up 6de32982-ee82-4603-b8ea-363931eaa624 2021-01-15T18:51:18.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:18.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2021-01-15T18:51:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:51:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:51:18.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:51:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:51:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:51:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:51:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:51:18.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:51:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:51:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:51:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:51:18.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:51:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:51:18.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:51:18.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:51:18.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:51:18.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:51:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:51:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:51:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:51:18.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:18.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:51:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:51:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:51:18.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:51:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:51:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:51:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:51:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:18.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:51:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:51:18.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:51:18.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:51:18.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=eb4372f9-6456-4420-a20b-704367d1ee9d 2021-01-15T18:51:18.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 eb4372f9-6456-4420-a20b-704367d1ee9d' 2021-01-15T18:51:18.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:51:18.865 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 eb4372f9-6456-4420-a20b-704367d1ee9d 2021-01-15T18:51:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAm5AFgepzPMxAASkeGRue+cel/ffPYlSWsaw== 2021-01-15T18:51:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAm5AFgepzPMxAASkeGRue+cel/ffPYlSWsaw=="}' 2021-01-15T18:51:18.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new eb4372f9-6456-4420-a20b-704367d1ee9d -i td/osd-recovery-stats/3/new.json 2021-01-15T18:51:19.411 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:51:19.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:51:19.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAm5AFgepzPMxAASkeGRue+cel/ffPYlSWsaw== --osd-uuid eb4372f9-6456-4420-a20b-704367d1ee9d 2021-01-15T18:51:19.446 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.445+0000 7f75959b7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:19.447 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.447+0000 7f75959b7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:19.449 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.448+0000 7f75959b7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:19.450 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.448+0000 7f75959b7080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:51:19.450 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.448+0000 7f75959b7080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:51:19.450 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.448+0000 7f75959b7080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:51:19.506 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:19.505+0000 7f75959b7080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:51:24.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:51:24.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:51:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:51:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:51:24.293 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:51:24.833 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:51:24.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:51:24.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:24.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:24.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:24.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:24.854 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:24.853+0000 7f8451a76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:24.873 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:24.873+0000 7f8451a76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:24.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:24.874+0000 7f8451a76080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:25.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:51:25.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:25.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:51:25.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:25.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:25.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:25.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:51:25.224 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:25.391 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:25.390+0000 7f8451a76080 -1 Falling back to public interface 2021-01-15T18:51:25.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:26.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:26.174+0000 7f8451a76080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:51:26.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:26.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:26.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:26.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:26.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:51:26.634 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:27.163 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/886498,v1:127.0.0.1:6827/886498] [v2:127.0.0.1:6828/886498,v1:127.0.0.1:6829/886498] exists,up eb4372f9-6456-4420-a20b-704367d1ee9d 2021-01-15T18:51:27.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2021-01-15T18:51:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:51:27.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:51:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:51:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:51:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/4 2021-01-15T18:51:27.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:51:27.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-01-15T18:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-01-15T18:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:51:27.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:51:27.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:51:27.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:51:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:51:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:51:27.168 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:51:27.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:51:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:51:27.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:51:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:51:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:51:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:51:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:51:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:51:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:51:27.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/4 2021-01-15T18:51:27.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:51:27.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=af00c34e-21e3-4a10-85c4-ca267668bee4 2021-01-15T18:51:27.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 af00c34e-21e3-4a10-85c4-ca267668bee4' 2021-01-15T18:51:27.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:51:27.173 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 af00c34e-21e3-4a10-85c4-ca267668bee4 2021-01-15T18:51:27.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAv5AFgrniTChAA8NikPNxBiO39LPpLYqQbpQ== 2021-01-15T18:51:27.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAv5AFgrniTChAA8NikPNxBiO39LPpLYqQbpQ=="}' 2021-01-15T18:51:27.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new af00c34e-21e3-4a10-85c4-ca267668bee4 -i td/osd-recovery-stats/4/new.json 2021-01-15T18:51:27.668 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:51:27.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/4/new.json 2021-01-15T18:51:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAv5AFgrniTChAA8NikPNxBiO39LPpLYqQbpQ== --osd-uuid af00c34e-21e3-4a10-85c4-ca267668bee4 2021-01-15T18:51:27.704 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.703+0000 7fd69e7d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:27.706 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.705+0000 7fd69e7d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:27.707 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.706+0000 7fd69e7d0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:27.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.707+0000 7fd69e7d0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:51:27.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.707+0000 7fd69e7d0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:51:27.709 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.707+0000 7fd69e7d0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:51:27.749 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:27.747+0000 7fd69e7d0080 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-01-15T18:51:32.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-01-15T18:51:32.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:51:32.785 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:51:32.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:51:32.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:51:33.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:51:33.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:33.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:33.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:33.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:33.315 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:51:33.335 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:33.334+0000 7fab74084080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:33.359 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:33.357+0000 7fab74084080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:33.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:33.359+0000 7fab74084080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:33.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:51:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:51:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:51:33.707 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:33.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:33.874+0000 7fab74084080 -1 Falling back to public interface 2021-01-15T18:51:34.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:34.744 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:34.743+0000 7fab74084080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:51:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:51:35.109 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:35.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:36.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:36.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:36.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:51:36.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:36.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:51:36.609 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:51:37.018 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/887762,v1:127.0.0.1:6835/887762] [v2:127.0.0.1:6836/887762,v1:127.0.0.1:6837/887762] exists,up af00c34e-21e3-4a10-85c4-ca267668bee4 2021-01-15T18:51:37.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:37.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2021-01-15T18:51:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:51:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:51:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:51:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:51:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/5 2021-01-15T18:51:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:51:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:51:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:51:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:51:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-01-15T18:51:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-01-15T18:51:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:51:37.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:51:37.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:51:37.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:51:37.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:51:37.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:51:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:51:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:51:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:51:37.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:51:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:51:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:51:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:51:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:51:37.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:51:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:51:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:51:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:51:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:51:37.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/5 2021-01-15T18:51:37.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:51:37.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6801e148-eb40-4bd0-b705-cbf607312f83 2021-01-15T18:51:37.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 6801e148-eb40-4bd0-b705-cbf607312f83' 2021-01-15T18:51:37.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:51:37.027 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 6801e148-eb40-4bd0-b705-cbf607312f83 2021-01-15T18:51:37.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA55AFgXGvuARAAzymEXNsIW4ee9EdM+g84kQ== 2021-01-15T18:51:37.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA55AFgXGvuARAAzymEXNsIW4ee9EdM+g84kQ=="}' 2021-01-15T18:51:37.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6801e148-eb40-4bd0-b705-cbf607312f83 -i td/osd-recovery-stats/5/new.json 2021-01-15T18:51:37.656 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:51:37.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/5/new.json 2021-01-15T18:51:37.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA55AFgXGvuARAAzymEXNsIW4ee9EdM+g84kQ== --osd-uuid 6801e148-eb40-4bd0-b705-cbf607312f83 2021-01-15T18:51:37.691 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.690+0000 7fbfd29a5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:37.693 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.692+0000 7fbfd29a5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.693+0000 7fbfd29a5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.694+0000 7fbfd29a5080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:51:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.694+0000 7fbfd29a5080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:51:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.694+0000 7fbfd29a5080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:51:37.829 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:37.828+0000 7fbfd29a5080 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-01-15T18:51:42.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-01-15T18:51:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:51:42.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:51:42.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:51:42.557 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:51:43.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:51:43.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:51:43.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:51:43.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:51:43.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:51:43.088 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:51:43.108 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:43.107+0000 7f252f5ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:43.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:43.109+0000 7f252f5ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:43.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:43.110+0000 7f252f5ae080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:51:43.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:51:43.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:51:43.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:51:43.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:51:43.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:51:43.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:43.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:51:43.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:43.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:51:43.511 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:51:43.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:43.625+0000 7f252f5ae080 -1 Falling back to public interface 2021-01-15T18:51:43.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:51:44.523 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:51:44.522+0000 7f252f5ae080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:51:44.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:51:44.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:51:44.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:51:44.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:51:44.920 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:51:44.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:51:45.341 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/889060,v1:127.0.0.1:6843/889060] [v2:127.0.0.1:6844/889060,v1:127.0.0.1:6845/889060] exists,up 6801e148-eb40-4bd0-b705-cbf607312f83 2021-01-15T18:51:45.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:51:45.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:51:45.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:51:45.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2021-01-15T18:51:45.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2021-01-15T18:51:45.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 replicated 2021-01-15T18:51:46.144 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:51:46.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:51:47.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2021-01-15T18:51:47.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:51:47.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:51:47.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:51:47.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:51:47.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:51:47.157 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:51:47.157 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:51:47.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:51:47.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:51:47.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:51:47.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:51:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:51:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:51:47.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:51:47.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:51:47.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:51:47.651 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:51:47.651 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:51:47.651 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:51:47.652 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:51:47.652 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:51:47.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:51:47.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:47.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:51:47.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T18:51:47.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T18:51:47.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T18:51:47.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:47.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:51:47.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T18:51:47.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T18:51:47.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2021-01-15T18:51:47.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:47.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:51:48.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T18:51:48.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T18:51:48.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448' 2021-01-15T18:51:48.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:48.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:51:48.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T18:51:48.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T18:51:48.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927' 2021-01-15T18:51:48.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:48.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:51:48.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T18:51:48.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T18:51:48.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405' 2021-01-15T18:51:48.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:51:48.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:51:48.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T18:51:48.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T18:51:48.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509448 3-85899345927 4-107374182405 5-128849018884' 2021-01-15T18:51:48.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:48.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T18:51:48.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:48.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:51:48.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T18:51:48.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:48.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T18:51:48.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T18:51:48.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:51:48.468 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T18:51:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836491 2021-01-15T18:51:48.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:48.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T18:51:48.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:48.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:51:48.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T18:51:48.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:48.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T18:51:48.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T18:51:48.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:51:48.858 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T18:51:49.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T18:51:49.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:49.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T18:51:49.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:49.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:51:49.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T18:51:49.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:49.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T18:51:49.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T18:51:49.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:51:49.265 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T18:51:49.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T18:51:49.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:49.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T18:51:49.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:51:49.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T18:51:49.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:49.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T18:51:49.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T18:51:49.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:51:49.667 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T18:51:50.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345927 2021-01-15T18:51:50.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:50.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T18:51:50.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:51:50.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T18:51:50.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:50.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T18:51:50.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T18:51:50.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:51:50.178 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T18:51:50.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T18:51:50.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:51:50.488 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T18:51:50.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:51:50.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:51:50.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T18:51:50.489 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:51:50.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T18:51:50.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T18:51:50.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:51:50.490 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T18:51:50.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018884 -lt 128849018884 2021-01-15T18:51:50.887 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:51:50.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:51:50.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:51:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:51:51.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:51:51.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:51:51.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:51:51.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:51:51.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:51:51.430 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:51:51.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:51:51.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:51:51.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:51:51.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:51:51.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:51:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:51:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:51:52.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:51:52.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2021-01-15T18:51:52.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2021-01-15T18:51:52.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2021-01-15T18:51:52.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2021-01-15T18:51:52.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2021-01-15T18:51:52.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2021-01-15T18:51:52.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:52.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2021-01-15T18:51:53.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2021-01-15T18:51:53.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2021-01-15T18:51:53.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2021-01-15T18:51:53.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2021-01-15T18:51:53.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2021-01-15T18:51:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2021-01-15T18:51:53.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2021-01-15T18:51:53.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:53.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2021-01-15T18:51:54.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2021-01-15T18:51:54.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2021-01-15T18:51:54.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2021-01-15T18:51:54.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2021-01-15T18:51:54.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2021-01-15T18:51:54.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2021-01-15T18:51:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2021-01-15T18:51:54.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:54.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2021-01-15T18:51:55.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2021-01-15T18:51:55.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2021-01-15T18:51:55.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2021-01-15T18:51:55.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2021-01-15T18:51:55.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2021-01-15T18:51:55.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2021-01-15T18:51:55.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2021-01-15T18:51:55.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:55.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2021-01-15T18:51:56.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2021-01-15T18:51:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2021-01-15T18:51:56.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2021-01-15T18:51:56.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2021-01-15T18:51:56.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2021-01-15T18:51:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2021-01-15T18:51:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2021-01-15T18:51:56.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:56.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2021-01-15T18:51:57.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2021-01-15T18:51:57.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2021-01-15T18:51:57.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2021-01-15T18:51:57.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2021-01-15T18:51:57.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2021-01-15T18:51:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2021-01-15T18:51:57.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2021-01-15T18:51:57.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2021-01-15T18:51:57.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:57.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2021-01-15T18:51:58.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2021-01-15T18:51:58.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2021-01-15T18:51:58.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2021-01-15T18:51:58.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2021-01-15T18:51:58.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2021-01-15T18:51:58.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2021-01-15T18:51:58.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2021-01-15T18:51:58.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:58.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2021-01-15T18:51:59.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2021-01-15T18:51:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2021-01-15T18:51:59.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2021-01-15T18:51:59.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2021-01-15T18:51:59.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2021-01-15T18:51:59.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2021-01-15T18:51:59.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2021-01-15T18:51:59.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2021-01-15T18:51:59.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:51:59.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2021-01-15T18:52:00.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2021-01-15T18:52:00.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2021-01-15T18:52:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2021-01-15T18:52:00.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2021-01-15T18:52:00.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2021-01-15T18:52:00.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2021-01-15T18:52:00.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2021-01-15T18:52:00.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:00.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2021-01-15T18:52:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2021-01-15T18:52:01.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2021-01-15T18:52:01.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2021-01-15T18:52:01.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2021-01-15T18:52:01.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2021-01-15T18:52:01.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2021-01-15T18:52:01.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2021-01-15T18:52:01.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:01.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2021-01-15T18:52:02.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2021-01-15T18:52:02.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2021-01-15T18:52:02.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2021-01-15T18:52:02.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2021-01-15T18:52:02.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2021-01-15T18:52:02.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2021-01-15T18:52:02.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2021-01-15T18:52:02.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2021-01-15T18:52:02.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:02.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2021-01-15T18:52:03.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2021-01-15T18:52:03.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2021-01-15T18:52:03.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2021-01-15T18:52:03.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2021-01-15T18:52:03.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2021-01-15T18:52:03.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2021-01-15T18:52:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2021-01-15T18:52:03.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:03.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2021-01-15T18:52:04.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2021-01-15T18:52:04.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2021-01-15T18:52:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2021-01-15T18:52:04.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2021-01-15T18:52:04.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2021-01-15T18:52:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2021-01-15T18:52:04.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2021-01-15T18:52:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2021-01-15T18:52:05.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2021-01-15T18:52:05.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2021-01-15T18:52:05.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2021-01-15T18:52:05.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2021-01-15T18:52:05.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2021-01-15T18:52:05.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2021-01-15T18:52:05.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2021-01-15T18:52:05.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:05.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2021-01-15T18:52:06.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2021-01-15T18:52:06.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2021-01-15T18:52:06.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2021-01-15T18:52:06.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2021-01-15T18:52:06.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2021-01-15T18:52:06.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2021-01-15T18:52:06.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2021-01-15T18:52:06.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:06.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2021-01-15T18:52:07.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2021-01-15T18:52:07.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2021-01-15T18:52:07.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2021-01-15T18:52:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2021-01-15T18:52:07.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2021-01-15T18:52:07.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2021-01-15T18:52:07.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2021-01-15T18:52:07.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2021-01-15T18:52:07.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:07.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2021-01-15T18:52:08.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2021-01-15T18:52:08.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2021-01-15T18:52:08.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2021-01-15T18:52:08.422 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2021-01-15T18:52:08.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2021-01-15T18:52:08.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2021-01-15T18:52:08.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2021-01-15T18:52:08.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:08.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2021-01-15T18:52:09.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2021-01-15T18:52:09.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2021-01-15T18:52:09.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2021-01-15T18:52:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2021-01-15T18:52:09.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2021-01-15T18:52:09.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2021-01-15T18:52:09.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2021-01-15T18:52:09.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:09.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2021-01-15T18:52:10.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2021-01-15T18:52:10.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2021-01-15T18:52:10.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2021-01-15T18:52:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2021-01-15T18:52:10.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2021-01-15T18:52:10.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2021-01-15T18:52:10.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:10.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2021-01-15T18:52:11.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2021-01-15T18:52:11.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2021-01-15T18:52:11.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2021-01-15T18:52:11.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2021-01-15T18:52:11.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2021-01-15T18:52:11.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2021-01-15T18:52:11.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2021-01-15T18:52:11.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2021-01-15T18:52:11.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:11.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2021-01-15T18:52:12.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2021-01-15T18:52:12.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2021-01-15T18:52:12.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2021-01-15T18:52:12.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2021-01-15T18:52:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2021-01-15T18:52:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2021-01-15T18:52:12.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2021-01-15T18:52:12.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:12.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2021-01-15T18:52:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2021-01-15T18:52:13.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2021-01-15T18:52:13.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2021-01-15T18:52:13.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2021-01-15T18:52:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2021-01-15T18:52:13.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2021-01-15T18:52:13.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2021-01-15T18:52:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2021-01-15T18:52:13.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:13.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2021-01-15T18:52:14.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2021-01-15T18:52:14.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2021-01-15T18:52:14.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2021-01-15T18:52:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2021-01-15T18:52:14.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2021-01-15T18:52:14.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2021-01-15T18:52:14.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2021-01-15T18:52:14.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:14.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2021-01-15T18:52:15.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2021-01-15T18:52:15.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2021-01-15T18:52:15.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2021-01-15T18:52:15.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2021-01-15T18:52:15.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2021-01-15T18:52:15.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2021-01-15T18:52:15.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2021-01-15T18:52:15.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2021-01-15T18:52:15.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:15.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2021-01-15T18:52:16.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:16.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2021-01-15T18:52:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2021-01-15T18:52:16.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:16.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2021-01-15T18:52:16.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2021-01-15T18:52:16.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2021-01-15T18:52:16.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2021-01-15T18:52:16.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:52:16.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:52:16.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:52:16.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:52:16.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2021-01-15T18:52:16.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2021-01-15T18:52:16.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:52:16.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:52:16.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:52:16.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:52:17.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2021-01-15T18:52:17.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2021-01-15T18:52:17.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:52:17.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:52:17.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:52:17.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:52:17.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:52:17.358 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:52:17.358 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:52:17.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:52:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:52:17.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:52:18.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2021-01-15T18:52:18.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2021-01-15T18:52:18.880 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:52:18.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2021-01-15T18:52:18.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 882706 2021-01-15T18:52:18.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2021-01-15T18:52:19.414 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T18:52:19.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2021-01-15T18:52:20.034 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:52:20.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2021-01-15T18:52:20.771 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:52:20.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2021-01-15T18:52:20.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:52:20.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:52:20.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:52:20.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:52:21.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T18:52:21.339 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:52:21.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2021-01-15T18:52:23.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2021-01-15T18:52:23.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:52:23.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:52:23.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:52:23.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:52:23.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:52:23.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:52:23.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:52:23.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:52:23.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:52:23.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:52:23.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:52:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:52:23.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:52:23.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:52:23.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:52:23.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:52:23.848 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:52:23.848 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:52:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:52:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:52:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:52:23.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:52:23.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:23.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:52:23.943 INFO:tasks.workunit.client.0.smithi145.stderr:Error ENXIO: problem getting command descriptions from osd.0 2021-01-15T18:52:23.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq= 2021-01-15T18:52:23.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z '' 2021-01-15T18:52:23.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: continue 2021-01-15T18:52:23.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:23.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:52:24.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T18:52:24.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T18:52:24.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979' 2021-01-15T18:52:24.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:24.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:52:24.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T18:52:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T18:52:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509457' 2021-01-15T18:52:24.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:24.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:52:24.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345936 2021-01-15T18:52:24.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345936 2021-01-15T18:52:24.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509457 3-85899345936' 2021-01-15T18:52:24.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:24.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:52:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T18:52:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T18:52:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509457 3-85899345936 4-107374182414' 2021-01-15T18:52:24.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:52:24.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:52:24.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018893 2021-01-15T18:52:24.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018893 2021-01-15T18:52:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 1-42949672979 2-64424509457 3-85899345936 4-107374182414 5-128849018893' 2021-01-15T18:52:24.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:52:24.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T18:52:24.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:52:24.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:52:24.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T18:52:24.627 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:52:24.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T18:52:24.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T18:52:24.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:52:24.628 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T18:52:25.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672979 -lt 42949672979 2021-01-15T18:52:25.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:52:25.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T18:52:25.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:52:25.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:52:25.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T18:52:25.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:52:25.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T18:52:25.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T18:52:25.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:52:25.027 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T18:52:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509458 -lt 64424509457 2021-01-15T18:52:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:52:25.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345936 2021-01-15T18:52:25.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:52:25.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:52:25.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345936 2021-01-15T18:52:25.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:52:25.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345936 2021-01-15T18:52:25.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-01-15T18:52:25.422 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345936 2021-01-15T18:52:25.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:52:25.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345936 2021-01-15T18:52:25.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:52:25.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T18:52:25.990 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:52:25.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:52:25.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T18:52:25.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:52:25.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T18:52:25.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T18:52:25.994 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:52:25.994 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T18:52:26.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2021-01-15T18:52:26.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:52:26.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018893 2021-01-15T18:52:26.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:52:26.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:52:26.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018893 2021-01-15T18:52:26.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:52:26.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018893 2021-01-15T18:52:26.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-01-15T18:52:26.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:52:26.384 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018893 2021-01-15T18:52:26.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018893 2021-01-15T18:52:26.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:52:26.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:26.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:27.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:52:27.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:27.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:27.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:27.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:27.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:27.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:27.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:27.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:27.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:27.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:27.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:52:28.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:52:28.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:52:28.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:28.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:28.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:28.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:28.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:28.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:28.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:28.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:28.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:28.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:28.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:28.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:28.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:28.680 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:29.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:29.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:29.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:29.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:29.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:29.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:29.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:52:29.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:52:29.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:29.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:29.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:29.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:29.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:29.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:29.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:29.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:29.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:29.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:29.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:30.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:30.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:30.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:30.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:30.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:30.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:30.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:30.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:30.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:31.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:52:31.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:52:31.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:31.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:31.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:31.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:31.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:31.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:31.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:31.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:31.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:31.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:31.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:31.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:31.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:31.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:31.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:32.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:32.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:32.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:32.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:32.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:32.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:32.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:32.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:32.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:32.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:32.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:32.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:32.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:32.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:32.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:32.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:32.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:32.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:32.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:32.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:32.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:32.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:33.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:33.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:33.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:33.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:33.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:33.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:34.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:34.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:34.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:34.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:34.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:34.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:34.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:34.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:34.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:34.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:34.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:34.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:34.750 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:34.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:34.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:35.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:35.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:35.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:35.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:35.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:35.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:35.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:35.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:35.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:35.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:52:35.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:52:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:35.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:35.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:35.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:35.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:35.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:35.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:35.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:35.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:35.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:36.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:36.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:36.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:36.263 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:36.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:36.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:36.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:36.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:36.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:36.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:37.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:52:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:52:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:37.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:37.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:37.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:37.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:37.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:37.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:37.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:37.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:37.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:37.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:37.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:38.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:38.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:38.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:38.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:38.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:38.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:38.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:38.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:38.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:38.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:38.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:38.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:38.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:38.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:38.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:39.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:39.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:39.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:39.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:39.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:39.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:39.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:39.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:39.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:39.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:39.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:39.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:39.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:40.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:40.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:40.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:40.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:40.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:40.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:40.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:40.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:40.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:40.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:40.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:40.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:40.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:40.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:40.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:41.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:41.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:41.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:41.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:41.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:41.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:41.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:41.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:41.366 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:41.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:41.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:41.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:41.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:41.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:41.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:41.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:41.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:41.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:41.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:41.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:41.982 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:41.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:42.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:42.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:42.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:42.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:42.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:42.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:42.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:42.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:42.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:43.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:52:43.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:52:43.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:43.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:43.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:43.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:43.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:43.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:43.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:43.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:43.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:43.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:43.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:43.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:43.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:43.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:43.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:44.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:44.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:44.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:44.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:44.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:44.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:45.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:45.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:45.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:45.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:45.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:45.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:45.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:45.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:45.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:45.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:45.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:45.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:45.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:45.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:45.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:45.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:45.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:45.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:45.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:45.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:45.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:46.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:46.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:46.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:46.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:46.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:46.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:46.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:46.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:46.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:46.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:46.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:46.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:46.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:46.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:46.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:46.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:46.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:47.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:47.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:47.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:47.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:47.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:47.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:47.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:47.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:47.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:47.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:52:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:52:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:47.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:48.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:48.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:48.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:48.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:48.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:48.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:48.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:48.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:48.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:48.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:48.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:48.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:48.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:49.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:49.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:49.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:49.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:49.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:49.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:49.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:49.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:49.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:49.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:49.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:49.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:49.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:49.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:49.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:49.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:49.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:49.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:49.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:49.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:49.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:49.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:49.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:49.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:49.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:50.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:50.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:50.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:50.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:50.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:51.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:51.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:51.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:51.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:51.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:51.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:51.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:51.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:51.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:51.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:51.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:51.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:51.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:51.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:51.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:51.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:51.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:51.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:51.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:52.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=5 2021-01-15T18:52:52.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 5 '!=' null 2021-01-15T18:52:52.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:52.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:52.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:52.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:52.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:52.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:52.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:52.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:52.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:52.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:52.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:52.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:52.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:52.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:52.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:53.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:53.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:53.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:53.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:54.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=8 2021-01-15T18:52:54.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 8 '!=' null 2021-01-15T18:52:54.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:54.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:54.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:54.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:54.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:54.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:54.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:54.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:54.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:54.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:54.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:54.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:54.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:54.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:54.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:55.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:55.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:55.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:55.089 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:55.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:52:55.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:52:55.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:55.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:55.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:55.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:55.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:55.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:55.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:55.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:55.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:55.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:55.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:56.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:52:56.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:56.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:56.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:56.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:52:56.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:52:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:52:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:52:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:52:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:52:56.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:52:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:52:56.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:52:57.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:52:57.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:52:57.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:52:57.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:52:57.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:52:57.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:52:57.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:52:57.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:52:57.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:52:57.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:52:57.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:52:57.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:52:57.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:52:57.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:52:57.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:52:57.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:52:57.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:52:58.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:52:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:52:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:52:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2021-01-15T18:52:58.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:52:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:52:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-01-15T18:52:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-01-15T18:52:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2021-01-15T18:52:58.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:52:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2021-01-15T18:52:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:52:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-01-15T18:52:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-01-15T18:52:58.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:52:58.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-01-15T18:52:58.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:52:58.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.121 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:52:58.121 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:52:58.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2021-01-15T18:52:58.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:52:58.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:52:58.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2021-01-15T18:52:58.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:52:58.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:52:58.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:52:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2021-01-15T18:52:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2021-01-15T18:52:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:52:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:52:58.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:52:58.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:52:58.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:52:58.136 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.136 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:52:58.137 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:52:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:52:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:52:58.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:52:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:52:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:52:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:52:58.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:52:58.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:52:58.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2021-01-15T18:52:58.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:52:58.145 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:52:58.146 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:52:58.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2021-01-15T18:52:58.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2021-01-15T18:52:58.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-01-15T18:52:58.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:52:58.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-01-15T18:52:58.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:52:58.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:52:58.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:52:58.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:52:58.150 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:52:58.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:52:58.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:52:58.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:52:58.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:52:58.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:52:58.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:52:58.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-01-15T18:52:58.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2021-01-15T18:52:58.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:52:58.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:52:58.694 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:52:58.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2021-01-15T18:52:58.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:52:58.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:52:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:52:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:52:58.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:52:59.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:52:59.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:52:59.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:52:59.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:52:59.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:52:59.013 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:52:59.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:52:59.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:52:59.014 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:52:59.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:52:59.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:52:59.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:52:59.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:52:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:52:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:52:59.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:52:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:52:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:52:59.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:52:59.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:52:59.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:52:59.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:52:59.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:52:59.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:52:59.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:52:59.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:52:59.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:52:59.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:52:59.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:52:59.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:52:59.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:52:59.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:52:59.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:52:59.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:52:59.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:52:59.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:52:59.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:52:59.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:52:59.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:52:59.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:52:59.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:52:59.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:52:59.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:52:59.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:52:59.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:52:59.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:52:59.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:52:59.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:52:59.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:52:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:52:59.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:52:59.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:52:59.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:52:59.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:52:59.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:52:59.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:52:59.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:52:59.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:52:59.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:52:59.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:52:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:52:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:52:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2021-01-15T18:52:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2021-01-15T18:52:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:52:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:52:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:52:59.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:52:59.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:52:59.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:52:59.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:52:59.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:52:59.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:52:59.353 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.353 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:52:59.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:52:59.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:52:59.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:52:59.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:52:59.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:52:59.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:52:59.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:52:59.538 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:52:59.538 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:52:59.538 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:52:59.539 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:52:59.539 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.540 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.540 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:52:59.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:52:59.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:52:59.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:52:59.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:52:59.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:52:59.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:52:59.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:52:59.640 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:52:59.640 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:52:59.641 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:52:59.642 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:52:59.642 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:52:59.642 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:52:59.642 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:52:59.642 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:52:59.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:52:59.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2021-01-15T18:52:59.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:52:59.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:52:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:52:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:52:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:52:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:53:00.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:53:00.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:00.639 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:00.640 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:00.640 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:00.640 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:00.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:00.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:53:00.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:53:00.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2021-01-15T18:53:00.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:00.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:00.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:00.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:00.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:53:00.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:53:00.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:00.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:00.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:00.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:00.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:00.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:00.667 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:00.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:00.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:00.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:00.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:53:00.671 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:00.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d9cfc2e1-1850-470c-8ecd-17f60d1c99c8 2021-01-15T18:53:00.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d9cfc2e1-1850-470c-8ecd-17f60d1c99c8' 2021-01-15T18:53:00.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:00.672 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d9cfc2e1-1850-470c-8ecd-17f60d1c99c8 2021-01-15T18:53:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCM5AFgMs1zKBAAlddGc0hwoRxblhS4DB6oWQ== 2021-01-15T18:53:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCM5AFgMs1zKBAAlddGc0hwoRxblhS4DB6oWQ=="}' 2021-01-15T18:53:00.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d9cfc2e1-1850-470c-8ecd-17f60d1c99c8 -i td/osd-recovery-stats/0/new.json 2021-01-15T18:53:01.021 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:01.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:53:01.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCM5AFgMs1zKBAAlddGc0hwoRxblhS4DB6oWQ== --osd-uuid d9cfc2e1-1850-470c-8ecd-17f60d1c99c8 2021-01-15T18:53:01.056 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.055+0000 7f620e8b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:01.057 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.056+0000 7f620e8b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:01.058 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.057+0000 7f620e8b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.058+0000 7f620e8b9080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:53:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.058+0000 7f620e8b9080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:53:01.059 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.058+0000 7f620e8b9080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:53:01.094 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:01.092+0000 7f620e8b9080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:53:05.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:53:05.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:05.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:53:05.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:05.778 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:53:06.306 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:53:06.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:53:06.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:06.327 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:06.326+0000 7f32868b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:06.329 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:06.328+0000 7f32868b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:06.330 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:06.329+0000 7f32868b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:06.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:53:06.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:06.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:53:06.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:06.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:06.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:06.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:06.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:06.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:53:06.720 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:06.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:06.849+0000 7f32868b2080 -1 Falling back to public interface 2021-01-15T18:53:07.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:07.644 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:07.643+0000 7f32868b2080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:53:08.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:08.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:08.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:08.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:08.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:53:08.110 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:08.520 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/897563,v1:127.0.0.1:6803/897563] [v2:127.0.0.1:6804/897563,v1:127.0.0.1:6805/897563] exists,up d9cfc2e1-1850-470c-8ecd-17f60d1c99c8 2021-01-15T18:53:08.521 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 282c11f3-ba66-4c4c-9e68-bc923faf4441 2021-01-15T18:53:08.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:08.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:08.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:08.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2021-01-15T18:53:08.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:08.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:08.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:53:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:53:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:08.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:08.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:08.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:53:08.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:53:08.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:08.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:08.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:08.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:08.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:08.525 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:08.526 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:08.526 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:08.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:08.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:08.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:08.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:08.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:08.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:08.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:08.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:08.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:08.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:08.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:08.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:08.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:53:08.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:08.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=282c11f3-ba66-4c4c-9e68-bc923faf4441 2021-01-15T18:53:08.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 282c11f3-ba66-4c4c-9e68-bc923faf4441' 2021-01-15T18:53:08.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:08.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCU5AFgy27XHxAAnQJTGu+vMl8ZXvzGergNTw== 2021-01-15T18:53:08.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCU5AFgy27XHxAAnQJTGu+vMl8ZXvzGergNTw=="}' 2021-01-15T18:53:08.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 282c11f3-ba66-4c4c-9e68-bc923faf4441 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:53:09.066 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:09.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:53:09.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCU5AFgy27XHxAAnQJTGu+vMl8ZXvzGergNTw== --osd-uuid 282c11f3-ba66-4c4c-9e68-bc923faf4441 2021-01-15T18:53:09.101 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.100+0000 7f48e40c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:09.102 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.102+0000 7f48e40c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.103+0000 7f48e40c3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.104+0000 7f48e40c3080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:53:09.105 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.104+0000 7f48e40c3080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:53:09.105 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.104+0000 7f48e40c3080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:53:09.166 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:09.165+0000 7f48e40c3080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:53:13.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:53:13.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:13.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:53:13.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:13.967 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:53:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:53:14.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:14.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:14.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:14.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:14.490 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:53:14.510 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:14.509+0000 7f55ac5de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:14.531 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:14.530+0000 7f55ac5de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:14.531 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:14.531+0000 7f55ac5de080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:14.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:53:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:53:14.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:14.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:14.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:14.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:14.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:14.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:53:14.915 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:15.048 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:15.047+0000 7f55ac5de080 -1 Falling back to public interface 2021-01-15T18:53:15.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:15.850 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:15.849+0000 7f55ac5de080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:53:16.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:16.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:16.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:53:16.305 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:16.715 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/898827,v1:127.0.0.1:6811/898827] [v2:127.0.0.1:6812/898827,v1:127.0.0.1:6813/898827] exists,up 282c11f3-ba66-4c4c-9e68-bc923faf4441 2021-01-15T18:53:16.716 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 68eee4b3-7045-4b57-969b-91e2ede3a2f2 2021-01-15T18:53:16.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:16.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:16.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2021-01-15T18:53:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:53:16.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:53:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:16.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:16.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:16.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:16.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:16.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:16.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:16.720 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:16.721 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:16.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:16.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:16.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:16.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:16.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:16.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:16.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:53:16.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:16.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=68eee4b3-7045-4b57-969b-91e2ede3a2f2 2021-01-15T18:53:16.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 68eee4b3-7045-4b57-969b-91e2ede3a2f2' 2021-01-15T18:53:16.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:16.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCc5AFgFc9yKxAARXSuNY5cHwElPhfzL+9EMQ== 2021-01-15T18:53:16.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCc5AFgFc9yKxAARXSuNY5cHwElPhfzL+9EMQ=="}' 2021-01-15T18:53:16.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 68eee4b3-7045-4b57-969b-91e2ede3a2f2 -i td/osd-recovery-stats/2/new.json 2021-01-15T18:53:17.224 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:53:17.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:53:17.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCc5AFgFc9yKxAARXSuNY5cHwElPhfzL+9EMQ== --osd-uuid 68eee4b3-7045-4b57-969b-91e2ede3a2f2 2021-01-15T18:53:17.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.259+0000 7f6aeabb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:17.261 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.261+0000 7f6aeabb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:17.263 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.262+0000 7f6aeabb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:17.264 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.263+0000 7f6aeabb8080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:53:17.264 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.263+0000 7f6aeabb8080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:53:17.264 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.263+0000 7f6aeabb8080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:53:17.313 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:17.312+0000 7f6aeabb8080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:53:21.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:53:21.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:53:21.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:21.890 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:53:22.415 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:53:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:53:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:22.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:22.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:22.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:22.436 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:22.435+0000 7faef7710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:22.463 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:22.462+0000 7faef7710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:22.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:22.464+0000 7faef7710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:22.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:53:22.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:22.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:53:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:22.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:22.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:53:22.813 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:22.979 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:22.978+0000 7faef7710080 -1 Falling back to public interface 2021-01-15T18:53:23.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:23.831 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:23.829+0000 7faef7710080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:53:24.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:24.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:24.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:24.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:24.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:53:24.204 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:24.607 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/900091,v1:127.0.0.1:6819/900091] [v2:127.0.0.1:6820/900091,v1:127.0.0.1:6821/900091] exists,up 68eee4b3-7045-4b57-969b-91e2ede3a2f2 2021-01-15T18:53:24.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:24.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:24.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2021-01-15T18:53:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:53:24.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:24.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:53:24.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:24.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:24.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:24.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:24.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:53:24.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:53:24.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:24.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:24.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:24.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:24.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:24.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:24.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:24.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:24.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:24.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:24.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:24.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:24.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:53:24.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:24.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0c0e555a-d1e4-4e97-b64b-73f533e29c2c 2021-01-15T18:53:24.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 0c0e555a-d1e4-4e97-b64b-73f533e29c2c' 2021-01-15T18:53:24.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:24.616 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 0c0e555a-d1e4-4e97-b64b-73f533e29c2c 2021-01-15T18:53:24.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCk5AFgLgEBJRAANinEVMYzRYdip6SJdPqSiw== 2021-01-15T18:53:24.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCk5AFgLgEBJRAANinEVMYzRYdip6SJdPqSiw=="}' 2021-01-15T18:53:24.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0c0e555a-d1e4-4e97-b64b-73f533e29c2c -i td/osd-recovery-stats/3/new.json 2021-01-15T18:53:25.251 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:53:25.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:53:25.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCk5AFgLgEBJRAANinEVMYzRYdip6SJdPqSiw== --osd-uuid 0c0e555a-d1e4-4e97-b64b-73f533e29c2c 2021-01-15T18:53:25.287 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.286+0000 7f14973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:25.288 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.287+0000 7f14973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:25.290 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.289+0000 7f14973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.290+0000 7f14973a8080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:53:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.290+0000 7f14973a8080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:53:25.291 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.290+0000 7f14973a8080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:53:25.328 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:25.327+0000 7f14973a8080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:53:30.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:53:30.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:53:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:30.124 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:53:30.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:53:30.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:30.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:30.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:30.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:30.657 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:53:30.677 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:30.676+0000 7f96b0969080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:30.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:30.678+0000 7f96b0969080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:30.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:30.679+0000 7f96b0969080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:53:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:31.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:53:31.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:31.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:31.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:31.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:31.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:31.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:53:31.050 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:31.196 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:31.195+0000 7f96b0969080 -1 Falling back to public interface 2021-01-15T18:53:31.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:32.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:32.004+0000 7f96b0969080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:53:32.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:32.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:32.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:32.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:53:32.455 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:32.944 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/901355,v1:127.0.0.1:6827/901355] [v2:127.0.0.1:6828/901355,v1:127.0.0.1:6829/901355] exists,up 0c0e555a-d1e4-4e97-b64b-73f533e29c2c 2021-01-15T18:53:32.945 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 96eaaae9-dafe-48e6-add9-3ffb5b03a812 2021-01-15T18:53:32.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:32.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:32.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:32.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2021-01-15T18:53:32.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:32.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:32.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:53:32.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/4 2021-01-15T18:53:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:32.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:32.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:32.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:32.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-01-15T18:53:32.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-01-15T18:53:32.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:32.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:32.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:32.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:32.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:32.952 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:32.953 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:32.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:32.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:32.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:32.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:32.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:32.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:32.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:32.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:32.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:32.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/4 2021-01-15T18:53:32.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:32.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=96eaaae9-dafe-48e6-add9-3ffb5b03a812 2021-01-15T18:53:32.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 96eaaae9-dafe-48e6-add9-3ffb5b03a812' 2021-01-15T18:53:32.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:32.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCs5AFg9yTjOBAAtsiEtvtnGikoretGk9njWA== 2021-01-15T18:53:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCs5AFg9yTjOBAAtsiEtvtnGikoretGk9njWA=="}' 2021-01-15T18:53:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 96eaaae9-dafe-48e6-add9-3ffb5b03a812 -i td/osd-recovery-stats/4/new.json 2021-01-15T18:53:33.475 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:53:33.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/4/new.json 2021-01-15T18:53:33.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCs5AFg9yTjOBAAtsiEtvtnGikoretGk9njWA== --osd-uuid 96eaaae9-dafe-48e6-add9-3ffb5b03a812 2021-01-15T18:53:33.510 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.509+0000 7f7a366c0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:33.512 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.511+0000 7f7a366c0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:33.513 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.512+0000 7f7a366c0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:33.514 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.513+0000 7f7a366c0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:53:33.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.513+0000 7f7a366c0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:53:33.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.513+0000 7f7a366c0080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:53:33.567 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:33.567+0000 7f7a366c0080 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-01-15T18:53:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-01-15T18:53:38.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:38.344 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:53:38.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:53:38.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:38.879 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:53:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:53:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:38.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:38.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:38.900 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:38.899+0000 7f11bd135080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:38.921 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:38.921+0000 7f11bd135080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:38.923 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:38.922+0000 7f11bd135080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:53:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:53:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:39.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:39.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:53:39.273 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:39.439 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:39.438+0000 7f11bd135080 -1 Falling back to public interface 2021-01-15T18:53:39.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:40.248 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:40.247+0000 7f11bd135080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:53:40.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:40.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:40.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:40.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:40.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:53:40.699 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:41.096 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/902627,v1:127.0.0.1:6835/902627] [v2:127.0.0.1:6836/902627,v1:127.0.0.1:6837/902627] exists,up 96eaaae9-dafe-48e6-add9-3ffb5b03a812 2021-01-15T18:53:41.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:41.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:41.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:41.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2021-01-15T18:53:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:53:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:53:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:53:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:53:41.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/5 2021-01-15T18:53:41.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:53:41.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:53:41.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:53:41.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:53:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-01-15T18:53:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-01-15T18:53:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:53:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:53:41.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:53:41.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:53:41.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:53:41.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:53:41.101 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:53:41.101 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:53:41.102 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:53:41.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:41.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:53:41.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:53:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:53:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:53:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:53:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:53:41.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:53:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:53:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:53:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:53:41.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/5 2021-01-15T18:53:41.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:53:41.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=022519d6-fee1-4b0e-8648-e648646a707f 2021-01-15T18:53:41.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 022519d6-fee1-4b0e-8648-e648646a707f' 2021-01-15T18:53:41.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:53:41.107 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 022519d6-fee1-4b0e-8648-e648646a707f 2021-01-15T18:53:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC15AFgBZqXBhAA8wXM/TSgmSL5ZCbkF38lAQ== 2021-01-15T18:53:41.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC15AFgBZqXBhAA8wXM/TSgmSL5ZCbkF38lAQ=="}' 2021-01-15T18:53:41.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 022519d6-fee1-4b0e-8648-e648646a707f -i td/osd-recovery-stats/5/new.json 2021-01-15T18:53:41.669 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:53:41.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/5/new.json 2021-01-15T18:53:41.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC15AFgBZqXBhAA8wXM/TSgmSL5ZCbkF38lAQ== --osd-uuid 022519d6-fee1-4b0e-8648-e648646a707f 2021-01-15T18:53:41.705 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.704+0000 7f4dedc81080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.705+0000 7f4dedc81080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.706+0000 7f4dedc81080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.707+0000 7f4dedc81080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:53:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.707+0000 7f4dedc81080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:53:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.707+0000 7f4dedc81080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:53:41.779 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:41.778+0000 7f4dedc81080 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-01-15T18:53:46.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-01-15T18:53:46.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:53:46.808 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:53:46.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:53:46.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:53:47.330 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:53:47.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:53:47.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:53:47.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:53:47.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:53:47.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:53:47.351 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:47.350+0000 7f787cc98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:47.352 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:47.352+0000 7f787cc98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:47.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:47.353+0000 7f787cc98080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:53:47.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:53:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:53:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:53:47.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:53:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:53:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:53:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:47.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:53:47.724 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:53:47.874 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:47.873+0000 7f787cc98080 -1 Falling back to public interface 2021-01-15T18:53:48.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:48.726 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:53:48.725+0000 7f787cc98080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:53:49.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:49.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:49.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:53:49.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:49.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:53:49.115 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:53:49.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:53:50.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:53:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:53:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:53:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:53:50.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:53:50.557 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:53:50.961 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/903891,v1:127.0.0.1:6843/903891] [v2:127.0.0.1:6844/903891,v1:127.0.0.1:6845/903891] exists,up 022519d6-fee1-4b0e-8648-e648646a707f 2021-01-15T18:53:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:53:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:53:50.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:53:50.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2021-01-15T18:53:50.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:53:51.522 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:53:51.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:53:52.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2021-01-15T18:53:53.407 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 4 2021-01-15T18:53:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2021-01-15T18:53:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:53:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:53:53.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:53:53.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:53:53.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:53:53.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:53:53.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:53:53.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:53:53.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:53:53.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:53:53.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:53:53.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:53:53.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:53:53.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:53:53.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:53:53.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:53:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:53:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:53:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:53:53.969 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:53:53.970 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:53:53.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:53:53.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:53.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:53:54.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836493 2021-01-15T18:53:54.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836493 2021-01-15T18:53:54.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493' 2021-01-15T18:53:54.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:54.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:53:54.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672970 2021-01-15T18:53:54.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672970 2021-01-15T18:53:54.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672970' 2021-01-15T18:53:54.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:54.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:53:54.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T18:53:54.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T18:53:54.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509448' 2021-01-15T18:53:54.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:54.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:53:54.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345928 2021-01-15T18:53:54.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345928 2021-01-15T18:53:54.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509448 3-85899345928' 2021-01-15T18:53:54.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:54.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:53:54.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T18:53:54.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T18:53:54.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509448 3-85899345928 4-107374182405' 2021-01-15T18:53:54.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:53:54.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:53:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T18:53:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T18:53:54.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672970 2-64424509448 3-85899345928 4-107374182405 5-128849018884' 2021-01-15T18:53:54.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:54.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2021-01-15T18:53:54.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:54.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:53:54.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-01-15T18:53:54.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:54.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836493 2021-01-15T18:53:54.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-01-15T18:53:54.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:53:54.814 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2021-01-15T18:53:55.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836493 2021-01-15T18:53:55.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:55.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2021-01-15T18:53:55.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:55.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:53:55.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672970 2021-01-15T18:53:55.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:55.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672970 2021-01-15T18:53:55.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2021-01-15T18:53:55.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:53:55.225 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672970 2021-01-15T18:53:55.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672970 -lt 42949672970 2021-01-15T18:53:55.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:55.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T18:53:55.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:55.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:53:55.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T18:53:55.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T18:53:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T18:53:55.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:53:55.801 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T18:53:56.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T18:53:56.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:56.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345928 2021-01-15T18:53:56.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:56.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:53:56.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345928 2021-01-15T18:53:56.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:56.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345928 2021-01-15T18:53:56.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2021-01-15T18:53:56.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:53:56.026 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345928 2021-01-15T18:53:56.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345926 -lt 85899345928 2021-01-15T18:53:56.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:53:57.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:53:57.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:53:57.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345928 -lt 85899345928 2021-01-15T18:53:57.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:57.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T18:53:57.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:57.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:53:57.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T18:53:57.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:57.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T18:53:57.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T18:53:57.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:53:57.820 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T18:53:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T18:53:58.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:53:58.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T18:53:58.216 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:53:58.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:53:58.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T18:53:58.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:53:58.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T18:53:58.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T18:53:58.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:53:58.218 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T18:53:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2021-01-15T18:53:58.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:53:58.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:53:58.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:53:59.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:53:59.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:53:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:53:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:53:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:53:59.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:53:59.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:53:59.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:53:59.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:53:59.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:53:59.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:53:59.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:00.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:54:00.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:54:00.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:54:00.060 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2021-01-15T18:54:00.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2021-01-15T18:54:00.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2021-01-15T18:54:00.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2021-01-15T18:54:00.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2021-01-15T18:54:00.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2021-01-15T18:54:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2021-01-15T18:54:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2021-01-15T18:54:00.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:00.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2021-01-15T18:54:01.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2021-01-15T18:54:01.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2021-01-15T18:54:01.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2021-01-15T18:54:01.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2021-01-15T18:54:01.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2021-01-15T18:54:01.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2021-01-15T18:54:01.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2021-01-15T18:54:01.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:01.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2021-01-15T18:54:02.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2021-01-15T18:54:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2021-01-15T18:54:02.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2021-01-15T18:54:02.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2021-01-15T18:54:02.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2021-01-15T18:54:02.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2021-01-15T18:54:02.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2021-01-15T18:54:02.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:02.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2021-01-15T18:54:03.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2021-01-15T18:54:03.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2021-01-15T18:54:03.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2021-01-15T18:54:03.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2021-01-15T18:54:03.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2021-01-15T18:54:03.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:03.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2021-01-15T18:54:04.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2021-01-15T18:54:04.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2021-01-15T18:54:04.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2021-01-15T18:54:04.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2021-01-15T18:54:04.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2021-01-15T18:54:04.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2021-01-15T18:54:04.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2021-01-15T18:54:04.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:04.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2021-01-15T18:54:05.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2021-01-15T18:54:05.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2021-01-15T18:54:05.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2021-01-15T18:54:05.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2021-01-15T18:54:05.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2021-01-15T18:54:05.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2021-01-15T18:54:05.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2021-01-15T18:54:05.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2021-01-15T18:54:05.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:05.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2021-01-15T18:54:06.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2021-01-15T18:54:06.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2021-01-15T18:54:06.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2021-01-15T18:54:06.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2021-01-15T18:54:06.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2021-01-15T18:54:06.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2021-01-15T18:54:06.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2021-01-15T18:54:06.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:06.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2021-01-15T18:54:07.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2021-01-15T18:54:07.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2021-01-15T18:54:07.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2021-01-15T18:54:07.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2021-01-15T18:54:07.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2021-01-15T18:54:07.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2021-01-15T18:54:07.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2021-01-15T18:54:07.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:07.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2021-01-15T18:54:08.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2021-01-15T18:54:08.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2021-01-15T18:54:08.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2021-01-15T18:54:08.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2021-01-15T18:54:08.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2021-01-15T18:54:08.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2021-01-15T18:54:08.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:08.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2021-01-15T18:54:09.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2021-01-15T18:54:09.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2021-01-15T18:54:09.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2021-01-15T18:54:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2021-01-15T18:54:09.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2021-01-15T18:54:09.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2021-01-15T18:54:09.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2021-01-15T18:54:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:09.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2021-01-15T18:54:10.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2021-01-15T18:54:10.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2021-01-15T18:54:10.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2021-01-15T18:54:10.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2021-01-15T18:54:10.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2021-01-15T18:54:10.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2021-01-15T18:54:10.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2021-01-15T18:54:10.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:10.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2021-01-15T18:54:11.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2021-01-15T18:54:11.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2021-01-15T18:54:11.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2021-01-15T18:54:11.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2021-01-15T18:54:11.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2021-01-15T18:54:11.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2021-01-15T18:54:11.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2021-01-15T18:54:11.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:11.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2021-01-15T18:54:12.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2021-01-15T18:54:12.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2021-01-15T18:54:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2021-01-15T18:54:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2021-01-15T18:54:12.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2021-01-15T18:54:12.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2021-01-15T18:54:12.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2021-01-15T18:54:12.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:12.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2021-01-15T18:54:13.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2021-01-15T18:54:13.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2021-01-15T18:54:13.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2021-01-15T18:54:13.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2021-01-15T18:54:13.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2021-01-15T18:54:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2021-01-15T18:54:13.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:13.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2021-01-15T18:54:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2021-01-15T18:54:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2021-01-15T18:54:14.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2021-01-15T18:54:14.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2021-01-15T18:54:14.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2021-01-15T18:54:14.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2021-01-15T18:54:14.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2021-01-15T18:54:14.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:14.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2021-01-15T18:54:15.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2021-01-15T18:54:15.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2021-01-15T18:54:15.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2021-01-15T18:54:15.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2021-01-15T18:54:15.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2021-01-15T18:54:15.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2021-01-15T18:54:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2021-01-15T18:54:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2021-01-15T18:54:15.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:15.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2021-01-15T18:54:16.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2021-01-15T18:54:16.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2021-01-15T18:54:16.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2021-01-15T18:54:16.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2021-01-15T18:54:16.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2021-01-15T18:54:16.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2021-01-15T18:54:16.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:16.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2021-01-15T18:54:17.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2021-01-15T18:54:17.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2021-01-15T18:54:17.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2021-01-15T18:54:17.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2021-01-15T18:54:17.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2021-01-15T18:54:17.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2021-01-15T18:54:17.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2021-01-15T18:54:17.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:17.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2021-01-15T18:54:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2021-01-15T18:54:18.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2021-01-15T18:54:18.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2021-01-15T18:54:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2021-01-15T18:54:18.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2021-01-15T18:54:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2021-01-15T18:54:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2021-01-15T18:54:18.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:18.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2021-01-15T18:54:19.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2021-01-15T18:54:19.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2021-01-15T18:54:19.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2021-01-15T18:54:19.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2021-01-15T18:54:19.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2021-01-15T18:54:19.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2021-01-15T18:54:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2021-01-15T18:54:19.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:19.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2021-01-15T18:54:20.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2021-01-15T18:54:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2021-01-15T18:54:20.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2021-01-15T18:54:20.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2021-01-15T18:54:20.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2021-01-15T18:54:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2021-01-15T18:54:20.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:20.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2021-01-15T18:54:21.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2021-01-15T18:54:21.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2021-01-15T18:54:21.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2021-01-15T18:54:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2021-01-15T18:54:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2021-01-15T18:54:21.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2021-01-15T18:54:21.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2021-01-15T18:54:21.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2021-01-15T18:54:22.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2021-01-15T18:54:22.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2021-01-15T18:54:22.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2021-01-15T18:54:22.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2021-01-15T18:54:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2021-01-15T18:54:22.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2021-01-15T18:54:22.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2021-01-15T18:54:22.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:22.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2021-01-15T18:54:23.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2021-01-15T18:54:23.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2021-01-15T18:54:23.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2021-01-15T18:54:23.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2021-01-15T18:54:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2021-01-15T18:54:23.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2021-01-15T18:54:23.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2021-01-15T18:54:23.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:23.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2021-01-15T18:54:24.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2021-01-15T18:54:24.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2021-01-15T18:54:24.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2021-01-15T18:54:24.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2021-01-15T18:54:24.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2021-01-15T18:54:24.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2021-01-15T18:54:24.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2021-01-15T18:54:24.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:24.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2021-01-15T18:54:25.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:25.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2021-01-15T18:54:25.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:25.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2021-01-15T18:54:25.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:25.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2021-01-15T18:54:25.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2021-01-15T18:54:25.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2021-01-15T18:54:25.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2021-01-15T18:54:25.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:54:25.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:54:25.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:54:25.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:54:25.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2021-01-15T18:54:25.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2021-01-15T18:54:25.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:54:25.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:54:25.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:54:25.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:54:26.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2021-01-15T18:54:26.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2021-01-15T18:54:26.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test 2021-01-15T18:54:26.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj1 2021-01-15T18:54:26.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test obj1 2021-01-15T18:54:26.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T18:54:26.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=1 2021-01-15T18:54:26.743 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T18:54:26.744 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:54:26.744 INFO:tasks.workunit.client.0.smithi145.stderr:4' 2021-01-15T18:54:26.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 1 0 2 4 2021-01-15T18:54:26.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2021-01-15T18:54:26.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2021-01-15T18:54:27.436 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:54:27.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-01-15T18:54:27.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2021-01-15T18:54:28.003 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T18:54:28.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-01-15T18:54:28.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2021-01-15T18:54:28.599 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:54:28.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-01-15T18:54:28.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2021-01-15T18:54:29.686 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 is already out. 2021-01-15T18:54:29.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2021-01-15T18:54:29.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2021-01-15T18:54:30.434 INFO:tasks.workunit.client.0.smithi145.stderr:osd.4 is already out. 2021-01-15T18:54:30.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2021-01-15T18:54:31.540 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:54:31.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2021-01-15T18:54:32.806 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:54:32.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2021-01-15T18:54:32.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:54:32.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:54:32.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:54:32.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:54:33.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2021-01-15T18:54:33.365 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:54:33.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2021-01-15T18:54:35.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2021-01-15T18:54:35.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:54:35.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:54:35.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:54:35.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:54:35.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:54:35.378 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:54:35.378 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:54:35.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:54:35.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:54:35.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:54:35.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:54:35.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:54:35.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:54:35.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:54:35.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:54:35.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:54:35.877 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:54:35.877 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:54:35.877 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:54:35.878 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:54:35.878 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:54:35.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:54:35.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:35.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:54:36.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836503 2021-01-15T18:54:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836503 2021-01-15T18:54:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503' 2021-01-15T18:54:36.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:36.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:54:36.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672980 2021-01-15T18:54:36.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672980 2021-01-15T18:54:36.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2021-01-15T18:54:36.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:36.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:54:36.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509459 2021-01-15T18:54:36.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509459 2021-01-15T18:54:36.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459' 2021-01-15T18:54:36.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:36.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:54:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345938 2021-01-15T18:54:36.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345938 2021-01-15T18:54:36.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345938' 2021-01-15T18:54:36.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:36.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:54:36.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182415 2021-01-15T18:54:36.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182415 2021-01-15T18:54:36.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345938 4-107374182415' 2021-01-15T18:54:36.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:54:36.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:54:36.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018895 2021-01-15T18:54:36.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018895 2021-01-15T18:54:36.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509459 3-85899345938 4-107374182415 5-128849018895' 2021-01-15T18:54:36.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:36.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836503 2021-01-15T18:54:36.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:36.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:54:36.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836503 2021-01-15T18:54:36.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:36.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836503 2021-01-15T18:54:36.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2021-01-15T18:54:36.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:54:36.738 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836503 2021-01-15T18:54:37.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836503 -lt 21474836503 2021-01-15T18:54:37.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:37.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672980 2021-01-15T18:54:37.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:37.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:54:37.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672980 2021-01-15T18:54:37.128 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:37.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672980 2021-01-15T18:54:37.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2021-01-15T18:54:37.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:54:37.130 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672980 2021-01-15T18:54:37.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672980 2021-01-15T18:54:37.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:37.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509459 2021-01-15T18:54:37.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:37.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:54:37.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509459 2021-01-15T18:54:37.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:37.571 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509459 2021-01-15T18:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509459 2021-01-15T18:54:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-01-15T18:54:37.572 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:54:37.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509459 -lt 64424509459 2021-01-15T18:54:37.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:38.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345938 2021-01-15T18:54:38.000 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:54:38.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345938 2021-01-15T18:54:38.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345938 2021-01-15T18:54:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2021-01-15T18:54:38.002 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345938 2021-01-15T18:54:38.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:54:38.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345936 -lt 85899345938 2021-01-15T18:54:38.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:54:39.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:54:39.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:54:39.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345938 -lt 85899345938 2021-01-15T18:54:39.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:39.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182415 2021-01-15T18:54:39.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:39.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:54:39.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182415 2021-01-15T18:54:39.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:39.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182415 2021-01-15T18:54:39.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2021-01-15T18:54:39.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:54:39.795 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182415 2021-01-15T18:54:40.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182416 -lt 107374182415 2021-01-15T18:54:40.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:54:40.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018895 2021-01-15T18:54:40.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:54:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:54:40.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018895 2021-01-15T18:54:40.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:54:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018895 2021-01-15T18:54:40.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2021-01-15T18:54:40.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:54:40.189 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018895 2021-01-15T18:54:40.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018896 -lt 128849018895 2021-01-15T18:54:40.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:54:40.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:40.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:41.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:54:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:41.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:41.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:41.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:41.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:41.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:41.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:41.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:41.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:41.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:41.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:42.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:54:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:54:42.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:54:42.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:42.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:42.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:42.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:42.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:42.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:42.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:42.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:42.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:42.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:42.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:42.505 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:42.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:43.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:43.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:43.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:43.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:43.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:43.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:43.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:43.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:54:43.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:54:43.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:43.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:43.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:43.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:43.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:43.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:43.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:44.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:44.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:44.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:44.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:44.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:44.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:44.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:44.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:44.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:44.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:44.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:44.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:44.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:54:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:54:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:45.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:45.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:45.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:45.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:45.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:45.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:45.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:45.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:45.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:45.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:45.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:45.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:45.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:46.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:46.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:46.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:46.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:46.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:46.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:46.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:46.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:46.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:54:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:54:46.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:46.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:46.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:46.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:46.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:46.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:46.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:46.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:46.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:46.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:47.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:47.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:47.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:47.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:47.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:47.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:47.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:47.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:47.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:47.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:47.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:48.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:54:48.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:54:48.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:48.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:48.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:48.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:48.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:48.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:48.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:48.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:48.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:48.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:48.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:48.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:48.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:49.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:49.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:49.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:49.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:49.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:49.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:49.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:49.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:54:49.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:54:49.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:49.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:49.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:49.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:49.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:49.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:49.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:49.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:49.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:49.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:49.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:50.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:50.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:50.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:50.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:50.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:50.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:50.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:50.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:50.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:50.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:50.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:50.747 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:50.748 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:51.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:54:51.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:54:51.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:51.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:51.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:51.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:51.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:51.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:51.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:51.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:51.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:51.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:51.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:51.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:51.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:51.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:51.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:52.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:52.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:52.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:52.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:52.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:52.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:52.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:54:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:54:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:52.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:52.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:52.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:52.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:52.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:52.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:52.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:52.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:52.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:52.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:53.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:53.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:53.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:53.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:53.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:53.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:53.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:53.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:54.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:54:54.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:54:54.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:54.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:54.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:54.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:54.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:54.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:54.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:54.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:54.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:54.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:54.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:55.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:55.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:55.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:55.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:55.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:55.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:54:55.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:54:55.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:55.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:55.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:56.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:56.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:56.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:56.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:56.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:56.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:56.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:56.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:56.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:56.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:56.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:56.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:56.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:56.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:56.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:56.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:56.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:56.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:56.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:56.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:56.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:57.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:54:57.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:54:57.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:57.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:57.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:57.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:57.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:57.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:57.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:57.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:57.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:57.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:58.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:58.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:58.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:58.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:58.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:58.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:54:58.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:54:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:54:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:54:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:54:58.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:54:59.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:54:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:54:59.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:54:59.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:54:59.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:54:59.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:54:59.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:54:59.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:54:59.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:54:59.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:54:59.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:54:59.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:54:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:54:59.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:54:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:54:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:54:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:54:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:54:59.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:54:59.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:54:59.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:55:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:55:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:00.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:00.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:00.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:00.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:00.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:00.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:00.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:00.616 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:00.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:00.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:00.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:00.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:00.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:00.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:01.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:01.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:01.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:01.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:01.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:01.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:02.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:55:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:55:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:02.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:02.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:02.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:02.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:02.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:02.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:02.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:02.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:02.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:02.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:02.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:02.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:02.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:03.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:03.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:03.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:03.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:03.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:03.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:03.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:55:03.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:55:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:03.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:03.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:03.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:03.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:03.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:03.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:03.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:03.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:03.654 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:04.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:04.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:04.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:04.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:04.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:04.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:04.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:04.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:04.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:05.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:55:05.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:55:05.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:05.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:05.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:05.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:05.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:05.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:05.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:05.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:05.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:05.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:05.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:05.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:06.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:06.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:06.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:06.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:06.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:06.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:06.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:55:06.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:55:06.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:06.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:06.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:06.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:06.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:06.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:06.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:06.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:06.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:06.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:07.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:07.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:07.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:07.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:07.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:07.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:07.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:07.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:07.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:08.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:55:08.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:55:08.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:08.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:08.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:08.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:08.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:08.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:08.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:08.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:08.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:08.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:08.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:08.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:08.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:08.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:08.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:09.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:09.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:09.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:09.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:09.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:09.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:55:09.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:55:09.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:09.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:09.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:09.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:09.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:09.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:09.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:09.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:09.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:09.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:09.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:10.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:10.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:10.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:10.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:10.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:10.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:10.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:10.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:10.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:10.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:11.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:55:11.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:55:11.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:11.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:11.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:11.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:11.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:11.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:11.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:11.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:11.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:11.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:11.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:11.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:11.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:11.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:11.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:12.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:12.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:12.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:12.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:12.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:12.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:12.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:55:12.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:55:12.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:12.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:12.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:12.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:12.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:12.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:12.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:12.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:12.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:12.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:12.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:13.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:13.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:13.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:13.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:13.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:13.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:13.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:13.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:13.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:13.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:13.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:14.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:55:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:55:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:14.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:14.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:14.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:14.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:14.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:14.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:14.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:14.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:14.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:14.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:14.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:14.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:14.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:15.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:15.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:15.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:15.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:15.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:15.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:15.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:55:15.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:55:15.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:15.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:15.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:15.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:15.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:15.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:15.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:15.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:15.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:15.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:16.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:16.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:16.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:16.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:16.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:17.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6 2021-01-15T18:55:17.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6 '!=' null 2021-01-15T18:55:17.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:17.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:17.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:17.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:17.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:17.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:17.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:17.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:17.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:17.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:17.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:17.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:17.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:17.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:18.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:18.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:18.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:18.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:18.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:18.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:18.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:18.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:18.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:18.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:55:18.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:55:18.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:18.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:18.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:18.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:18.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:18.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:18.956 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:18.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:18.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:18.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:18.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:19.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:55:19.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:19.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:19.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:19.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:55:19.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:55:19.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:55:19.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:55:19.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:55:19.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:55:19.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:55:19.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:55:19.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:55:20.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:55:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:55:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:55:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:55:20.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:55:20.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:55:20.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:55:20.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:55:20.461 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:55:20.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:55:20.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:55:20.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:55:20.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:55:20.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:55:20.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:55:20.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:55:20.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:55:21.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:55:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:55:21.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:55:21.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2021-01-15T18:55:21.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:55:21.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:55:21.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:55:21.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:55:21.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2021-01-15T18:55:21.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2021-01-15T18:55:21.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2021-01-15T18:55:21.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2021-01-15T18:55:21.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:55:21.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:55:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2021-01-15T18:55:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-01-15T18:55:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2021-01-15T18:55:21.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:55:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2021-01-15T18:55:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:55:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-01-15T18:55:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-01-15T18:55:21.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:55:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-01-15T18:55:21.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:55:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.745 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:55:21.746 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:55:21.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2021-01-15T18:55:21.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:55:21.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:55:21.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2021-01-15T18:55:21.758 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:55:21.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:55:21.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2021-01-15T18:55:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2021-01-15T18:55:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-01-15T18:55:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:55:21.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-01-15T18:55:21.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:55:21.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:55:21.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:55:21.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:55:21.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:55:21.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:55:21.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:55:21.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:55:21.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:55:21.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:55:21.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:55:21.773 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2021-01-15T18:55:21.774 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:55:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:55:21.775 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2021-01-15T18:55:21.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2021-01-15T18:55:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-01-15T18:55:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:55:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-01-15T18:55:21.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:55:21.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:55:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:55:21.778 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:55:21.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:55:21.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:55:21.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:55:21.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:55:21.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:55:21.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-01-15T18:55:21.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2021-01-15T18:55:21.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:55:21.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2021-01-15T18:55:21.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2021-01-15T18:55:21.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2021-01-15T18:55:21.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2021-01-15T18:55:21.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2021-01-15T18:55:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:55:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:55:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:55:21.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:55:21.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2021-01-15T18:55:21.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2021-01-15T18:55:21.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:55:21.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2021-01-15T18:55:21.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2021-01-15T18:55:21.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:55:21.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:55:21.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:55:21.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:55:21.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2021-01-15T18:55:21.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:55:21.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:55:22.320 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:55:22.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2021-01-15T18:55:22.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:55:22.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:55:22.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:55:22.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:55:22.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:55:22.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:55:22.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:55:22.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:55:22.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:55:22.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:55:22.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:55:22.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:55:22.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:55:22.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:55:22.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:55:22.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:55:22.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:55:22.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:55:22.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:55:22.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:55:22.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:55:22.450 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:55:22.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:55:22.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:55:22.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:55:22.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:55:22.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:55:22.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:55:22.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:55:22.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:55:22.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:22.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:22.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:55:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:55:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:55:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:55:22.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:55:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:55:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:55:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:55:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:55:22.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:55:22.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:55:22.474 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:55:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:55:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:55:22.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:55:22.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:55:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:55:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:55:22.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:55:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:55:22.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:55:22.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:55:22.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:55:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:55:22.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:55:22.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:55:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:55:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:55:22.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:55:22.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:55:22.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:22.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:22.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:55:22.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:55:22.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:55:22.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:55:22.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:55:22.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:22.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:22.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:55:22.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:55:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:55:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:55:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:55:22.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2021-01-15T18:55:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2021-01-15T18:55:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2021-01-15T18:55:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:55:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:55:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:55:22.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:55:22.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:55:22.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:55:22.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:55:22.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:22.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:22.788 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:22.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:22.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:55:22.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:55:22.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:55:22.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:55:22.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:55:22.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:55:22.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:55:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:55:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:55:22.969 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:55:22.970 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:55:22.970 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:22.970 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:22.970 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:55:22.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:55:22.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:55:23.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:55:23.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:55:23.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:55:23.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:55:23.066 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:55:23.066 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:55:23.067 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:55:23.067 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:55:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:55:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:23.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:55:23.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:55:23.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:55:23.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2021-01-15T18:55:23.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:55:23.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:55:23.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:55:23.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:55:23.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:55:23.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:55:23.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:23.666 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:23.667 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:23.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:23.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:55:23.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:55:23.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2021-01-15T18:55:23.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:55:23.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:55:23.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:55:23.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:55:23.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:55:23.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:55:23.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:55:23.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:55:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:55:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:55:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:55:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:55:23.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:55:23.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:55:23.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:55:23.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:23.693 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:23.693 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:23.694 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:23.694 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:23.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:23.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:23.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:55:23.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:55:23.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:55:23.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:55:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:55:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:55:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:55:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:23.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:55:23.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:55:23.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:55:23.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:55:23.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f635ba04-8e0c-4998-a717-4f3140a27352 2021-01-15T18:55:23.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 f635ba04-8e0c-4998-a717-4f3140a27352' 2021-01-15T18:55:23.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:55:23.699 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 f635ba04-8e0c-4998-a717-4f3140a27352 2021-01-15T18:55:23.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAb5QFgLpf4KRAAMZ3s7DyFfpuhuVmAkFPfCg== 2021-01-15T18:55:23.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAb5QFgLpf4KRAAMZ3s7DyFfpuhuVmAkFPfCg=="}' 2021-01-15T18:55:23.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f635ba04-8e0c-4998-a717-4f3140a27352 -i td/osd-recovery-stats/0/new.json 2021-01-15T18:55:24.047 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:55:24.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:55:24.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAb5QFgLpf4KRAAMZ3s7DyFfpuhuVmAkFPfCg== --osd-uuid f635ba04-8e0c-4998-a717-4f3140a27352 2021-01-15T18:55:24.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.082+0000 7fe1185f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:24.085 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.084+0000 7fe1185f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:24.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.085+0000 7fe1185f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:24.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.086+0000 7fe1185f2080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:55:24.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.086+0000 7fe1185f2080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:55:24.087 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.086+0000 7fe1185f2080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:55:24.129 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:24.128+0000 7fe1185f2080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:55:28.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:55:28.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:55:28.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:55:28.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:55:28.807 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:55:29.335 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:55:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:55:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:55:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:55:29.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:55:29.357 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:29.356+0000 7f818f4a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:29.359 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:29.358+0000 7f818f4a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:29.360 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:29.359+0000 7f818f4a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:29.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:55:29.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:55:29.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:55:29.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:55:29.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:55:29.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:29.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:55:29.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:29.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:55:29.751 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:55:29.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:29.879+0000 7f818f4a6080 -1 Falling back to public interface 2021-01-15T18:55:30.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:55:30.686 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:30.685+0000 7f818f4a6080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:55:31.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:55:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:55:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:31.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:55:31.139 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:55:31.572 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/913090,v1:127.0.0.1:6803/913090] [v2:127.0.0.1:6804/913090,v1:127.0.0.1:6805/913090] exists,up f635ba04-8e0c-4998-a717-4f3140a27352 2021-01-15T18:55:31.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:55:31.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:55:31.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:55:31.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2021-01-15T18:55:31.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:55:31.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:55:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:55:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:55:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:55:31.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:55:31.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:55:31.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:55:31.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:55:31.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:55:31.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:55:31.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:55:31.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:55:31.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:55:31.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:55:31.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:31.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:31.577 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:31.577 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:31.577 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:31.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:31.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:31.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:55:31.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:55:31.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:55:31.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:55:31.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:55:31.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:55:31.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:55:31.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:31.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:55:31.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:55:31.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:55:31.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:55:31.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=c91da906-9625-408b-ad56-5318c2930168 2021-01-15T18:55:31.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 c91da906-9625-408b-ad56-5318c2930168' 2021-01-15T18:55:31.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:55:31.582 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 c91da906-9625-408b-ad56-5318c2930168 2021-01-15T18:55:31.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAj5QFgK3XvIhAAFSm62E4g+6mVkv1JUVKBTQ== 2021-01-15T18:55:31.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAj5QFgK3XvIhAAFSm62E4g+6mVkv1JUVKBTQ=="}' 2021-01-15T18:55:31.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new c91da906-9625-408b-ad56-5318c2930168 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:55:32.134 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:55:32.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:55:32.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAj5QFgK3XvIhAAFSm62E4g+6mVkv1JUVKBTQ== --osd-uuid c91da906-9625-408b-ad56-5318c2930168 2021-01-15T18:55:32.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.168+0000 7f264e70e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:32.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.170+0000 7f264e70e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:32.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.171+0000 7f264e70e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:32.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.172+0000 7f264e70e080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:55:32.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.172+0000 7f264e70e080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:55:32.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.172+0000 7f264e70e080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:55:32.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:32.233+0000 7f264e70e080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:55:36.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:55:36.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:55:36.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:55:36.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:55:36.727 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:55:37.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:55:37.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:55:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:55:37.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:55:37.250 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:55:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:37.270+0000 7f28a8e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:37.272 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:37.272+0000 7f28a8e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:37.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:37.273+0000 7f28a8e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:37.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:55:37.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:55:37.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:55:37.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:55:37.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:55:37.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:37.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:55:37.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:37.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:55:37.644 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:55:37.790 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:37.789+0000 7f28a8e9e080 -1 Falling back to public interface 2021-01-15T18:55:38.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:55:38.643 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:38.642+0000 7f28a8e9e080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:55:39.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:55:39.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:39.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:55:39.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:39.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:55:39.051 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:55:39.503 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/914362,v1:127.0.0.1:6811/914362] [v2:127.0.0.1:6812/914362,v1:127.0.0.1:6813/914362] exists,up c91da906-9625-408b-ad56-5318c2930168 2021-01-15T18:55:39.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:55:39.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:55:39.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:55:39.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2021-01-15T18:55:39.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:55:39.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:55:39.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:55:39.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:55:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:55:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:55:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:55:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:55:39.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:55:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:55:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:55:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:55:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:55:39.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:55:39.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:55:39.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:39.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:39.507 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:39.507 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:39.508 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:39.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:39.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:39.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:55:39.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:55:39.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:55:39.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:55:39.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:55:39.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:55:39.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:55:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:55:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:55:39.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:55:39.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:55:39.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=6900dfa4-4f7a-4697-bcce-74af989dfd2a 2021-01-15T18:55:39.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 6900dfa4-4f7a-4697-bcce-74af989dfd2a' 2021-01-15T18:55:39.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:55:39.512 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 6900dfa4-4f7a-4697-bcce-74af989dfd2a 2021-01-15T18:55:39.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAr5QFgY5jLHhAArJ5wwzy4JKe3aycCkC3qgg== 2021-01-15T18:55:39.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAr5QFgY5jLHhAArJ5wwzy4JKe3aycCkC3qgg=="}' 2021-01-15T18:55:39.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 6900dfa4-4f7a-4697-bcce-74af989dfd2a -i td/osd-recovery-stats/2/new.json 2021-01-15T18:55:40.134 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:55:40.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:55:40.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAr5QFgY5jLHhAArJ5wwzy4JKe3aycCkC3qgg== --osd-uuid 6900dfa4-4f7a-4697-bcce-74af989dfd2a 2021-01-15T18:55:40.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.168+0000 7f35d4407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:40.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.170+0000 7f35d4407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:40.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.171+0000 7f35d4407080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:40.173 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.173+0000 7f35d4407080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:55:40.174 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.173+0000 7f35d4407080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:55:40.174 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.173+0000 7f35d4407080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:55:40.215 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:40.214+0000 7f35d4407080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:55:44.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:55:44.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:55:44.990 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:55:44.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:55:44.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:55:45.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:55:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:55:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:55:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:55:45.513 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:55:45.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:45.532+0000 7f2a5eaa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:45.555 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:45.554+0000 7f2a5eaa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:45.557 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:45.556+0000 7f2a5eaa9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:55:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:55:45.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:55:45.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:55:45.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:55:45.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:45.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:55:45.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:45.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:55:45.907 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:55:46.074 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:46.073+0000 7f2a5eaa9080 -1 Falling back to public interface 2021-01-15T18:55:46.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:55:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:46.873+0000 7f2a5eaa9080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:55:47.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:55:47.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:47.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:55:47.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:47.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:55:47.298 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:55:47.699 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/915626,v1:127.0.0.1:6819/915626] [v2:127.0.0.1:6820/915626,v1:127.0.0.1:6821/915626] exists,up 6900dfa4-4f7a-4697-bcce-74af989dfd2a 2021-01-15T18:55:48.036 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 a360404a-93c4-46b5-b9a4-11d0cec37ce4 2021-01-15T18:55:48.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:55:48.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:55:48.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:55:48.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2021-01-15T18:55:48.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:55:48.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:55:48.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:55:48.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:55:48.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/3 2021-01-15T18:55:48.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:55:48.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:55:48.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:55:48.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:55:48.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2021-01-15T18:55:48.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2021-01-15T18:55:48.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:55:48.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:55:48.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:55:48.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:55:48.042 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:48.043 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:48.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:48.043 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:48.044 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:48.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:48.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:55:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:55:48.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:55:48.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:55:48.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:55:48.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:55:48.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:55:48.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:48.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:55:48.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:55:48.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/3 2021-01-15T18:55:48.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:55:48.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=a360404a-93c4-46b5-b9a4-11d0cec37ce4 2021-01-15T18:55:48.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 a360404a-93c4-46b5-b9a4-11d0cec37ce4' 2021-01-15T18:55:48.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:55:48.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAz5QFgmPJ+KhAAdL52e3vLW+Z9ul9fH0TDHw== 2021-01-15T18:55:48.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAz5QFgmPJ+KhAAdL52e3vLW+Z9ul9fH0TDHw=="}' 2021-01-15T18:55:48.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new a360404a-93c4-46b5-b9a4-11d0cec37ce4 -i td/osd-recovery-stats/3/new.json 2021-01-15T18:55:48.264 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:55:48.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/3/new.json 2021-01-15T18:55:48.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAz5QFgmPJ+KhAAdL52e3vLW+Z9ul9fH0TDHw== --osd-uuid a360404a-93c4-46b5-b9a4-11d0cec37ce4 2021-01-15T18:55:48.299 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.298+0000 7f2a3fb33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:48.301 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.300+0000 7f2a3fb33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.301+0000 7f2a3fb33080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.301+0000 7f2a3fb33080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:55:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.301+0000 7f2a3fb33080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:55:48.303 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.301+0000 7f2a3fb33080 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2021-01-15T18:55:48.366 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:48.366+0000 7f2a3fb33080 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2021-01-15T18:55:53.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2021-01-15T18:55:53.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:55:53.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:55:53.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:55:53.125 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:55:53.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:55:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:53.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:55:53.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:55:53.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:55:53.675 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:55:53.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:53.692+0000 7f5613ad1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:53.729 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:53.728+0000 7f5613ad1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:53.731 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:53.730+0000 7f5613ad1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:54.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:55:54.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:55:54.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:55:54.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:55:54.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:55:54.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:54.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:55:54.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:54.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:55:54.087 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:55:54.245 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:54.244+0000 7f5613ad1080 -1 Falling back to public interface 2021-01-15T18:55:54.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:55:55.039 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:55.038+0000 7f5613ad1080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:55:55.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:55:55.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:55:55.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:55:55.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:55:55.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:55:55.477 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:55:55.876 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/916890,v1:127.0.0.1:6827/916890] [v2:127.0.0.1:6828/916890,v1:127.0.0.1:6829/916890] exists,up a360404a-93c4-46b5-b9a4-11d0cec37ce4 2021-01-15T18:55:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:55:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:55:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:55:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2021-01-15T18:55:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:55:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:55:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=4 2021-01-15T18:55:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:55:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/4 2021-01-15T18:55:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:55:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:55:55.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:55:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:55:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2021-01-15T18:55:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2021-01-15T18:55:55.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:55:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:55:55.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:55:55.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:55:55.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:55:55.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:55:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:55:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:55:55.882 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:55:55.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:55:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:55:55.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:55:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:55:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:55:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:55:55.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:55:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:55:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:55:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:55:55.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:55:55.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/4 2021-01-15T18:55:55.886 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:55:55.887 INFO:tasks.workunit.client.0.smithi145.stdout:add osd4 da0015ee-16e2-4b7d-b8cb-6bd94f0986cc 2021-01-15T18:55:55.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=da0015ee-16e2-4b7d-b8cb-6bd94f0986cc 2021-01-15T18:55:55.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd4 da0015ee-16e2-4b7d-b8cb-6bd94f0986cc' 2021-01-15T18:55:55.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:55:55.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA75QFgLAkTNRAACj1drLRv/VH5/NM5wfE6iQ== 2021-01-15T18:55:55.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA75QFgLAkTNRAACj1drLRv/VH5/NM5wfE6iQ=="}' 2021-01-15T18:55:55.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new da0015ee-16e2-4b7d-b8cb-6bd94f0986cc -i td/osd-recovery-stats/4/new.json 2021-01-15T18:55:56.448 INFO:tasks.workunit.client.0.smithi145.stdout:4 2021-01-15T18:55:56.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/4/new.json 2021-01-15T18:55:56.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA75QFgLAkTNRAACj1drLRv/VH5/NM5wfE6iQ== --osd-uuid da0015ee-16e2-4b7d-b8cb-6bd94f0986cc 2021-01-15T18:55:56.484 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.483+0000 7fc3cc1a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:56.485 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.485+0000 7fc3cc1a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:56.487 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.486+0000 7fc3cc1a2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:55:56.488 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.487+0000 7fc3cc1a2080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:55:56.488 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.487+0000 7fc3cc1a2080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:55:56.488 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.487+0000 7fc3cc1a2080 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2021-01-15T18:55:56.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:55:56.537+0000 7fc3cc1a2080 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2021-01-15T18:56:01.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2021-01-15T18:56:01.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:56:01.260 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd4 key to auth repository 2021-01-15T18:56:01.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd4 key to auth repository 2021-01-15T18:56:01.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:56:01.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.4 2021-01-15T18:56:01.787 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.4 2021-01-15T18:56:01.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 4 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:56:01.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:56:01.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:56:01.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:56:01.807 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:01.806+0000 7fd13cdd9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:01.815 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:01.814+0000 7fd13cdd9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:01.816 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:01.816+0000 7fd13cdd9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 4 2021-01-15T18:56:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:56:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=4 2021-01-15T18:56:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:56:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:56:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:56:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:56:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:56:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:56:02.181 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:56:02.334 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:02.333+0000 7fd13cdd9080 -1 Falling back to public interface 2021-01-15T18:56:02.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:56:03.222 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:03.220+0000 7fd13cdd9080 -1 osd.4 0 log_to_monitors {default=true} 2021-01-15T18:56:03.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:56:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:56:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:56:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:56:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:56:03.588 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:56:03.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:56:04.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:56:04.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:56:04.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:56:04.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:56:04.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.4 up' 2021-01-15T18:56:04.997 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:56:05.394 INFO:tasks.workunit.client.0.smithi145.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/918154,v1:127.0.0.1:6835/918154] [v2:127.0.0.1:6836/918154,v1:127.0.0.1:6837/918154] exists,up da0015ee-16e2-4b7d-b8cb-6bd94f0986cc 2021-01-15T18:56:05.394 INFO:tasks.workunit.client.0.smithi145.stdout:add osd5 3db90ef1-fe9c-4623-9679-32b4a75edbf4 2021-01-15T18:56:05.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:56:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:56:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:56:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2021-01-15T18:56:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:56:05.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:56:05.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=5 2021-01-15T18:56:05.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:56:05.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/5 2021-01-15T18:56:05.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:56:05.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:56:05.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:56:05.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:56:05.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2021-01-15T18:56:05.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2021-01-15T18:56:05.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:56:05.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:56:05.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:56:05.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:56:05.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:56:05.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:56:05.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:56:05.400 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:56:05.400 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:56:05.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:56:05.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:56:05.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:56:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:56:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:56:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:56:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:56:05.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:56:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:56:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:56:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:56:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:56:05.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/5 2021-01-15T18:56:05.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:56:05.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3db90ef1-fe9c-4623-9679-32b4a75edbf4 2021-01-15T18:56:05.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd5 3db90ef1-fe9c-4623-9679-32b4a75edbf4' 2021-01-15T18:56:05.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:56:05.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBF5QFgEcRJGBAAvE4FuqmTaL7T//P5Su+YAQ== 2021-01-15T18:56:05.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBF5QFgEcRJGBAAvE4FuqmTaL7T//P5Su+YAQ=="}' 2021-01-15T18:56:05.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3db90ef1-fe9c-4623-9679-32b4a75edbf4 -i td/osd-recovery-stats/5/new.json 2021-01-15T18:56:05.918 INFO:tasks.workunit.client.0.smithi145.stdout:5 2021-01-15T18:56:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/5/new.json 2021-01-15T18:56:05.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBF5QFgEcRJGBAAvE4FuqmTaL7T//P5Su+YAQ== --osd-uuid 3db90ef1-fe9c-4623-9679-32b4a75edbf4 2021-01-15T18:56:05.953 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.952+0000 7f391e32c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:05.954 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.954+0000 7f391e32c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:05.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.955+0000 7f391e32c080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:05.956 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.956+0000 7f391e32c080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:56:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.956+0000 7f391e32c080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:56:05.957 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:05.956+0000 7f391e32c080 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2021-01-15T18:56:06.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:06.004+0000 7f391e32c080 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2021-01-15T18:56:10.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2021-01-15T18:56:10.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:56:10.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd5 key to auth repository 2021-01-15T18:56:10.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:56:10.761 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd5 key to auth repository 2021-01-15T18:56:11.286 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.5 2021-01-15T18:56:11.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.5 2021-01-15T18:56:11.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 5 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:56:11.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:56:11.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:56:11.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:56:11.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:11.306+0000 7fc95df02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:11.331 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:11.330+0000 7fc95df02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:11.333 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:11.332+0000 7fc95df02080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:56:11.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 5 2021-01-15T18:56:11.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:56:11.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=5 2021-01-15T18:56:11.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:56:11.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:56:11.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:56:11.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:56:11.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:56:11.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:56:11.684 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:56:11.849 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:11.848+0000 7fc95df02080 -1 Falling back to public interface 2021-01-15T18:56:12.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:56:12.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:56:12.632+0000 7fc95df02080 -1 osd.5 0 log_to_monitors {default=true} 2021-01-15T18:56:13.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:56:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:56:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:56:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:56:13.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.5 up' 2021-01-15T18:56:13.083 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:56:13.495 INFO:tasks.workunit.client.0.smithi145.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/919444,v1:127.0.0.1:6843/919444] [v2:127.0.0.1:6844/919444,v1:127.0.0.1:6845/919444] exists,up 3db90ef1-fe9c-4623-9679-32b4a75edbf4 2021-01-15T18:56:13.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:56:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:56:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:56:13.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2021-01-15T18:56:13.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:56:14.099 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:56:14.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:56:15.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2021-01-15T18:56:15.877 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T18:56:15.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2021-01-15T18:56:15.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:56:15.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:56:15.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:56:15.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:56:15.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:56:15.891 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:56:15.891 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:56:15.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:56:15.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:56:15.892 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:56:15.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:56:15.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:56:15.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:56:15.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:56:15.980 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:56:16.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:56:16.388 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:56:16.388 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:56:16.388 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:56:16.388 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:56:16.389 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:56:16.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:56:16.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:16.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:56:16.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836491 2021-01-15T18:56:16.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836491 2021-01-15T18:56:16.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491' 2021-01-15T18:56:16.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:16.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:56:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T18:56:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T18:56:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971' 2021-01-15T18:56:16.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:16.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:56:16.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509448 2021-01-15T18:56:16.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509448 2021-01-15T18:56:16.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448' 2021-01-15T18:56:16.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:16.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:56:16.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345927 2021-01-15T18:56:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345927 2021-01-15T18:56:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927' 2021-01-15T18:56:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:16.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:56:17.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182405 2021-01-15T18:56:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182405 2021-01-15T18:56:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927 4-107374182405' 2021-01-15T18:56:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:17.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:56:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018884 2021-01-15T18:56:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018884 2021-01-15T18:56:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836491 1-42949672971 2-64424509448 3-85899345927 4-107374182405 5-128849018884' 2021-01-15T18:56:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2021-01-15T18:56:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:56:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836491 2021-01-15T18:56:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:17.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836491 2021-01-15T18:56:17.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2021-01-15T18:56:17.207 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836491 2021-01-15T18:56:17.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:56:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836492 -lt 21474836491 2021-01-15T18:56:17.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:17.605 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T18:56:17.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:17.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:56:17.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T18:56:17.608 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:17.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T18:56:17.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T18:56:17.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:56:17.610 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T18:56:18.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T18:56:18.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:18.047 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509448 2021-01-15T18:56:18.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:18.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:56:18.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509448 2021-01-15T18:56:18.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:18.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509448 2021-01-15T18:56:18.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2021-01-15T18:56:18.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:56:18.051 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509448 2021-01-15T18:56:18.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509448 -lt 64424509448 2021-01-15T18:56:18.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:18.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345927 2021-01-15T18:56:18.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:18.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:56:18.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345927 2021-01-15T18:56:18.442 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:18.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345927 2021-01-15T18:56:18.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2021-01-15T18:56:18.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:56:18.444 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345927 2021-01-15T18:56:18.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345927 2021-01-15T18:56:18.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:18.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182405 2021-01-15T18:56:18.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:18.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:56:18.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182405 2021-01-15T18:56:18.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:18.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182405 2021-01-15T18:56:18.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182405' 2021-01-15T18:56:18.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:56:18.836 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182405 2021-01-15T18:56:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182405 -lt 107374182405 2021-01-15T18:56:19.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:19.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018884 2021-01-15T18:56:19.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:19.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:56:19.252 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018884 2021-01-15T18:56:19.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:19.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018884 2021-01-15T18:56:19.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018884' 2021-01-15T18:56:19.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:56:19.254 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018884 2021-01-15T18:56:19.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018885 -lt 128849018884 2021-01-15T18:56:19.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:56:19.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:19.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:20.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:56:20.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:56:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:56:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:56:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:56:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:56:20.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:56:20.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:56:20.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:56:20.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:56:20.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:20.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:21.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:56:21.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:56:21.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:56:21.044 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2021-01-15T18:56:21.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2021-01-15T18:56:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2021-01-15T18:56:21.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2021-01-15T18:56:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2021-01-15T18:56:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2021-01-15T18:56:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2021-01-15T18:56:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2021-01-15T18:56:21.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2021-01-15T18:56:21.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2021-01-15T18:56:21.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:21.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2021-01-15T18:56:22.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2021-01-15T18:56:22.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2021-01-15T18:56:22.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2021-01-15T18:56:22.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2021-01-15T18:56:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2021-01-15T18:56:22.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2021-01-15T18:56:22.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2021-01-15T18:56:22.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2021-01-15T18:56:22.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2021-01-15T18:56:22.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:22.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2021-01-15T18:56:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2021-01-15T18:56:23.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2021-01-15T18:56:23.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2021-01-15T18:56:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2021-01-15T18:56:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2021-01-15T18:56:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2021-01-15T18:56:23.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2021-01-15T18:56:23.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2021-01-15T18:56:23.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2021-01-15T18:56:23.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:23.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2021-01-15T18:56:24.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2021-01-15T18:56:24.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2021-01-15T18:56:24.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2021-01-15T18:56:24.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2021-01-15T18:56:24.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2021-01-15T18:56:24.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2021-01-15T18:56:24.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2021-01-15T18:56:24.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2021-01-15T18:56:24.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:24.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2021-01-15T18:56:25.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2021-01-15T18:56:25.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2021-01-15T18:56:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2021-01-15T18:56:25.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2021-01-15T18:56:25.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2021-01-15T18:56:25.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2021-01-15T18:56:25.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2021-01-15T18:56:25.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2021-01-15T18:56:25.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2021-01-15T18:56:25.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:25.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2021-01-15T18:56:26.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2021-01-15T18:56:26.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2021-01-15T18:56:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2021-01-15T18:56:26.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2021-01-15T18:56:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2021-01-15T18:56:26.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2021-01-15T18:56:26.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2021-01-15T18:56:26.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2021-01-15T18:56:26.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2021-01-15T18:56:26.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:26.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2021-01-15T18:56:27.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2021-01-15T18:56:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2021-01-15T18:56:27.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2021-01-15T18:56:27.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2021-01-15T18:56:27.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2021-01-15T18:56:27.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2021-01-15T18:56:27.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2021-01-15T18:56:27.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:27.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2021-01-15T18:56:28.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2021-01-15T18:56:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2021-01-15T18:56:28.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2021-01-15T18:56:28.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2021-01-15T18:56:28.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2021-01-15T18:56:28.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2021-01-15T18:56:28.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2021-01-15T18:56:28.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2021-01-15T18:56:28.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2021-01-15T18:56:28.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:28.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2021-01-15T18:56:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2021-01-15T18:56:29.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2021-01-15T18:56:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2021-01-15T18:56:29.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2021-01-15T18:56:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2021-01-15T18:56:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2021-01-15T18:56:29.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2021-01-15T18:56:29.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2021-01-15T18:56:29.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2021-01-15T18:56:29.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:29.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2021-01-15T18:56:30.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2021-01-15T18:56:30.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2021-01-15T18:56:30.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2021-01-15T18:56:30.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2021-01-15T18:56:30.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2021-01-15T18:56:30.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2021-01-15T18:56:30.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2021-01-15T18:56:30.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2021-01-15T18:56:30.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2021-01-15T18:56:30.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:30.988 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2021-01-15T18:56:31.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2021-01-15T18:56:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2021-01-15T18:56:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2021-01-15T18:56:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2021-01-15T18:56:31.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2021-01-15T18:56:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2021-01-15T18:56:31.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2021-01-15T18:56:31.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2021-01-15T18:56:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:31.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2021-01-15T18:56:32.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2021-01-15T18:56:32.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2021-01-15T18:56:32.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2021-01-15T18:56:32.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2021-01-15T18:56:32.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2021-01-15T18:56:32.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2021-01-15T18:56:32.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2021-01-15T18:56:32.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2021-01-15T18:56:32.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2021-01-15T18:56:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:32.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2021-01-15T18:56:33.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2021-01-15T18:56:33.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2021-01-15T18:56:33.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2021-01-15T18:56:33.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2021-01-15T18:56:33.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2021-01-15T18:56:33.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2021-01-15T18:56:33.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2021-01-15T18:56:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2021-01-15T18:56:33.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2021-01-15T18:56:33.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:33.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2021-01-15T18:56:34.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2021-01-15T18:56:34.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2021-01-15T18:56:34.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2021-01-15T18:56:34.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2021-01-15T18:56:34.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.502 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2021-01-15T18:56:34.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2021-01-15T18:56:34.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2021-01-15T18:56:34.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2021-01-15T18:56:34.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:34.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2021-01-15T18:56:35.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2021-01-15T18:56:35.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2021-01-15T18:56:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2021-01-15T18:56:35.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2021-01-15T18:56:35.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2021-01-15T18:56:35.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2021-01-15T18:56:35.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2021-01-15T18:56:35.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2021-01-15T18:56:35.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2021-01-15T18:56:35.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:35.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2021-01-15T18:56:36.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2021-01-15T18:56:36.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2021-01-15T18:56:36.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2021-01-15T18:56:36.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2021-01-15T18:56:36.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2021-01-15T18:56:36.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2021-01-15T18:56:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2021-01-15T18:56:36.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2021-01-15T18:56:36.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2021-01-15T18:56:36.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:36.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2021-01-15T18:56:37.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2021-01-15T18:56:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2021-01-15T18:56:37.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2021-01-15T18:56:37.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2021-01-15T18:56:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2021-01-15T18:56:37.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2021-01-15T18:56:37.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2021-01-15T18:56:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2021-01-15T18:56:37.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2021-01-15T18:56:37.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:37.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2021-01-15T18:56:38.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2021-01-15T18:56:38.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2021-01-15T18:56:38.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2021-01-15T18:56:38.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2021-01-15T18:56:38.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2021-01-15T18:56:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2021-01-15T18:56:38.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2021-01-15T18:56:38.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2021-01-15T18:56:38.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:38.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2021-01-15T18:56:39.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2021-01-15T18:56:39.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2021-01-15T18:56:39.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2021-01-15T18:56:39.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2021-01-15T18:56:39.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2021-01-15T18:56:39.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2021-01-15T18:56:39.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2021-01-15T18:56:39.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2021-01-15T18:56:39.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2021-01-15T18:56:39.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:39.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2021-01-15T18:56:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2021-01-15T18:56:40.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2021-01-15T18:56:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2021-01-15T18:56:40.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2021-01-15T18:56:40.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2021-01-15T18:56:40.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2021-01-15T18:56:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2021-01-15T18:56:40.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2021-01-15T18:56:40.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2021-01-15T18:56:40.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:40.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2021-01-15T18:56:41.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2021-01-15T18:56:41.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2021-01-15T18:56:41.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2021-01-15T18:56:41.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.397 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2021-01-15T18:56:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2021-01-15T18:56:41.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2021-01-15T18:56:41.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2021-01-15T18:56:41.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2021-01-15T18:56:41.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:56:41.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:56:41.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:56:41.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:56:42.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2021-01-15T18:56:42.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2021-01-15T18:56:42.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:56:42.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:56:42.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:56:42.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:56:42.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2021-01-15T18:56:42.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2021-01-15T18:56:42.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test 2021-01-15T18:56:42.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=obj1 2021-01-15T18:56:42.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test obj1 2021-01-15T18:56:42.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:56:42.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:56:42.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:56:42.533 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:56:42.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=1 2021-01-15T18:56:42.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test obj1 2021-01-15T18:56:42.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-01-15T18:56:43.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2021-01-15T18:56:43.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2021-01-15T18:56:44.264 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:56:44.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2021-01-15T18:56:44.831 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. osd.0 is already out. 2021-01-15T18:56:44.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2021-01-15T18:56:45.615 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 4 2021-01-15T18:56:45.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2021-01-15T18:56:46.483 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:56:46.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2021-01-15T18:56:46.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:56:46.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:56:46.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:56:46.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:56:46.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2021-01-15T18:56:46.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2021-01-15T18:56:47.019 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:56:47.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2021-01-15T18:56:49.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2021-01-15T18:56:49.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:56:49.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:56:49.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:56:49.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:56:49.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:56:49.034 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:56:49.035 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:56:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:56:49.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:56:49.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:56:49.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:56:49.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:56:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:56:49.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:56:49.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:56:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:56:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:56:49.528 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:56:49.529 INFO:tasks.workunit.client.0.smithi145.stderr:3 2021-01-15T18:56:49.529 INFO:tasks.workunit.client.0.smithi145.stderr:4 2021-01-15T18:56:49.529 INFO:tasks.workunit.client.0.smithi145.stderr:5' 2021-01-15T18:56:49.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:56:49.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:49.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:56:49.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836500 2021-01-15T18:56:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836500 2021-01-15T18:56:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500' 2021-01-15T18:56:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:49.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:56:49.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T18:56:49.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T18:56:49.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979' 2021-01-15T18:56:49.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:49.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:56:49.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509457 2021-01-15T18:56:49.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509457 2021-01-15T18:56:49.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457' 2021-01-15T18:56:49.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:49.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:56:50.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345935 2021-01-15T18:56:50.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345935 2021-01-15T18:56:50.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935' 2021-01-15T18:56:50.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:50.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-01-15T18:56:50.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=107374182414 2021-01-15T18:56:50.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 107374182414 2021-01-15T18:56:50.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935 4-107374182414' 2021-01-15T18:56:50.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:56:50.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-01-15T18:56:50.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018893 2021-01-15T18:56:50.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018893 2021-01-15T18:56:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836500 1-42949672979 2-64424509457 3-85899345935 4-107374182414 5-128849018893' 2021-01-15T18:56:50.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:50.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2021-01-15T18:56:50.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:50.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:56:50.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836500 2021-01-15T18:56:50.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:50.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836500 2021-01-15T18:56:50.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2021-01-15T18:56:50.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:56:50.350 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836500 2021-01-15T18:56:50.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836498 -lt 21474836500 2021-01-15T18:56:50.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:56:51.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:56:51.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:56:52.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836500 2021-01-15T18:56:52.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:52.144 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T18:56:52.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:52.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:56:52.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T18:56:52.146 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:52.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T18:56:52.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T18:56:52.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:56:52.148 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T18:56:52.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672979 2021-01-15T18:56:52.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:52.565 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509457 2021-01-15T18:56:52.566 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:52.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:56:52.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509457 2021-01-15T18:56:52.569 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:52.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509457 2021-01-15T18:56:52.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2021-01-15T18:56:52.570 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:56:52.571 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509457 2021-01-15T18:56:52.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509457 -lt 64424509457 2021-01-15T18:56:52.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:52.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345935 2021-01-15T18:56:52.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:52.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:56:52.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-85899345935 2021-01-15T18:56:52.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:52.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345935 2021-01-15T18:56:52.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 85899345935' 2021-01-15T18:56:52.971 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:56:52.972 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 85899345935 2021-01-15T18:56:53.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345935 -lt 85899345935 2021-01-15T18:56:53.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:53.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182414 2021-01-15T18:56:53.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:53.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=4 2021-01-15T18:56:53.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 4-107374182414 2021-01-15T18:56:53.372 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:53.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=107374182414 2021-01-15T18:56:53.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2021-01-15T18:56:53.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 4 2021-01-15T18:56:53.374 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.4 seq 107374182414 2021-01-15T18:56:53.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 107374182414 -lt 107374182414 2021-01-15T18:56:53.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:56:53.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018893 2021-01-15T18:56:53.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:56:53.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=5 2021-01-15T18:56:53.774 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 5-128849018893 2021-01-15T18:56:53.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:56:53.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018893 2021-01-15T18:56:53.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2021-01-15T18:56:53.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 5 2021-01-15T18:56:53.776 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.5 seq 128849018893 2021-01-15T18:56:54.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018893 -lt 128849018893 2021-01-15T18:56:54.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:56:54.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:54.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:54.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:56:54.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:56:54.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:56:54.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:56:54.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:56:54.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:56:54.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:56:54.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:56:55.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:56:55.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:56:55.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:55.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:55.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:56:55.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T18:56:55.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T18:56:55.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T18:56:55.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:56:55.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:56:55.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:56:55.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:56:55.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:56:55.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:56:55.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:56:55.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:56:55.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:56:55.711 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:56:56.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:56:56.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:56:56.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:56.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:56.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:56:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:56:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:56:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:56:56.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:56:56.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:56:56.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:56:56.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:56:56.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:56:57.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:56:57.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:56:57.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:56:57.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:56:57.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:56:57.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:56:57.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:56:57.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:56:57.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:56:57.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:56:57.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:56:57.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:56:57.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:56:57.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:56:57.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:56:57.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:57.641 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:56:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:56:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:56:58.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:56:58.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:56:58.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:56:58.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:56:58.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:56:58.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:56:58.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:56:58.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:56:58.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:56:58.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:56:58.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:56:58.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:56:58.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:56:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:56:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:56:58.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:56:58.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:56:58.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:56:58.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:56:59.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:56:59.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:56:59.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:56:59.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:56:59.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:56:59.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:56:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:56:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:56:59.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:56:59.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:56:59.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:56:59.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:56:59.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:00.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:00.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:00.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:00.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:00.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:00.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:00.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:00.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:00.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:00.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:00.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:00.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:00.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:01.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:01.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:01.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:01.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:01.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:01.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:01.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:01.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:01.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:01.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:01.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:01.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:01.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:01.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:01.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:01.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:01.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:01.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:01.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:01.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:01.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:01.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:02.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:02.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:02.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:02.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:02.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:02.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:02.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:02.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:02.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:02.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:02.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:02.751 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:03.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:03.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:03.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:03.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:03.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:03.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:03.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:03.369 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:03.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:03.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:03.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:03.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:03.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:03.752 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:04.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:04.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:04.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:04.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:04.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:04.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:04.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:04.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:04.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:04.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:04.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:04.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:04.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:04.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:04.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:04.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:04.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:04.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:04.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:04.884 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:05.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:05.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:05.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:05.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:05.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:05.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:05.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:05.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:05.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:05.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:05.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:05.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:05.782 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:06.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:06.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:06.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:06.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:06.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:06.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:06.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:06.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:06.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:06.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:06.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:06.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:06.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:06.788 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:07.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:07.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:07.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:07.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:07.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:07.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:07.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:07.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:07.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:07.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:07.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:07.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:07.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:07.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:07.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:07.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:07.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:07.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:07.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:07.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:07.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:07.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:08.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:08.326 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:08.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:08.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:08.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:08.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:08.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:08.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:08.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:08.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:08.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:08.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:08.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:09.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:09.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:09.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:09.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:09.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:09.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:09.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:09.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:09.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:09.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:09.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:09.889 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:09.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:09.890 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:10.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:10.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:10.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:10.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:10.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:10.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:10.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:10.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:10.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:10.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:11.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:11.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:11.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:11.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:11.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:11.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:11.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:11.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:11.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:11.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:11.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:11.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:11.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:11.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:11.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:11.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:11.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:11.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:11.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:11.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:11.938 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:12.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:12.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:12.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:12.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:12.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:12.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:12.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:12.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:12.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:12.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:12.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:12.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:12.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:12.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:12.941 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:12.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:12.942 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:13.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:13.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:13.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:13.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:13.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:13.458 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:14.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:14.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:14.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:14.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:14.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:14.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:14.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:14.122 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:14.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:14.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:14.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:14.503 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:15.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:15.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:15.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:15.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:15.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:15.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:15.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:15.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:15.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:15.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:15.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:15.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:15.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:15.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:15.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:15.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:15.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:15.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:16.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:16.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:16.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:16.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:16.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:16.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:16.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:16.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:16.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:16.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:16.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:17.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:17.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:17.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:17.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:17.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:17.206 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:17.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:17.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:17.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:17.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:18.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:18.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:18.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:18.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:18.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:18.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:18.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:18.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:18.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:18.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:18.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:18.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:18.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:18.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:18.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:18.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:18.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:18.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:18.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:18.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:18.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:18.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:19.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:19.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:19.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:19.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:19.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:19.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:19.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:19.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:19.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:19.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:19.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:20.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:20.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:20.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:20.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:20.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:20.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:20.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:20.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:20.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:20.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:20.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:20.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:20.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:20.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:20.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:20.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:20.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:21.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:21.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:21.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:21.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:21.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:21.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:21.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:21.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:21.166 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:21.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:21.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:21.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:21.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:21.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:21.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:21.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:21.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:21.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:21.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:21.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:21.791 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:22.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:22.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:22.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:22.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:22.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:22.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:22.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:22.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:22.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:22.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:22.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:23.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:23.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:23.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:23.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:23.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:23.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:23.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:23.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:23.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:23.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:23.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:23.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:23.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:23.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:23.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:23.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:23.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:24.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:24.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:24.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:24.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:24.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:24.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:24.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:24.229 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:24.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:24.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:24.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:24.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:24.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:24.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:24.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:24.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:24.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:24.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:24.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:24.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:24.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:24.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:25.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:25.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:25.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:25.255 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:25.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:25.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:25.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:25.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:25.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:25.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:25.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:25.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:25.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:26.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:26.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:26.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:26.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:26.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:26.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:26.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:26.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:26.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:26.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:26.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:26.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:26.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:26.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:26.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:26.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:26.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:27.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:27.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:27.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:27.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:27.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:27.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:27.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:27.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:27.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:27.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:27.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:27.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:27.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:27.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:27.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:27.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:27.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:27.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:27.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:27.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:27.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:27.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:28.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:28.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:28.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:28.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:28.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:28.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:28.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:28.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:28.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:29.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:29.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:29.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:29.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:29.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:29.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:29.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:29.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:29.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:29.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:29.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:29.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:29.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:29.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:29.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:29.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:29.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:30.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:30.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:30.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:30.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:30.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:30.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:30.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:30.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:30.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:30.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:30.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:30.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:30.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:30.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:30.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:30.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:30.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:30.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:31.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:31.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:31.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:31.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:31.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:31.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:31.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:31.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:31.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:32.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:32.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:32.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:32.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:32.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:32.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:32.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:32.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:32.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:32.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:32.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:32.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:32.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:32.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:32.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:33.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:33.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:33.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:33.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:33.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:33.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:33.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:33.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:33.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:33.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:33.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:33.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:33.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:33.996 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:33.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:33.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:33.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:33.997 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:34.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:34.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:34.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:34.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:34.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:34.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:34.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:34.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:34.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:34.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:34.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:34.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:34.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:35.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:35.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:35.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:35.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:35.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:35.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:35.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:35.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:35.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:35.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:35.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:35.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:35.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:35.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:35.902 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:36.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:36.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:36.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:36.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:36.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:36.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:36.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:37.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:37.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:37.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:37.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:37.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:37.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:37.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:37.037 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:37.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:37.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:37.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:37.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:37.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:37.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:37.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:37.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:37.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:37.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:38.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=3 2021-01-15T18:57:38.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 3 '!=' null 2021-01-15T18:57:38.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:38.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:38.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:38.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:38.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:38.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:38.556 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:38.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:38.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:38.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:38.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:38.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:38.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:38.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:38.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:39.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:39.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:39.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:39.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:39.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:39.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:39.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:39.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:39.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:40.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4 2021-01-15T18:57:40.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4 '!=' null 2021-01-15T18:57:40.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:40.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:40.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:40.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:40.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:40.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:40.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:40.201 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:40.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:40.202 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:40.602 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:40.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:40.602 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:40.603 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:41.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:41.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:41.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:41.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:41.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:41.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:57:41.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:57:41.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:41.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:41.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:41.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:41.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:41.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:41.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:41.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:41.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:41.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:41.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:42.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T18:57:42.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:42.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:42.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:42.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T18:57:42.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T18:57:42.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T18:57:42.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T18:57:42.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T18:57:42.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T18:57:42.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T18:57:42.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T18:57:42.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T18:57:43.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=2 2021-01-15T18:57:43.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 2 '!=' null 2021-01-15T18:57:43.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T18:57:43.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T18:57:43.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T18:57:43.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T18:57:43.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:57:43.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:57:43.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:57:43.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:57:43.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:57:43.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:57:43.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:57:43.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:57:43.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:57:43.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:57:43.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:57:44.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:57:44.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:57:44.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:57:44.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2021-01-15T18:57:44.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2021-01-15T18:57:44.133 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2021-01-15T18:57:44.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2021-01-15T18:57:44.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2021-01-15T18:57:44.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:57:44.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2021-01-15T18:57:44.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-01-15T18:57:44.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2021-01-15T18:57:44.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2021-01-15T18:57:44.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2021-01-15T18:57:44.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:57:44.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2021-01-15T18:57:44.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2021-01-15T18:57:44.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:57:44.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-01-15T18:57:44.138 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:57:44.138 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.139 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:57:44.139 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:57:44.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2021-01-15T18:57:44.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:57:44.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.160 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.161 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-01-15T18:57:44.161 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:57:44.161 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:57:44.161 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2021-01-15T18:57:44.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2021-01-15T18:57:44.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-01-15T18:57:44.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:57:44.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-01-15T18:57:44.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:57:44.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:57:44.165 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2021-01-15T18:57:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2021-01-15T18:57:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:57:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:57:44.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:57:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:57:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:57:44.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.175 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-01-15T18:57:44.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:57:44.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:57:44.177 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2021-01-15T18:57:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2021-01-15T18:57:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2021-01-15T18:57:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:57:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2021-01-15T18:57:44.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:57:44.179 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.179 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:57:44.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:57:44.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:57:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:57:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:57:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:57:44.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:57:44.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2021-01-15T18:57:44.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2021-01-15T18:57:44.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.189 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.189 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2021-01-15T18:57:44.190 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2021-01-15T18:57:44.191 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2021-01-15T18:57:44.191 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2021-01-15T18:57:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2021-01-15T18:57:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2021-01-15T18:57:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:57:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2021-01-15T18:57:44.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:57:44.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:57:44.193 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2021-01-15T18:57:44.193 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2021-01-15T18:57:44.193 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:57:44.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2021-01-15T18:57:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2021-01-15T18:57:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:57:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:57:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:57:44.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:57:44.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2021-01-15T18:57:44.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:57:44.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:57:44.709 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:57:44.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2021-01-15T18:57:44.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:57:44.719 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:57:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:57:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:57:44.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:57:44.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:57:44.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:57:44.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:57:44.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:57:44.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:57:44.831 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:57:44.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:57:44.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:57:44.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:57:44.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:57:44.835 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:57:44.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:57:44.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:57:44.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:57:44.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:57:44.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:57:44.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:57:44.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:57:44.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:57:44.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:57:44.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:57:44.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:57:44.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:57:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:57:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:44.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:44.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:57:44.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:57:44.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:57:44.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2021-01-15T18:57:44.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2021-01-15T18:57:44.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2021-01-15T18:57:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:57:44.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:57:44.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:57:44.864 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:57:44.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:57:44.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:57:44.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:57:44.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:57:44.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:57:44.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:57:44.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:57:44.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:57:44.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:57:44.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:57:44.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:57:44.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:57:44.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:57:44.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:57:44.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:57:44.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:57:44.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:57:44.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:57:44.870 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:44.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:57:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:57:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:57:44.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2021-01-15T18:57:44.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:57:44.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:44.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:44.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.837713 2021-01-15T18:57:44.874 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:57:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T18:57:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:57:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2021-01-15T18:57:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2021-01-15T18:57:44.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2021-01-15T18:57:44.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2021-01-15T18:57:44.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2021-01-15T18:57:44.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:57:44.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:57:44.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:57:44.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2021-01-15T18:57:44.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2021-01-15T18:57:45.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:57:45.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:57:45.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:57:45.178 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:57:45.178 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:45.178 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:45.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:45.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2021-01-15T18:57:45.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:57:45.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:57:45.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:57:45.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:57:45.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:57:45.362 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:57:45.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:57:45.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:57:45.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:57:45.364 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:57:45.364 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:45.365 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:45.365 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:57:45.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:57:45.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get fsid 2021-01-15T18:57:45.464 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:57:45.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:57:45.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:57:45.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:57:45.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:57:45.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:57:45.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:57:45.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:57:45.467 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:57:45.467 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:45.467 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:45.468 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.837713/ceph-mon.a.asok 2021-01-15T18:57:45.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:57:45.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.837713/ceph-mon.a.asok config get mon_host 2021-01-15T18:57:45.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2021-01-15T18:57:45.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-recovery-stats 2021-01-15T18:57:45.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:57:45.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:57:45.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:57:45.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-recovery-stats/x 2021-01-15T18:57:45.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:57:46.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:57:46.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:57:46.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:57:46.456 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:57:46.456 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:46.457 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:46.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:46.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:57:46.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:57:46.478 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2021-01-15T18:57:46.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2021-01-15T18:57:46.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:57:46.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2021-01-15T18:57:46.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:57:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:57:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:57:46.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:57:46.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/0 2021-01-15T18:57:46.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:57:46.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:57:46.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:57:46.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:57:46.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2021-01-15T18:57:46.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2021-01-15T18:57:46.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:57:46.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:57:46.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:57:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:57:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:57:46.490 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:57:46.490 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:57:46.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:46.491 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:46.491 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:46.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:46.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:57:46.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:57:46.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:57:46.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:57:46.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:57:46.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:57:46.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:57:46.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:57:46.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:57:46.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:57:46.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/0 2021-01-15T18:57:46.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:57:46.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=3aee6aef-62fb-4887-a61d-d53f9e52484e 2021-01-15T18:57:46.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 3aee6aef-62fb-4887-a61d-d53f9e52484e' 2021-01-15T18:57:46.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:57:46.496 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 3aee6aef-62fb-4887-a61d-d53f9e52484e 2021-01-15T18:57:46.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCq5QFgmkuhHRAA+TzSNXIp++6Rh7ohsNYQjw== 2021-01-15T18:57:46.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCq5QFgmkuhHRAA+TzSNXIp++6Rh7ohsNYQjw=="}' 2021-01-15T18:57:46.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 3aee6aef-62fb-4887-a61d-d53f9e52484e -i td/osd-recovery-stats/0/new.json 2021-01-15T18:57:46.837 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:57:46.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/0/new.json 2021-01-15T18:57:46.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCq5QFgmkuhHRAA+TzSNXIp++6Rh7ohsNYQjw== --osd-uuid 3aee6aef-62fb-4887-a61d-d53f9e52484e 2021-01-15T18:57:46.871 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.870+0000 7fb6e92a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:46.873 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.872+0000 7fb6e92a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:46.874 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.873+0000 7fb6e92a6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:46.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.874+0000 7fb6e92a6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:57:46.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.874+0000 7fb6e92a6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:57:46.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.874+0000 7fb6e92a6080 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2021-01-15T18:57:46.919 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:46.917+0000 7fb6e92a6080 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2021-01-15T18:57:51.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2021-01-15T18:57:51.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:57:51.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:57:51.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:57:51.673 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:57:52.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:57:52.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:57:52.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:57:52.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:57:52.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:57:52.198 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:57:52.218 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:52.217+0000 7f586d0f1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:52.243 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:52.242+0000 7f586d0f1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:52.244 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:52.242+0000 7f586d0f1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:52.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:57:52.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:57:52.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:57:52.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:57:52.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:57:52.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:57:52.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:57:52.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:57:52.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:57:52.607 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:57:52.762 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:52.761+0000 7f586d0f1080 -1 Falling back to public interface 2021-01-15T18:57:52.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:57:53.544 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:53.543+0000 7f586d0f1080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:57:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:57:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:57:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:57:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:57:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:57:54.000 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:57:54.411 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/928999,v1:127.0.0.1:6803/928999] [v2:127.0.0.1:6804/928999,v1:127.0.0.1:6805/928999] exists,up 3aee6aef-62fb-4887-a61d-d53f9e52484e 2021-01-15T18:57:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:57:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:57:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:57:54.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:57:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2021-01-15T18:57:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:57:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:57:54.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:57:54.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:57:54.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/1 2021-01-15T18:57:54.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:57:54.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:57:54.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:57:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:57:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2021-01-15T18:57:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2021-01-15T18:57:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:57:54.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:57:54.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:57:54.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:57:54.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:57:54.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:57:54.416 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:57:54.417 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:57:54.417 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:57:54.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:54.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:57:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:57:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:57:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:57:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:57:54.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:57:54.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:57:54.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:57:54.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:57:54.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:57:54.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:57:54.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/1 2021-01-15T18:57:54.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:57:54.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=81fca2c9-98fb-4869-b938-c070a7a7ea93 2021-01-15T18:57:54.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 81fca2c9-98fb-4869-b938-c070a7a7ea93' 2021-01-15T18:57:54.421 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:57:54.421 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 81fca2c9-98fb-4869-b938-c070a7a7ea93 2021-01-15T18:57:54.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCy5QFgSOBgGRAAG9JHfMxb5A/pVyLtk7RJNg== 2021-01-15T18:57:54.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCy5QFgSOBgGRAAG9JHfMxb5A/pVyLtk7RJNg=="}' 2021-01-15T18:57:54.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 81fca2c9-98fb-4869-b938-c070a7a7ea93 -i td/osd-recovery-stats/1/new.json 2021-01-15T18:57:55.066 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:57:55.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/1/new.json 2021-01-15T18:57:55.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCy5QFgSOBgGRAAG9JHfMxb5A/pVyLtk7RJNg== --osd-uuid 81fca2c9-98fb-4869-b938-c070a7a7ea93 2021-01-15T18:57:55.103 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.101+0000 7f6ac9ab1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:55.104 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.103+0000 7f6ac9ab1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:55.106 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.105+0000 7f6ac9ab1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:57:55.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.106+0000 7f6ac9ab1080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:57:55.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.106+0000 7f6ac9ab1080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:57:55.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.106+0000 7f6ac9ab1080 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2021-01-15T18:57:55.150 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:57:55.149+0000 7f6ac9ab1080 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2021-01-15T18:57:59.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2021-01-15T18:57:59.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:57:59.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:57:59.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:57:59.984 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:58:00.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:58:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:58:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:58:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:58:00.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:58:00.511 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:58:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:00.529+0000 7f67d4140080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:00.531+0000 7f67d4140080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:00.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:00.532+0000 7f67d4140080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:01.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:58:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:58:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:58:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:58:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:58:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:58:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:58:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:58:01.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:58:01.012 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:58:01.052 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:01.051+0000 7f67d4140080 -1 Falling back to public interface 2021-01-15T18:58:01.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:58:01.882 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:01.881+0000 7f67d4140080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:58:02.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:58:02.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:58:02.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:58:02.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:58:02.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:58:02.436 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:58:02.871 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/930263,v1:127.0.0.1:6811/930263] [v2:127.0.0.1:6812/930263,v1:127.0.0.1:6813/930263] exists,up 81fca2c9-98fb-4869-b938-c070a7a7ea93 2021-01-15T18:58:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:58:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:58:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:58:02.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:58:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2021-01-15T18:58:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-recovery-stats 2021-01-15T18:58:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:58:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:58:02.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:58:02.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-recovery-stats/2 2021-01-15T18:58:02.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2021-01-15T18:58:02.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:58:02.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:58:02.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:58:02.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2021-01-15T18:58:02.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2021-01-15T18:58:02.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:58:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:58:02.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2021-01-15T18:58:02.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:58:02.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:58:02.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:58:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:58:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:58:02.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:58:02.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:58:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' 2021-01-15T18:58:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:58:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:58:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:58:02.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2021-01-15T18:58:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2021-01-15T18:58:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:58:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:58:02.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:58:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:58:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:58:02.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-recovery-stats/2 2021-01-15T18:58:02.880 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:58:02.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=43d9fc14-e954-46d7-b766-4e57113ecf22 2021-01-15T18:58:02.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 43d9fc14-e954-46d7-b766-4e57113ecf22' 2021-01-15T18:58:02.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:58:02.882 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 43d9fc14-e954-46d7-b766-4e57113ecf22 2021-01-15T18:58:02.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQC65QFgR7rMNBAA1c4Ped/GfHR+OUbjOKYoFQ== 2021-01-15T18:58:02.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQC65QFgR7rMNBAA1c4Ped/GfHR+OUbjOKYoFQ=="}' 2021-01-15T18:58:02.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 43d9fc14-e954-46d7-b766-4e57113ecf22 -i td/osd-recovery-stats/2/new.json 2021-01-15T18:58:03.409 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:58:03.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-recovery-stats/2/new.json 2021-01-15T18:58:03.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC65QFgR7rMNBAA1c4Ped/GfHR+OUbjOKYoFQ== --osd-uuid 43d9fc14-e954-46d7-b766-4e57113ecf22 2021-01-15T18:58:03.445 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.444+0000 7fa24a01e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:03.446 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.446+0000 7fa24a01e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:03.448 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.447+0000 7fa24a01e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.448+0000 7fa24a01e080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:58:03.449 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.448+0000 7fa24a01e080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:58:03.450 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.448+0000 7fa24a01e080 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2021-01-15T18:58:03.496 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:03.495+0000 7fa24a01e080 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2021-01-15T18:58:08.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2021-01-15T18:58:08.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:58:08.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:58:08.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:58:08.092 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:58:08.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:58:08.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=951329bf-cca6-4cba-9766-1799a23d5518 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.837713/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:58:08.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:58:08.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:58:08.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:58:08.613 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:58:08.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:08.632+0000 7f007615e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:08.635 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:08.634+0000 7f007615e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:08.636 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:08.635+0000 7f007615e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:58:09.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:58:09.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:58:09.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:58:09.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:58:09.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:58:09.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:58:09.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:58:09.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:58:09.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:58:09.007 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:58:09.152 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:09.151+0000 7f007615e080 -1 Falling back to public interface 2021-01-15T18:58:09.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:58:09.949 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:58:09.948+0000 7f007615e080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:58:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:58:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:58:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:58:10.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:58:10.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:58:10.405 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:58:10.811 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/931527,v1:127.0.0.1:6819/931527] [v2:127.0.0.1:6820/931527,v1:127.0.0.1:6821/931527] exists,up 43d9fc14-e954-46d7-b766-4e57113ecf22 2021-01-15T18:58:10.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:58:10.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:58:10.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:58:10.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2021-01-15T18:58:10.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test 1 1 2021-01-15T18:58:11.451 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' already exists 2021-01-15T18:58:11.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:58:12.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2021-01-15T18:58:13.280 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 1 2021-01-15T18:58:13.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2021-01-15T18:58:13.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:58:13.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:58:13.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:58:13.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:58:13.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:58:13.292 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:58:13.292 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:58:13.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:58:13.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:58:13.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:58:13.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:58:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:58:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:58:13.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:58:13.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:58:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:58:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:58:13.861 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:58:13.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:58:13.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:13.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:58:14.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836486 2021-01-15T18:58:14.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836486 2021-01-15T18:58:14.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486' 2021-01-15T18:58:14.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:14.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:58:14.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672966 2021-01-15T18:58:14.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672966 2021-01-15T18:58:14.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2021-01-15T18:58:14.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:14.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:58:14.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509443 2021-01-15T18:58:14.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509443 2021-01-15T18:58:14.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509443' 2021-01-15T18:58:14.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:14.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2021-01-15T18:58:14.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:14.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:58:14.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836486 2021-01-15T18:58:14.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:14.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836486 2021-01-15T18:58:14.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2021-01-15T18:58:14.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:58:14.304 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836486 2021-01-15T18:58:14.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836486 -lt 21474836486 2021-01-15T18:58:14.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:14.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2021-01-15T18:58:14.689 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:14.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:58:14.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672966 2021-01-15T18:58:14.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:14.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672966 2021-01-15T18:58:14.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2021-01-15T18:58:14.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:58:14.692 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672966 2021-01-15T18:58:15.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672966 -lt 42949672966 2021-01-15T18:58:15.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:15.081 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509443 2021-01-15T18:58:15.082 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:15.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:58:15.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509443 2021-01-15T18:58:15.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:15.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509443 2021-01-15T18:58:15.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-01-15T18:58:15.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:58:15.086 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509443 2021-01-15T18:58:15.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509441 -lt 64424509443 2021-01-15T18:58:15.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:58:16.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:58:16.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:58:16.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509443 -lt 64424509443 2021-01-15T18:58:16.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:58:16.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:58:16.882 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:58:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T18:58:17.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:58:17.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:58:17.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:58:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:58:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:58:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:58:17.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:58:17.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T18:58:17.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:58:17.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:58:17.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:58:18.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T18:58:18.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:58:18.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:58:18.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2021-01-15T18:58:18.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2021-01-15T18:58:18.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2021-01-15T18:58:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2021-01-15T18:58:18.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2021-01-15T18:58:18.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2021-01-15T18:58:18.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2021-01-15T18:58:18.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2021-01-15T18:58:18.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2021-01-15T18:58:18.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2021-01-15T18:58:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:18.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2021-01-15T18:58:19.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2021-01-15T18:58:19.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2021-01-15T18:58:19.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2021-01-15T18:58:19.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2021-01-15T18:58:19.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2021-01-15T18:58:19.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2021-01-15T18:58:19.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2021-01-15T18:58:19.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2021-01-15T18:58:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2021-01-15T18:58:19.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2021-01-15T18:58:19.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2021-01-15T18:58:19.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2021-01-15T18:58:19.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2021-01-15T18:58:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2021-01-15T18:58:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2021-01-15T18:58:19.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:19.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2021-01-15T18:58:20.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2021-01-15T18:58:20.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2021-01-15T18:58:20.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2021-01-15T18:58:20.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2021-01-15T18:58:20.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2021-01-15T18:58:20.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2021-01-15T18:58:20.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2021-01-15T18:58:20.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2021-01-15T18:58:20.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2021-01-15T18:58:20.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2021-01-15T18:58:20.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.777 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2021-01-15T18:58:20.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2021-01-15T18:58:20.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2021-01-15T18:58:20.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:20.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2021-01-15T18:58:21.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2021-01-15T18:58:21.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2021-01-15T18:58:21.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2021-01-15T18:58:21.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2021-01-15T18:58:21.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2021-01-15T18:58:21.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2021-01-15T18:58:21.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2021-01-15T18:58:21.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2021-01-15T18:58:21.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2021-01-15T18:58:21.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2021-01-15T18:58:21.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2021-01-15T18:58:21.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2021-01-15T18:58:21.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2021-01-15T18:58:21.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2021-01-15T18:58:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2021-01-15T18:58:21.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:21.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2021-01-15T18:58:22.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2021-01-15T18:58:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2021-01-15T18:58:22.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2021-01-15T18:58:22.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2021-01-15T18:58:22.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2021-01-15T18:58:22.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2021-01-15T18:58:22.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2021-01-15T18:58:22.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2021-01-15T18:58:22.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2021-01-15T18:58:22.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2021-01-15T18:58:22.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2021-01-15T18:58:22.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2021-01-15T18:58:22.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2021-01-15T18:58:22.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:22.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2021-01-15T18:58:23.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2021-01-15T18:58:23.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2021-01-15T18:58:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2021-01-15T18:58:23.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2021-01-15T18:58:23.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2021-01-15T18:58:23.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2021-01-15T18:58:23.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2021-01-15T18:58:23.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2021-01-15T18:58:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2021-01-15T18:58:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2021-01-15T18:58:23.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2021-01-15T18:58:23.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2021-01-15T18:58:23.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2021-01-15T18:58:23.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2021-01-15T18:58:23.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2021-01-15T18:58:23.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:23.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2021-01-15T18:58:24.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2021-01-15T18:58:24.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2021-01-15T18:58:24.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2021-01-15T18:58:24.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2021-01-15T18:58:24.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2021-01-15T18:58:24.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2021-01-15T18:58:24.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2021-01-15T18:58:24.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2021-01-15T18:58:24.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2021-01-15T18:58:24.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2021-01-15T18:58:24.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2021-01-15T18:58:24.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2021-01-15T18:58:24.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2021-01-15T18:58:24.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2021-01-15T18:58:24.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:24.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2021-01-15T18:58:25.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2021-01-15T18:58:25.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2021-01-15T18:58:25.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2021-01-15T18:58:25.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2021-01-15T18:58:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2021-01-15T18:58:25.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2021-01-15T18:58:25.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2021-01-15T18:58:25.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2021-01-15T18:58:25.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2021-01-15T18:58:25.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2021-01-15T18:58:25.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2021-01-15T18:58:25.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2021-01-15T18:58:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2021-01-15T18:58:25.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2021-01-15T18:58:25.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:25.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2021-01-15T18:58:26.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2021-01-15T18:58:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2021-01-15T18:58:26.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2021-01-15T18:58:26.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2021-01-15T18:58:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2021-01-15T18:58:26.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2021-01-15T18:58:26.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2021-01-15T18:58:26.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2021-01-15T18:58:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2021-01-15T18:58:26.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.828 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2021-01-15T18:58:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2021-01-15T18:58:26.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:26.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2021-01-15T18:58:27.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2021-01-15T18:58:27.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2021-01-15T18:58:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2021-01-15T18:58:27.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2021-01-15T18:58:27.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2021-01-15T18:58:27.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2021-01-15T18:58:27.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2021-01-15T18:58:27.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2021-01-15T18:58:27.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2021-01-15T18:58:27.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2021-01-15T18:58:27.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2021-01-15T18:58:27.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2021-01-15T18:58:27.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2021-01-15T18:58:27.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2021-01-15T18:58:27.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2021-01-15T18:58:27.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:27.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2021-01-15T18:58:28.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2021-01-15T18:58:28.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2021-01-15T18:58:28.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2021-01-15T18:58:28.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2021-01-15T18:58:28.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2021-01-15T18:58:28.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2021-01-15T18:58:28.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.449 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2021-01-15T18:58:28.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2021-01-15T18:58:28.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2021-01-15T18:58:28.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2021-01-15T18:58:28.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2021-01-15T18:58:28.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2021-01-15T18:58:28.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2021-01-15T18:58:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:28.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2021-01-15T18:58:29.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2021-01-15T18:58:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2021-01-15T18:58:29.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2021-01-15T18:58:29.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2021-01-15T18:58:29.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2021-01-15T18:58:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2021-01-15T18:58:29.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2021-01-15T18:58:29.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2021-01-15T18:58:29.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2021-01-15T18:58:29.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2021-01-15T18:58:29.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2021-01-15T18:58:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2021-01-15T18:58:29.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2021-01-15T18:58:29.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2021-01-15T18:58:29.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:29.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2021-01-15T18:58:30.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2021-01-15T18:58:30.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2021-01-15T18:58:30.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2021-01-15T18:58:30.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2021-01-15T18:58:30.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2021-01-15T18:58:30.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2021-01-15T18:58:30.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2021-01-15T18:58:30.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2021-01-15T18:58:30.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2021-01-15T18:58:30.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2021-01-15T18:58:30.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2021-01-15T18:58:30.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2021-01-15T18:58:30.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2021-01-15T18:58:30.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:30.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2021-01-15T18:58:31.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2021-01-15T18:58:31.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2021-01-15T18:58:31.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2021-01-15T18:58:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2021-01-15T18:58:31.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2021-01-15T18:58:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2021-01-15T18:58:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2021-01-15T18:58:31.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2021-01-15T18:58:31.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2021-01-15T18:58:31.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2021-01-15T18:58:31.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2021-01-15T18:58:31.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2021-01-15T18:58:31.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2021-01-15T18:58:31.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2021-01-15T18:58:32.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2021-01-15T18:58:32.136 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:58:32.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:58:32.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:58:32.137 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:58:32.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2021-01-15T18:58:32.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2021-01-15T18:58:32.531 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=test 2021-01-15T18:58:32.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj1 2021-01-15T18:58:32.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map test obj1 2021-01-15T18:58:32.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T18:58:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2021-01-15T18:58:32.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2021-01-15T18:58:33.654 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is set 2021-01-15T18:58:33.664 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2021-01-15T18:58:33.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2021-01-15T18:58:33.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2021-01-15T18:58:33.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2021-01-15T18:58:33.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2021-01-15T18:58:34.205 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T18:58:34.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2021-01-15T18:58:34.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2021-01-15T18:58:34.950 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 4 2021-01-15T18:58:34.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2021-01-15T18:58:35.677 INFO:tasks.workunit.client.0.smithi145.stderr:norecover is unset 2021-01-15T18:58:35.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2021-01-15T18:58:35.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:58:35.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:58:35.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:58:35.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:58:36.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2021-01-15T18:58:36.213 INFO:tasks.workunit.client.0.smithi145.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2021-01-15T18:58:36.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2021-01-15T18:58:46.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2021-01-15T18:58:46.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:58:46.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:58:46.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:58:46.630 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:58:46.630 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T18:58:46.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:58:46.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:46.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:58:46.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836495 2021-01-15T18:58:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836495 2021-01-15T18:58:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495' 2021-01-15T18:58:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:46.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:58:46.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672974 2021-01-15T18:58:46.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672974 2021-01-15T18:58:46.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2021-01-15T18:58:46.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:58:46.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:58:47.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509452 2021-01-15T18:58:47.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509452 2021-01-15T18:58:47.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509452' 2021-01-15T18:58:47.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:47.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2021-01-15T18:58:47.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:47.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:58:47.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836495 2021-01-15T18:58:47.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:47.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836495 2021-01-15T18:58:47.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2021-01-15T18:58:47.055 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836495 2021-01-15T18:58:47.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:58:47.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836493 -lt 21474836495 2021-01-15T18:58:47.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:58:48.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:58:48.448 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:58:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836495 2021-01-15T18:58:48.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:48.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2021-01-15T18:58:48.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:48.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:58:48.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672974 2021-01-15T18:58:48.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:48.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672974 2021-01-15T18:58:48.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2021-01-15T18:58:48.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:58:48.878 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672974 2021-01-15T18:58:49.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672974 -lt 42949672974 2021-01-15T18:58:49.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:58:49.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2021-01-15T18:58:49.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:58:49.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:58:49.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509452 2021-01-15T18:58:49.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:58:49.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509452 2021-01-15T18:58:49.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2021-01-15T18:58:49.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:58:49.273 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509452 2021-01-15T18:58:49.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509452 -lt 64424509452 2021-01-15T18:58:49.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 60 2021-01-15T18:58:49.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:49.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:49.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:50.033 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 1 = 60 ']' 2021-01-15T18:58:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:51.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:51.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:51.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:51.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:51.418 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:51.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 2 = 60 ']' 2021-01-15T18:58:51.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:52.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:52.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:52.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:52.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:52.810 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:53.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 3 = 60 ']' 2021-01-15T18:58:53.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:53.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:54.194 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:54.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 4 = 60 ']' 2021-01-15T18:58:54.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:55.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:55.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:55.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:55.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:55.587 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:55.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 5 = 60 ']' 2021-01-15T18:58:55.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:56.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:56.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:56.969 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:56.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 6 = 60 ']' 2021-01-15T18:58:56.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:57.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:57.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:57.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:57.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:58.354 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:58.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 7 = 60 ']' 2021-01-15T18:58:58.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:58:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:58:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:58:59.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:58:59.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:58:59.736 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:58:59.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 8 = 60 ']' 2021-01-15T18:58:59.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:00.746 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:00.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:00.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:00.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:01.160 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:01.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 9 = 60 ']' 2021-01-15T18:59:01.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:02.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:02.573 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:02.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 10 = 60 ']' 2021-01-15T18:59:02.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:03.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:03.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:03.957 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:03.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 11 = 60 ']' 2021-01-15T18:59:03.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:04.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:04.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:04.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:04.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:05.344 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:05.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 12 = 60 ']' 2021-01-15T18:59:05.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:06.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:06.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:06.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:06.726 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:06.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 13 = 60 ']' 2021-01-15T18:59:06.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:07.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:07.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:08.117 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:08.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 14 = 60 ']' 2021-01-15T18:59:08.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:09.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:09.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:09.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:09.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:09.507 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 15 = 60 ']' 2021-01-15T18:59:09.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2021-01-15T18:59:10.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 60) 2021-01-15T18:59:10.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2021-01-15T18:59:10.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2021-01-15T18:59:10.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2021-01-15T18:59:10.898 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T18:59:10.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2021-01-15T18:59:10.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2021-01-15T18:59:10.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test 2021-01-15T18:59:10.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=obj1 2021-01-15T18:59:10.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test obj1 2021-01-15T18:59:10.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T18:59:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2021-01-15T18:59:11.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2021-01-15T18:59:11.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2021-01-15T18:59:11.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2021-01-15T18:59:11.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2021-01-15T18:59:11.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2021-01-15T18:59:11.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2021-01-15T18:59:11.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2021-01-15T18:59:11.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2021-01-15T18:59:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2021-01-15T18:59:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2021-01-15T18:59:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2021-01-15T18:59:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2021-01-15T18:59:11.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2021-01-15T18:59:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2021-01-15T18:59:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2021-01-15T18:59:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2021-01-15T18:59:11.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2021-01-15T18:59:11.330 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2021-01-15T18:59:11.330 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.330 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2021-01-15T18:59:11.330 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2021-01-15T18:59:11.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2021-01-15T18:59:11.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2021-01-15T18:59:11.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:59:11.338 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.338 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2021-01-15T18:59:11.338 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2021-01-15T18:59:11.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2021-01-15T18:59:11.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:59:11.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2021-01-15T18:59:11.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2021-01-15T18:59:11.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2021-01-15T18:59:11.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:59:11.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2021-01-15T18:59:11.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:59:11.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:59:11.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2021-01-15T18:59:11.344 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:59:11.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2021-01-15T18:59:11.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2021-01-15T18:59:11.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2021-01-15T18:59:11.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:59:11.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2021-01-15T18:59:11.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:59:11.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2021-01-15T18:59:11.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:59:11.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.351 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2021-01-15T18:59:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2021-01-15T18:59:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2021-01-15T18:59:11.352 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:59:11.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2021-01-15T18:59:11.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2021-01-15T18:59:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2021-01-15T18:59:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2021-01-15T18:59:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2021-01-15T18:59:11.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2021-01-15T18:59:11.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2021-01-15T18:59:11.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2021-01-15T18:59:11.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2021-01-15T18:59:11.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2021-01-15T18:59:11.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2021-01-15T18:59:11.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2021-01-15T18:59:11.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2021-01-15T18:59:11.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2021-01-15T18:59:11.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2021-01-15T18:59:11.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2021-01-15T18:59:11.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2021-01-15T18:59:11.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2021-01-15T18:59:11.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test 2021-01-15T18:59:11.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2021-01-15T18:59:11.964 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test' does not exist 2021-01-15T18:59:11.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2021-01-15T18:59:11.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:59:11.975 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:59:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:59:11.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:59:11.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:59:12.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:59:12.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2021-01-15T18:59:12.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:59:12.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:59:12.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:59:12.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:59:12.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:59:12.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:59:12.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:59:12.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:59:12.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:59:12.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:59:12.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:59:12.088 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:59:12.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:59:12.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:59:12.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:59:12.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:59:12.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:59:12.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:59:12.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:59:12.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:59:12.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:59:12.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:59:12.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:59:12.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:59:12.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:59:12.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T18:59:12.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-recovery-stats 0 2021-01-15T18:59:12.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2021-01-15T18:59:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T18:59:12.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2021-01-15T18:59:12.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:59:12.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:59:12.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:59:12.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:59:12.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:59:12.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:59:12.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:59:12.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:59:12.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:59:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:59:12.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:59:12.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:59:12.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:59:12.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:59:12.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:59:12.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T18:59:12.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2021-01-15T18:59:12.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:59:12.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.837713 2021-01-15T18:59:12.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.837713 2021-01-15T18:59:12.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:59:12.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:59:12.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T18:59:12.115 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T18:59:12.115 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T18:59:12.147 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2021-01-15T18:59:12.147 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2021-01-15T18:59:12.207 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T18:59:12.215 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T18:59:12.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:59:12.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T18:59:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T18:59:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T18:59:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T18:59:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T18:59:12.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T18:59:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T18:59:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2021-01-15T18:59:12.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2021-01-15T18:59:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2021-01-15T18:59:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2021-01-15T18:59:12.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2021-01-15T18:59:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none ' 2021-01-15T18:59:12.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2021-01-15T18:59:12.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2021-01-15T18:59:12.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T18:59:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2021-01-15T18:59:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_rados_repair_warning 2021-01-15T18:59:12.221 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_rep_backfill_unfound 2021-01-15T18:59:12.222 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_rep_read_unfound 2021-01-15T18:59:12.222 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_rep_recovery_unfound' 2021-01-15T18:59:12.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2021-01-15T18:59:12.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T18:59:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T18:59:12.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T18:59:12.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T18:59:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T18:59:12.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T18:59:12.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T18:59:12.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T18:59:12.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T18:59:12.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T18:59:12.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T18:59:12.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T18:59:12.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T18:59:12.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T18:59:12.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T18:59:12.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T18:59:12.232 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T18:59:12.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T18:59:12.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T18:59:12.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T18:59:12.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:12.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T18:59:12.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T18:59:12.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T18:59:12.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T18:59:12.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:12.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.937479 2021-01-15T18:59:12.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T18:59:12.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T18:59:12.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T18:59:12.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T18:59:12.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2021-01-15T18:59:12.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2021-01-15T18:59:12.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T18:59:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T18:59:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T18:59:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2021-01-15T18:59:12.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2021-01-15T18:59:12.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T18:59:12.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:12.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:12.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:12.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:12.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:12.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon_osd_warn_num_repaired=10 2021-01-15T18:59:12.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T18:59:12.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T18:59:12.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:59:12.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:59:12.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T18:59:12.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T18:59:12.706 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:59:12.706 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:59:12.707 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:59:12.707 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:59:12.707 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.707 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:12.708 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T18:59:12.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:59:12.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get fsid 2021-01-15T18:59:12.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T18:59:12.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T18:59:12.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T18:59:12.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T18:59:12.807 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T18:59:12.807 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T18:59:12.807 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T18:59:12.807 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T18:59:12.808 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T18:59:12.808 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:12.808 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:12.808 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T18:59:12.808 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T18:59:12.809 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get mon_host 2021-01-15T18:59:12.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2021-01-15T18:59:12.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:12.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T18:59:12.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T18:59:12.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T18:59:12.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2021-01-15T18:59:12.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T18:59:13.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T18:59:13.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:13.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:13.800 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:13.800 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:13.800 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:13.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:13.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:59:13.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T18:59:13.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2021-01-15T18:59:14.132 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T18:59:14.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2021-01-15T18:59:14.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:14.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2021-01-15T18:59:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2021-01-15T18:59:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2021-01-15T18:59:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2021-01-15T18:59:14.143 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2021-01-15T18:59:14.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2021-01-15T18:59:14.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T18:59:14.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2021-01-15T18:59:14.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:14.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:59:14.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T18:59:14.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:59:14.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T18:59:14.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T18:59:14.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:59:14.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:59:14.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:59:14.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T18:59:14.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T18:59:14.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:59:14.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:59:14.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T18:59:14.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:59:14.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:14.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:14.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:14.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:14.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:14.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:14.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:14.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:59:14.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:59:14.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:59:14.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T18:59:14.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T18:59:14.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:59:14.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:59:14.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:14.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:59:14.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:59:14.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T18:59:14.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:59:14.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=46cd3dd3-fc19-491b-98a7-b929296b20b3 2021-01-15T18:59:14.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 46cd3dd3-fc19-491b-98a7-b929296b20b3' 2021-01-15T18:59:14.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:59:14.157 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 46cd3dd3-fc19-491b-98a7-b929296b20b3 2021-01-15T18:59:14.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAC5gFgWj10CRAAhvoBbnWap+eaJo9o/r50SA== 2021-01-15T18:59:14.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAC5gFgWj10CRAAhvoBbnWap+eaJo9o/r50SA=="}' 2021-01-15T18:59:14.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 46cd3dd3-fc19-491b-98a7-b929296b20b3 -i td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T18:59:14.500 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:59:14.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T18:59:14.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAC5gFgWj10CRAAhvoBbnWap+eaJo9o/r50SA== --osd-uuid 46cd3dd3-fc19-491b-98a7-b929296b20b3 2021-01-15T18:59:14.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.532+0000 7f5336e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:14.535 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.534+0000 7f5336e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:14.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.535+0000 7f5336e9e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:14.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.536+0000 7f5336e9e080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T18:59:14.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.536+0000 7f5336e9e080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T18:59:14.537 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.536+0000 7f5336e9e080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T18:59:14.578 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:14.577+0000 7f5336e9e080 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2021-01-15T18:59:19.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2021-01-15T18:59:19.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:59:19.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T18:59:19.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:59:19.179 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T18:59:19.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T18:59:19.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:19.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:59:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:59:19.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:59:19.712 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T18:59:19.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:19.719+0000 7f264a566080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:19.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:19.721+0000 7f264a566080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:19.723 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:19.723+0000 7f264a566080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:20.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T18:59:20.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:59:20.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T18:59:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:59:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:59:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:59:20.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:20.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:59:20.091 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:59:20.247 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:20.246+0000 7f264a566080 -1 Falling back to public interface 2021-01-15T18:59:20.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:21.029 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:21.028+0000 7f264a566080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T18:59:21.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:21.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:21.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:59:21.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:21.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T18:59:21.499 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:59:21.900 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/938547,v1:127.0.0.1:6803/938547] [v2:127.0.0.1:6804/938547,v1:127.0.0.1:6805/938547] exists,up 46cd3dd3-fc19-491b-98a7-b929296b20b3 2021-01-15T18:59:21.900 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 87bd569d-ebec-49a1-8bcc-aafe44544944 2021-01-15T18:59:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:59:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:59:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:59:21.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T18:59:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2021-01-15T18:59:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:59:21.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T18:59:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:59:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T18:59:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T18:59:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:59:21.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:59:21.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:59:21.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T18:59:21.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T18:59:21.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:59:21.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:59:21.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T18:59:21.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:59:21.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:21.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:21.905 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:21.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:21.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:59:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:59:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:59:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T18:59:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T18:59:21.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:59:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:59:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:59:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:59:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T18:59:21.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:59:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=87bd569d-ebec-49a1-8bcc-aafe44544944 2021-01-15T18:59:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 87bd569d-ebec-49a1-8bcc-aafe44544944' 2021-01-15T18:59:21.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:59:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAJ5gFgtRIUNhAANtOm4FC+GLu6tqvBrPkprg== 2021-01-15T18:59:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAJ5gFgtRIUNhAANtOm4FC+GLu6tqvBrPkprg=="}' 2021-01-15T18:59:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 87bd569d-ebec-49a1-8bcc-aafe44544944 -i td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T18:59:22.579 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:59:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T18:59:22.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAJ5gFgtRIUNhAANtOm4FC+GLu6tqvBrPkprg== --osd-uuid 87bd569d-ebec-49a1-8bcc-aafe44544944 2021-01-15T18:59:22.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.611+0000 7feda23fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:22.614 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.613+0000 7feda23fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:22.615 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.614+0000 7feda23fe080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:22.615 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.615+0000 7feda23fe080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T18:59:22.616 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.615+0000 7feda23fe080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T18:59:22.616 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.615+0000 7feda23fe080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T18:59:22.710 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:22.709+0000 7feda23fe080 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2021-01-15T18:59:27.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2021-01-15T18:59:27.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:59:27.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T18:59:27.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:59:27.454 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T18:59:27.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T18:59:27.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:27.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:59:27.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:59:27.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:59:27.977 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T18:59:27.995 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:27.994+0000 7f596b4f3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:28.018 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:28.016+0000 7f596b4f3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:28.018 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:28.017+0000 7f596b4f3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:28.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T18:59:28.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:59:28.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T18:59:28.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:59:28.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:59:28.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:28.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:59:28.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:28.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:59:28.370 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:59:28.534 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:28.533+0000 7f596b4f3080 -1 Falling back to public interface 2021-01-15T18:59:28.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:29.377 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:29.376+0000 7f596b4f3080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T18:59:29.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:29.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:29.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:59:29.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:29.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T18:59:29.754 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:59:30.255 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/939811,v1:127.0.0.1:6811/939811] [v2:127.0.0.1:6812/939811,v1:127.0.0.1:6813/939811] exists,up 87bd569d-ebec-49a1-8bcc-aafe44544944 2021-01-15T18:59:30.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:59:30.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:59:30.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:59:30.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T18:59:30.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2021-01-15T18:59:30.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:30.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:59:30.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T18:59:30.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:59:30.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T18:59:30.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T18:59:30.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:59:30.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:59:30.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:59:30.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T18:59:30.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T18:59:30.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:59:30.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:59:30.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T18:59:30.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:59:30.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:30.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:30.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:30.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:30.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:30.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:59:30.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:59:30.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:59:30.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T18:59:30.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T18:59:30.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:59:30.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:59:30.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:30.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:59:30.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:59:30.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T18:59:30.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:59:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d3180277-0ce9-44ef-a659-588b7fea3280 2021-01-15T18:59:30.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 d3180277-0ce9-44ef-a659-588b7fea3280' 2021-01-15T18:59:30.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:59:30.266 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 d3180277-0ce9-44ef-a659-588b7fea3280 2021-01-15T18:59:30.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAS5gFgEvwTEBAAa2iaLNIPqx/Gw6GfI66oxA== 2021-01-15T18:59:30.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAS5gFgEvwTEBAAa2iaLNIPqx/Gw6GfI66oxA=="}' 2021-01-15T18:59:30.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d3180277-0ce9-44ef-a659-588b7fea3280 -i td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T18:59:31.098 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:59:31.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T18:59:31.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAS5gFgEvwTEBAAa2iaLNIPqx/Gw6GfI66oxA== --osd-uuid d3180277-0ce9-44ef-a659-588b7fea3280 2021-01-15T18:59:31.129 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.128+0000 7f8c37218080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:31.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.130+0000 7f8c37218080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:31.132 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.131+0000 7f8c37218080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.132+0000 7f8c37218080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T18:59:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.132+0000 7f8c37218080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T18:59:31.133 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.132+0000 7f8c37218080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T18:59:31.177 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:31.176+0000 7f8c37218080 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2021-01-15T18:59:35.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2021-01-15T18:59:35.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:59:35.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T18:59:35.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:59:35.949 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T18:59:36.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T18:59:36.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:36.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:59:36.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:59:36.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:59:36.470 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T18:59:36.487 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:36.486+0000 7f776a09a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:36.489 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:36.488+0000 7f776a09a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:36.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:36.489+0000 7f776a09a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:36.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T18:59:36.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:59:36.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T18:59:36.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:59:36.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:59:36.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:36.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:59:36.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:36.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:59:36.861 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:59:37.005 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:37.004+0000 7f776a09a080 -1 Falling back to public interface 2021-01-15T18:59:37.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:37.935 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:37.934+0000 7f776a09a080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T18:59:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:59:38.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:38.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:59:38.247 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:59:38.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:59:39.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:39.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T18:59:39.745 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:59:40.281 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/941083,v1:127.0.0.1:6819/941083] [v2:127.0.0.1:6820/941083,v1:127.0.0.1:6821/941083] exists,up d3180277-0ce9-44ef-a659-588b7fea3280 2021-01-15T18:59:40.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:59:40.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:59:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:59:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T18:59:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2021-01-15T18:59:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T18:59:40.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T18:59:40.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T18:59:40.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T18:59:40.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2021-01-15T18:59:40.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T18:59:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T18:59:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T18:59:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T18:59:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2021-01-15T18:59:40.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2021-01-15T18:59:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T18:59:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T18:59:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T18:59:40.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T18:59:40.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T18:59:40.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T18:59:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T18:59:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T18:59:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T18:59:40.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:40.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T18:59:40.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T18:59:40.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T18:59:40.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T18:59:40.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T18:59:40.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T18:59:40.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T18:59:40.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T18:59:40.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:40.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T18:59:40.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T18:59:40.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2021-01-15T18:59:40.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T18:59:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=349bd717-875d-417c-9384-2e5329e121c2 2021-01-15T18:59:40.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 349bd717-875d-417c-9384-2e5329e121c2' 2021-01-15T18:59:40.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T18:59:40.295 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 349bd717-875d-417c-9384-2e5329e121c2 2021-01-15T18:59:40.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAc5gFgccKgERAA96bwrYCAhHlBTDMz0g/FWw== 2021-01-15T18:59:40.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAc5gFgccKgERAA96bwrYCAhHlBTDMz0g/FWw=="}' 2021-01-15T18:59:40.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 349bd717-875d-417c-9384-2e5329e121c2 -i td/osd-rep-recov-eio.sh/3/new.json 2021-01-15T18:59:40.788 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T18:59:40.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2021-01-15T18:59:40.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAc5gFgccKgERAA96bwrYCAhHlBTDMz0g/FWw== --osd-uuid 349bd717-875d-417c-9384-2e5329e121c2 2021-01-15T18:59:40.903 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.820+0000 7ff53c8cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:40.904 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.822+0000 7ff53c8cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:40.904 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.824+0000 7ff53c8cc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:40.905 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.825+0000 7ff53c8cc080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T18:59:40.905 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.825+0000 7ff53c8cc080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T18:59:40.906 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.825+0000 7ff53c8cc080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T18:59:40.907 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:40.874+0000 7ff53c8cc080 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2021-01-15T18:59:45.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2021-01-15T18:59:45.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T18:59:45.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T18:59:45.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T18:59:45.666 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T18:59:46.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T18:59:46.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T18:59:46.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T18:59:46.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T18:59:46.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T18:59:46.191 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T18:59:46.208 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:46.207+0000 7f602bf1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:46.210 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:46.209+0000 7f602bf1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:46.212 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:46.211+0000 7f602bf1b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T18:59:46.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T18:59:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T18:59:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T18:59:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T18:59:46.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T18:59:46.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:46.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T18:59:46.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:46.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:59:46.579 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T18:59:46.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:46.726+0000 7f602bf1b080 -1 Falling back to public interface 2021-01-15T18:59:46.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:47.545 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T18:59:47.544+0000 7f602bf1b080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T18:59:47.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:47.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:47.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T18:59:47.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:47.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:59:47.980 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T18:59:48.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T18:59:49.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T18:59:49.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T18:59:49.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T18:59:49.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T18:59:49.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T18:59:49.416 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T18:59:50.025 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 27 up_thru 27 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/942373,v1:127.0.0.1:6827/942373] [v2:127.0.0.1:6828/942373,v1:127.0.0.1:6829/942373] exists,up 349bd717-875d-417c-9384-2e5329e121c2 2021-01-15T18:59:50.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T18:59:50.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T18:59:50.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T18:59:50.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2021-01-15T18:59:50.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:59:50.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:59:50.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:59:50.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:59:50.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:59:50.031 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:59:50.031 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:59:50.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:59:50.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:59:50.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:59:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:59:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:59:50.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:59:50.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:59:50.114 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:59:50.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:59:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:59:50.530 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:59:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:59:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:59:50.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:50.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:59:50.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803784 2021-01-15T18:59:50.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803784 2021-01-15T18:59:50.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784' 2021-01-15T18:59:50.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:50.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:59:50.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607560 2021-01-15T18:59:50.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607560 2021-01-15T18:59:50.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607560' 2021-01-15T18:59:50.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:50.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:59:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378629 2021-01-15T18:59:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378629 2021-01-15T18:59:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607560 2-81604378629' 2021-01-15T18:59:50.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:50.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:59:51.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116995 2021-01-15T18:59:51.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116995 2021-01-15T18:59:51.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-51539607560 2-81604378629 3-115964116995' 2021-01-15T18:59:51.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:51.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803784 2021-01-15T18:59:51.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:51.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:59:51.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803784 2021-01-15T18:59:51.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:51.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803784 2021-01-15T18:59:51.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2021-01-15T18:59:51.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:59:51.073 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803784 2021-01-15T18:59:51.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803782 -lt 25769803784 2021-01-15T18:59:51.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:59:52.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:59:52.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:59:52.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803784 2021-01-15T18:59:52.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:52.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607560 2021-01-15T18:59:52.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:52.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:59:52.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607560 2021-01-15T18:59:52.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:52.858 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607560 2021-01-15T18:59:52.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607560 2021-01-15T18:59:52.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2021-01-15T18:59:52.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:59:53.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607560 -lt 51539607560 2021-01-15T18:59:53.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:53.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378629 2021-01-15T18:59:53.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:53.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T18:59:53.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378629 2021-01-15T18:59:53.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:53.285 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378629 2021-01-15T18:59:53.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378629 2021-01-15T18:59:53.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378629' 2021-01-15T18:59:53.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T18:59:53.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378629 2021-01-15T18:59:53.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:53.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964116995 2021-01-15T18:59:53.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:53.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T18:59:53.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964116995 2021-01-15T18:59:53.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:53.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116995 2021-01-15T18:59:53.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964116995' 2021-01-15T18:59:53.726 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964116995 2021-01-15T18:59:53.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T18:59:54.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116995 -lt 115964116995 2021-01-15T18:59:54.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T18:59:54.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:59:54.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:59:54.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T18:59:54.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T18:59:54.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T18:59:54.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T18:59:54.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T18:59:54.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T18:59:54.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T18:59:54.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T18:59:55.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T18:59:55.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T18:59:55.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T18:59:55.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T18:59:55.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T18:59:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T18:59:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T18:59:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2021-01-15T18:59:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2021-01-15T18:59:55.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-rep 1 1 2021-01-15T18:59:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'pool-rep' already exists 2021-01-15T18:59:56.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T18:59:57.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2021-01-15T18:59:57.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T18:59:57.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T18:59:57.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T18:59:57.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T18:59:57.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T18:59:57.105 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T18:59:57.105 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T18:59:57.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T18:59:57.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T18:59:57.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T18:59:57.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T18:59:57.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T18:59:57.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T18:59:57.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T18:59:57.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T18:59:57.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T18:59:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T18:59:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T18:59:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T18:59:57.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T18:59:57.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:57.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T18:59:57.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803788 2021-01-15T18:59:57.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803788 2021-01-15T18:59:57.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788' 2021-01-15T18:59:57.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:57.780 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T18:59:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607563 2021-01-15T18:59:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607563 2021-01-15T18:59:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-51539607563' 2021-01-15T18:59:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:57.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T18:59:58.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378632 2021-01-15T18:59:58.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378632 2021-01-15T18:59:58.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-51539607563 2-81604378632' 2021-01-15T18:59:58.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T18:59:58.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T18:59:58.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964116998 2021-01-15T18:59:58.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964116998 2021-01-15T18:59:58.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-51539607563 2-81604378632 3-115964116998' 2021-01-15T18:59:58.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:58.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803788 2021-01-15T18:59:58.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:58.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T18:59:58.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803788 2021-01-15T18:59:58.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:58.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803788 2021-01-15T18:59:58.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2021-01-15T18:59:58.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:59:58.187 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803788 2021-01-15T18:59:58.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803786 -lt 25769803788 2021-01-15T18:59:58.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T18:59:59.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T18:59:59.575 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T18:59:59.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803788 2021-01-15T18:59:59.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T18:59:59.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607563 2021-01-15T18:59:59.972 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T18:59:59.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T18:59:59.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607563 2021-01-15T18:59:59.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T18:59:59.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607563 2021-01-15T18:59:59.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607563' 2021-01-15T18:59:59.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T18:59:59.978 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607563 2021-01-15T19:00:00.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607563 -lt 51539607563 2021-01-15T19:00:00.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:00.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378632 2021-01-15T19:00:00.391 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:00.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:00.392 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378632 2021-01-15T19:00:00.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:00.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378632 2021-01-15T19:00:00.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2021-01-15T19:00:00.394 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378632 2021-01-15T19:00:00.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:00.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378633 -lt 81604378632 2021-01-15T19:00:00.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:00.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964116998 2021-01-15T19:00:00.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:00.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:00.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964116998 2021-01-15T19:00:00.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:00.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964116998 2021-01-15T19:00:00.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964116998' 2021-01-15T19:00:00.786 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964116998 2021-01-15T19:00:00.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:01.174 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964116999 -lt 115964116998 2021-01-15T19:00:01.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:00:01.175 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:01.176 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:01.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:00:01.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:01.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:01.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:01.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:01.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:01.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:01.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:02.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:00:02.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:02.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:02.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:02.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:00:02.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:00:02.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:00:02.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2021-01-15T19:00:02.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2021-01-15T19:00:02.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2021-01-15T19:00:02.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:02.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2021-01-15T19:00:02.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-937479 2021-01-15T19:00:02.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-937479 2021-01-15T19:00:02.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=pool-rep 2021-01-15T19:00:02.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj-eio-937479 2021-01-15T19:00:02.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:02.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T19:00:02.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2021-01-15T19:00:02.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-937479 2021-01-15T19:00:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:00:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-937479 2021-01-15T19:00:02.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:00:02.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:00:02.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:00:02.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:00:02.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:00:02.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:00:02.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:00:02.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:00:02.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-937479 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:00:03.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 0 2021-01-15T19:00:03.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:03.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:03.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:03.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:03.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:03.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:03.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:03.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:00:03.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:03.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:03.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:03.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:03.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:03.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:03.200 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:03.586 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:03.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:03.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:03.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:00:03.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:03.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:03.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:00:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:00:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:00:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:03.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:00:03.592 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:03.592 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:03.592 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:03.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:03.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:03.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:03.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:03.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:00:03.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:03.699 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:03.700 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:03.700 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:03.700 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:03.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:03.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:03.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:03.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-937479 2021-01-15T19:00:03.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:03.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:00:03.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-937479 2021-01-15T19:00:03.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:00:03.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:00:03.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:00:03.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-937479 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:03.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:04.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:04.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2021-01-15T19:00:04.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:00:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:00:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:00:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:00:04.045 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:00:04.045 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:00:04.046 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:00:04.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:00:04.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:00:04.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:00:04.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:00:04.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:00:04.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:00:04.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:04.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:04.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:04.417 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:04.418 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:04.418 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:04.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:04.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:04.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:04.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803791 2021-01-15T19:00:04.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803791 2021-01-15T19:00:04.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791' 2021-01-15T19:00:04.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:04.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:04.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607566 2021-01-15T19:00:04.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607566 2021-01-15T19:00:04.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-51539607566' 2021-01-15T19:00:04.780 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:04.781 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378636 2021-01-15T19:00:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378636 2021-01-15T19:00:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-51539607566 2-81604378636' 2021-01-15T19:00:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:04.914 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:05.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117002 2021-01-15T19:00:05.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117002 2021-01-15T19:00:05.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-51539607566 2-81604378636 3-115964117002' 2021-01-15T19:00:05.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803791 2021-01-15T19:00:05.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:05.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:05.050 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803791 2021-01-15T19:00:05.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:05.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803791 2021-01-15T19:00:05.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2021-01-15T19:00:05.052 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803791 2021-01-15T19:00:05.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:05.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803791 2021-01-15T19:00:05.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:06.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:06.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:06.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803791 -lt 25769803791 2021-01-15T19:00:06.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:06.844 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607566 2021-01-15T19:00:06.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:06.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:06.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607566 2021-01-15T19:00:06.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:06.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607566 2021-01-15T19:00:06.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607566' 2021-01-15T19:00:06.848 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607566 2021-01-15T19:00:06.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:07.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607566 -lt 51539607566 2021-01-15T19:00:07.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:07.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378636 2021-01-15T19:00:07.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:07.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:07.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378636 2021-01-15T19:00:07.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:07.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378636 2021-01-15T19:00:07.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378636' 2021-01-15T19:00:07.248 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:07.248 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378636 2021-01-15T19:00:07.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378636 -lt 81604378636 2021-01-15T19:00:07.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:07.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117002 2021-01-15T19:00:07.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:07.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:07.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117002 2021-01-15T19:00:07.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:07.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117002 2021-01-15T19:00:07.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117002' 2021-01-15T19:00:07.649 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:07.649 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117002 2021-01-15T19:00:08.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117002 -lt 115964117002 2021-01-15T19:00:08.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:00:08.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:08.052 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:08.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:00:08.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:08.562 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:08.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:08.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:08.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:08.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:08.564 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:08.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:00:08.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:08.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:08.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:09.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:00:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:00:09.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:00:09.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2021-01-15T19:00:09.445 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2021-01-15T19:00:09.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2021-01-15T19:00:09.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2021-01-15T19:00:09.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2021-01-15T19:00:09.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:09.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:09.978 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:09.979 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:09.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:09.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:09.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:10.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803794 2021-01-15T19:00:10.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803794 2021-01-15T19:00:10.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794' 2021-01-15T19:00:10.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:10.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:10.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607569 2021-01-15T19:00:10.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607569 2021-01-15T19:00:10.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-51539607569' 2021-01-15T19:00:10.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:10.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:10.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378639 2021-01-15T19:00:10.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378639 2021-01-15T19:00:10.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-51539607569 2-81604378639' 2021-01-15T19:00:10.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:10.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:10.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117005 2021-01-15T19:00:10.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117005 2021-01-15T19:00:10.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803794 1-51539607569 2-81604378639 3-115964117005' 2021-01-15T19:00:10.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:10.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803794 2021-01-15T19:00:10.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:10.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:10.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803794 2021-01-15T19:00:10.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:10.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803794 2021-01-15T19:00:10.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803794' 2021-01-15T19:00:10.515 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803794 2021-01-15T19:00:10.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:10.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803792 -lt 25769803794 2021-01-15T19:00:10.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:11.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:11.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:12.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803794 -lt 25769803794 2021-01-15T19:00:12.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:12.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607569 2021-01-15T19:00:12.304 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:12.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:12.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607569 2021-01-15T19:00:12.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:12.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607569 2021-01-15T19:00:12.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607569' 2021-01-15T19:00:12.307 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607569 2021-01-15T19:00:12.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:12.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607570 -lt 51539607569 2021-01-15T19:00:12.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:12.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378639 2021-01-15T19:00:12.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:12.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:12.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378639 2021-01-15T19:00:12.726 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:12.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378639 2021-01-15T19:00:12.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378639' 2021-01-15T19:00:12.728 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378639 2021-01-15T19:00:12.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:13.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378639 -lt 81604378639 2021-01-15T19:00:13.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:13.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117005 2021-01-15T19:00:13.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:13.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:13.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117005 2021-01-15T19:00:13.132 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:13.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117005 2021-01-15T19:00:13.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117005' 2021-01-15T19:00:13.134 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117005 2021-01-15T19:00:13.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:13.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117005 -lt 115964117005 2021-01-15T19:00:13.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2021-01-15T19:00:13.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-01-15T19:00:13.939 INFO:tasks.workunit.client.0.smithi145.stderr:dumped all 2021-01-15T19:00:13.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2021-01-15T19:00:13.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2021-01-15T19:00:13.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:13.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:13.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:13.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:13.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:13.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:14.344 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:14.344 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:14.344 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:14.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:14.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2021-01-15T19:00:14.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2021-01-15T19:00:14.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2021-01-15T19:00:14.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 0 2021-01-15T19:00:14.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:14.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:14.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:14.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:14.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:00:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:14.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:14.348 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:14.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:14.349 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:14.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:14.349 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:14.758 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:14.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:14.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:00:14.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:00:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:00:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:14.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:14.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:00:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:14.763 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:14.764 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:14.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:14.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:14.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:14.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:14.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:00:14.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:14.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:14.871 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:14.871 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:14.872 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:14.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:14.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:14.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:14.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 1 2021-01-15T19:00:14.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:14.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:14.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:14.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:14.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:14.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:14.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:00:14.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:14.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:14.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:14.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:14.983 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:14.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:14.984 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:15.373 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:15.374 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:15.374 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:15.374 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:00:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:15.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:15.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:00:15.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:15.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:00:15.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:15.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:00:15.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:15.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:15.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:00:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:00:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:00:15.380 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:15.381 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:15.382 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:00:15.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:15.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:15.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:15.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:15.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:00:15.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:00:15.495 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:00:15.496 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:15.496 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:15.496 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:15.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:00:15.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:15.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:15.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:00:15.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:15.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:00:15.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:00:15.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:00:15.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:15.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:00:15.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:15.602 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:15.602 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:15.603 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:15.603 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:15.604 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:15.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:00:15.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:15.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-937479 2021-01-15T19:00:15.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:00:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-937479 2021-01-15T19:00:15.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:00:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:00:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:00:15.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-937479 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:15.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:15.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:15.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2021-01-15T19:00:15.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:00:15.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:00:15.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:00:15.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:00:15.943 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:00:15.944 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:00:15.944 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:00:15.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:00:15.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:00:15.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:00:16.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:00:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:00:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:00:16.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:16.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:16.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:16.414 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:16.415 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:16.415 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:16.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:16.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:16.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:16.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803797 2021-01-15T19:00:16.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803797 2021-01-15T19:00:16.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797' 2021-01-15T19:00:16.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:16.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:16.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607573 2021-01-15T19:00:16.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607573 2021-01-15T19:00:16.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-51539607573' 2021-01-15T19:00:16.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:16.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:16.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378642 2021-01-15T19:00:16.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378642 2021-01-15T19:00:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-51539607573 2-81604378642' 2021-01-15T19:00:16.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:16.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117008 2021-01-15T19:00:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117008 2021-01-15T19:00:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803797 1-51539607573 2-81604378642 3-115964117008' 2021-01-15T19:00:16.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:16.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803797 2021-01-15T19:00:16.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:16.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:16.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803797 2021-01-15T19:00:16.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:16.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803797 2021-01-15T19:00:16.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2021-01-15T19:00:16.952 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803797 2021-01-15T19:00:16.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:17.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803795 -lt 25769803797 2021-01-15T19:00:17.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:18.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:18.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:18.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803798 -lt 25769803797 2021-01-15T19:00:18.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:18.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607573 2021-01-15T19:00:18.738 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:18.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:18.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607573 2021-01-15T19:00:18.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:18.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607573 2021-01-15T19:00:18.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2021-01-15T19:00:18.741 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607573 2021-01-15T19:00:18.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:19.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607573 -lt 51539607573 2021-01-15T19:00:19.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:19.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378642 2021-01-15T19:00:19.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:19.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:19.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378642 2021-01-15T19:00:19.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:19.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378642 2021-01-15T19:00:19.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378642' 2021-01-15T19:00:19.174 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378642 2021-01-15T19:00:19.174 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:19.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378642 -lt 81604378642 2021-01-15T19:00:19.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:19.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117008 2021-01-15T19:00:19.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:19.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117008 2021-01-15T19:00:19.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:19.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117008 2021-01-15T19:00:19.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117008' 2021-01-15T19:00:19.597 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117008 2021-01-15T19:00:19.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:20.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117009 -lt 115964117008 2021-01-15T19:00:20.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:00:20.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:20.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:20.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:00:20.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:20.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:20.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:20.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:20.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:20.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:20.514 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:20.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:00:20.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:20.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:20.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:21.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:00:21.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:00:21.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 0 2021-01-15T19:00:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:21.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:21.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:21.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:21.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:00:21.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:21.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:21.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:21.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:21.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:21.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:21.413 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:21.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:21.797 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:21.798 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:21.798 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:21.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:21.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:00:21.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:21.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:21.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:00:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:00:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:21.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:00:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:21.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:21.802 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:00:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:21.803 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:21.804 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:21.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:21.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:21.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:21.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:21.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:00:21.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:21.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:21.911 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:21.911 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:21.911 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:21.912 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:21.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:22.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 2 2021-01-15T19:00:22.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:22.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:22.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:22.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:22.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:22.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:22.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:22.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:22.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:22.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=2 2021-01-15T19:00:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:22.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:22.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:22.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:22.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:22.025 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:22.025 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:22.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:22.413 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:22.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:22.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=0 2021-01-15T19:00:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:22.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:22.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2021-01-15T19:00:22.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:22.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-01-15T19:00:22.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:22.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=0 2021-01-15T19:00:22.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:22.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:22.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:22.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.0 2021-01-15T19:00:22.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T19:00:22.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T19:00:22.420 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:22.421 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.0.asok 2021-01-15T19:00:22.422 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:22.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:22.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:22.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.0 2021-01-15T19:00:22.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T19:00:22.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T19:00:22.527 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:22.527 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:22.527 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:22.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.0.asok 2021-01-15T19:00:22.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:22.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.0.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:22.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-937479 2021-01-15T19:00:22.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:22.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:00:22.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-937479 2021-01-15T19:00:22.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:00:22.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:00:22.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:00:22.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-937479 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:22.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:22.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:22.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2021-01-15T19:00:22.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:00:22.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:00:22.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:00:22.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:00:22.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:00:22.765 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:00:22.765 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:00:22.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:00:22.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:00:22.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:00:22.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:00:22.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:00:22.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:00:22.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:22.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:23.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:23.262 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:23.263 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:23.263 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:23.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:23.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:23.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803801 2021-01-15T19:00:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803801 2021-01-15T19:00:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801' 2021-01-15T19:00:23.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:23.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:23.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607576 2021-01-15T19:00:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607576 2021-01-15T19:00:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-51539607576' 2021-01-15T19:00:23.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:23.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:23.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378645 2021-01-15T19:00:23.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378645 2021-01-15T19:00:23.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-51539607576 2-81604378645' 2021-01-15T19:00:23.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:23.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:23.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117011 2021-01-15T19:00:23.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117011 2021-01-15T19:00:23.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803801 1-51539607576 2-81604378645 3-115964117011' 2021-01-15T19:00:23.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:23.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803801 2021-01-15T19:00:23.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:23.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:23.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803801 2021-01-15T19:00:23.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:23.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803801 2021-01-15T19:00:23.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2021-01-15T19:00:23.822 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803801 2021-01-15T19:00:23.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:24.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803801 -lt 25769803801 2021-01-15T19:00:24.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:24.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607576 2021-01-15T19:00:24.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:24.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:24.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607576 2021-01-15T19:00:24.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:24.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607576 2021-01-15T19:00:24.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607576' 2021-01-15T19:00:24.210 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607576 2021-01-15T19:00:24.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:24.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607576 -lt 51539607576 2021-01-15T19:00:24.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:24.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378645 2021-01-15T19:00:24.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:24.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:24.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378645 2021-01-15T19:00:24.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:24.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378645 2021-01-15T19:00:24.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378645' 2021-01-15T19:00:24.623 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:24.624 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378645 2021-01-15T19:00:25.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378643 -lt 81604378645 2021-01-15T19:00:25.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:26.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:26.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:26.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378646 -lt 81604378645 2021-01-15T19:00:26.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:26.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117011 2021-01-15T19:00:26.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:26.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:26.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117011 2021-01-15T19:00:26.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:26.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117011 2021-01-15T19:00:26.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117011' 2021-01-15T19:00:26.416 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117011 2021-01-15T19:00:26.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:26.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117012 -lt 115964117011 2021-01-15T19:00:26.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:00:26.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:26.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:27.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:00:27.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:27.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:27.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:27.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:27.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:27.346 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:27.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:27.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:00:27.727 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:27.728 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:27.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:28.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:00:28.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:00:28.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:00:28.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2021-01-15T19:00:28.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2021-01-15T19:00:28.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2021-01-15T19:00:28.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2021-01-15T19:00:28.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2021-01-15T19:00:28.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:28.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:28.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:28.769 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:28.770 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:28.770 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:28.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:28.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:28.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:28.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803804 2021-01-15T19:00:28.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803804 2021-01-15T19:00:28.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803804' 2021-01-15T19:00:28.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:28.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607579 2021-01-15T19:00:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607579 2021-01-15T19:00:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803804 1-51539607579' 2021-01-15T19:00:29.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:29.046 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:29.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378648 2021-01-15T19:00:29.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378648 2021-01-15T19:00:29.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803804 1-51539607579 2-81604378648' 2021-01-15T19:00:29.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:29.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:29.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117015 2021-01-15T19:00:29.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117015 2021-01-15T19:00:29.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803804 1-51539607579 2-81604378648 3-115964117015' 2021-01-15T19:00:29.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:29.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803804 2021-01-15T19:00:29.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:29.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:29.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803804 2021-01-15T19:00:29.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:29.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803804 2021-01-15T19:00:29.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2021-01-15T19:00:29.322 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803804 2021-01-15T19:00:29.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:29.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803802 -lt 25769803804 2021-01-15T19:00:29.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:30.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:30.724 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:31.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803804 -lt 25769803804 2021-01-15T19:00:31.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:31.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607579 2021-01-15T19:00:31.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:31.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:31.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607579 2021-01-15T19:00:31.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:31.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607579 2021-01-15T19:00:31.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607579' 2021-01-15T19:00:31.127 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:31.128 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607579 2021-01-15T19:00:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607579 -lt 51539607579 2021-01-15T19:00:31.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:31.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378648 2021-01-15T19:00:31.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:31.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:31.538 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378648 2021-01-15T19:00:31.539 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:31.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378648 2021-01-15T19:00:31.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378648' 2021-01-15T19:00:31.540 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378648 2021-01-15T19:00:31.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:31.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378649 -lt 81604378648 2021-01-15T19:00:31.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:31.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117015 2021-01-15T19:00:31.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:31.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:31.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117015 2021-01-15T19:00:31.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117015 2021-01-15T19:00:31.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117015' 2021-01-15T19:00:31.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:31.937 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117015 2021-01-15T19:00:32.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117015 -lt 115964117015 2021-01-15T19:00:32.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2021-01-15T19:00:32.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-01-15T19:00:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:dumped all 2021-01-15T19:00:32.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2021-01-15T19:00:32.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2021-01-15T19:00:32.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 0 2021-01-15T19:00:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:32.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:32.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:32.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:32.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:32.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:32.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:32.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:32.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:00:32.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:32.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:32.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:32.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:32.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:32.730 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:32.731 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:33.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:33.120 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:33.120 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:33.120 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:33.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:33.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:00:33.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:33.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:33.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:00:33.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:33.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:00:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:00:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:33.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:33.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:33.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:00:33.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:33.125 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:33.125 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:33.126 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:33.126 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:33.126 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:33.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:33.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:33.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:33.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:33.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:00:33.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:00:33.235 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:00:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:33.236 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:33.236 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:00:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:33.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 1 2021-01-15T19:00:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:33.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:33.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:33.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:33.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:33.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:00:33.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:33.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:33.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:33.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:33.347 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:33.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:33.348 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:33.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:33.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:33.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:33.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:00:33.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:33.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:33.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:00:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:00:33.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:00:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:33.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:33.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:00:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:00:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:00:33.744 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:33.745 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:00:33.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:33.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:33.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:33.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:33.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:00:33.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:00:33.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:00:33.849 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:33.849 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:33.849 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:33.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:00:33.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:33.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-937479 td/osd-rep-recov-eio.sh 2 2021-01-15T19:00:33.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:00:33.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:00:33.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:00:33.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:00:33.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:00:33.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:00:33.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-eio-937479 2021-01-15T19:00:33.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:00:33.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:33.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:00:33.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=2 2021-01-15T19:00:33.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:00:33.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:00:33.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-eio-937479 2021-01-15T19:00:33.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:00:33.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-eio-937479 2021-01-15T19:00:33.966 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-eio-937479 2021-01-15T19:00:33.966 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:00:34.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:00:34.352 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:34.353 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:00:34.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:00:34.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:00:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=0 2021-01-15T19:00:34.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:00:34.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:00:34.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2021-01-15T19:00:34.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:00:34.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2021-01-15T19:00:34.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:00:34.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=0 2021-01-15T19:00:34.359 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:00:34.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:00:34.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:00:34.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.0 2021-01-15T19:00:34.361 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T19:00:34.361 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T19:00:34.361 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:34.362 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:34.362 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:34.362 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.0.asok 2021-01-15T19:00:34.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:00:34.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:00:34.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:00:34.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:00:34.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.0 2021-01-15T19:00:34.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-01-15T19:00:34.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-01-15T19:00:34.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:00:34.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:00:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:00:34.467 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.0.asok 2021-01-15T19:00:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:00:34.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.0.asok injectdataerr pool-rep obj-eio-937479 2021-01-15T19:00:34.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-937479 hang 2021-01-15T19:00:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:00:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:00:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-937479 2021-01-15T19:00:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2021-01-15T19:00:34.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2021-01-15T19:00:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2021-01-15T19:00:34.576 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-937479 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:00:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2021-01-15T19:00:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2021-01-15T19:00:39.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2021-01-15T19:00:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:00:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:00:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:00:39.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:00:39.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:00:39.580 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:00:39.580 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:00:39.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:00:39.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:00:39.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:00:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:00:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:00:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:00:39.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:00:39.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:00:40.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:00:40.076 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:00:40.076 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:00:40.077 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:00:40.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:00:40.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:40.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:00:40.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803808 2021-01-15T19:00:40.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803808 2021-01-15T19:00:40.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803808' 2021-01-15T19:00:40.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:40.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:00:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607583 2021-01-15T19:00:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607583 2021-01-15T19:00:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803808 1-51539607583' 2021-01-15T19:00:40.344 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:40.345 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:00:40.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378653 2021-01-15T19:00:40.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378653 2021-01-15T19:00:40.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803808 1-51539607583 2-81604378653' 2021-01-15T19:00:40.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:00:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:00:40.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117019 2021-01-15T19:00:40.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117019 2021-01-15T19:00:40.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803808 1-51539607583 2-81604378653 3-115964117019' 2021-01-15T19:00:40.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:40.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803808 2021-01-15T19:00:40.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:40.614 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:00:40.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803808 2021-01-15T19:00:40.615 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:40.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803808 2021-01-15T19:00:40.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2021-01-15T19:00:40.616 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803808 2021-01-15T19:00:40.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:41.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803806 -lt 25769803808 2021-01-15T19:00:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:00:42.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:00:42.004 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:00:42.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803808 -lt 25769803808 2021-01-15T19:00:42.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:42.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607583 2021-01-15T19:00:42.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:42.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:00:42.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607583 2021-01-15T19:00:42.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:42.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607583 2021-01-15T19:00:42.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607583' 2021-01-15T19:00:42.403 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607583 2021-01-15T19:00:42.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:00:42.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607584 -lt 51539607583 2021-01-15T19:00:42.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:42.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378653 2021-01-15T19:00:42.823 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:42.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:00:42.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378653 2021-01-15T19:00:42.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:42.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378653 2021-01-15T19:00:42.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378653' 2021-01-15T19:00:42.826 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378653 2021-01-15T19:00:42.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:00:43.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378653 -lt 81604378653 2021-01-15T19:00:43.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:00:43.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117019 2021-01-15T19:00:43.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:00:43.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:00:43.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117019 2021-01-15T19:00:43.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:00:43.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117019 2021-01-15T19:00:43.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117019' 2021-01-15T19:00:43.249 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117019 2021-01-15T19:00:43.249 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:00:43.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117019 -lt 115964117019 2021-01-15T19:00:43.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:00:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:43.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:44.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:00:44.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:44.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:44.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:44.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:44.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:44.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:44.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:44.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:44.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:44.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:44.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:45.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' -1 2021-01-15T19:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T19:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=8 2021-01-15T19:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:45.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:00:45.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:45.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:45.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:45.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:45.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:45.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:45.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:45.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:45.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:45.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:45.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:45.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:46.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:46.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:46.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:46.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:46.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:46.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:46.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:46.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:46.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:46.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:46.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:46.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 1 >= 13 )) 2021-01-15T19:00:46.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:46.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.2 2021-01-15T19:00:46.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:46.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:46.826 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:46.827 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:46.828 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:47.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:47.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:47.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:47.209 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:47.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:47.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:47.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:47.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:47.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:47.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:47.725 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:48.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:48.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 2 >= 13 )) 2021-01-15T19:00:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:48.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.4 2021-01-15T19:00:48.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:48.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:48.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:48.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:48.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:48.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:48.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:48.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:49.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:49.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:49.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:49.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:49.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:49.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:49.537 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:49.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:49.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:50.041 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 3 >= 13 )) 2021-01-15T19:00:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:50.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.8 2021-01-15T19:00:50.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:50.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:50.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:50.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:50.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:50.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:50.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:50.846 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:51.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:51.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:51.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:51.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:51.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:51.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:51.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:51.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:51.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:51.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:51.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:51.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:52.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:52.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:52.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 4 >= 13 )) 2021-01-15T19:00:52.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:52.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 1.6 2021-01-15T19:00:53.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:53.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:53.861 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:53.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:53.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:53.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:53.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:53.863 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:54.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:54.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:54.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:54.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:54.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:54.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:54.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:54.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:54.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:54.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:54.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:54.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:54.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 5 >= 13 )) 2021-01-15T19:00:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:55.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 3.2 2021-01-15T19:00:58.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:00:58.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:00:58.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:00:58.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:00:58.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:00:58.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:00:58.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:00:58.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:00:58.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:00:58.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:00:58.866 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:00:58.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:00:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:00:59.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:00:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:00:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:00:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:00:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:00:59.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:00:59.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:00:59.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:00:59.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:00:59.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:00:59.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 6 >= 13 )) 2021-01-15T19:00:59.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:00:59.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 6.4 2021-01-15T19:01:06.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:01:06.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:01:06.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:01:06.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:01:06.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:01:06.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:01:06.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:01:06.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:01:06.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:01:06.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:01:06.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:01:06.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:01:07.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:01:07.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:01:07.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:01:07.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:01:07.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:01:07.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:01:07.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:01:07.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:01:07.199 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:01:07.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:01:07.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:01:07.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 7 >= 13 )) 2021-01-15T19:01:07.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:01:07.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 12.8 2021-01-15T19:01:20.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:01:20.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:01:20.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:01:20.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:01:20.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:01:20.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:01:20.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:01:20.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:01:20.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:01:20.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:01:20.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:01:20.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:01:21.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:01:21.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:01:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:01:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:01:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:01:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:01:21.448 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:01:21.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:01:21.449 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:01:21.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:01:21.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:01:21.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 8 >= 13 )) 2021-01-15T19:01:21.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:01:21.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 15 2021-01-15T19:01:36.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:01:36.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:01:36.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:01:36.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:01:36.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:01:36.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:01:36.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:01:36.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:01:37.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:01:37.350 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:01:37.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:01:37.351 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:01:37.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:01:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:01:37.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:01:37.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:01:37.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:01:37.882 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:01:37.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:01:37.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:01:37.883 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:01:38.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:01:38.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:01:38.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 9 >= 13 )) 2021-01-15T19:01:38.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:01:38.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 15 2021-01-15T19:01:53.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:01:53.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:01:53.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:01:53.396 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:01:53.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:01:53.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:01:53.397 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:01:53.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:01:53.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:01:53.783 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:01:53.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:01:53.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:01:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:01:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:01:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:01:54.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:01:54.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:01:54.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:01:54.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:01:54.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:01:54.303 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:01:54.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:01:54.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:01:54.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 10 >= 13 )) 2021-01-15T19:01:54.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:01:54.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 15 2021-01-15T19:02:09.830 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:02:09.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:02:09.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:02:09.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:02:09.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:02:09.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:02:09.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:02:09.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:02:10.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:02:10.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:02:10.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:02:10.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:02:10.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:02:10.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:02:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:02:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:02:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:02:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:02:10.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:02:10.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:02:10.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:02:11.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:02:11.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:02:11.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 11 >= 13 )) 2021-01-15T19:02:11.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:02:11.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 15 2021-01-15T19:02:26.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:02:26.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:02:26.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:02:26.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:02:26.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:02:26.239 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:02:26.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:02:26.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:02:26.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:02:26.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:02:26.625 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:02:26.626 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:02:27.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:02:27.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:02:27.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:02:27.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:02:27.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:02:27.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:02:27.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:02:27.139 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:02:27.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:02:27.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:02:27.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:02:27.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 12 >= 13 )) 2021-01-15T19:02:27.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:02:27.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 4.5 2021-01-15T19:02:32.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:02:32.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:02:32.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:02:32.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:02:32.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:02:32.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:02:32.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:02:32.157 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:02:32.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:02:32.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:02:32.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:02:32.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:02:33.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:02:33.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:02:33.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:02:33.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:02:33.075 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:02:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=null 2021-01-15T19:02:33.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test null '!=' null 2021-01-15T19:02:33.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: (( 13 >= 13 )) 2021-01-15T19:02:33.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: ceph report 2021-01-15T19:02:34.132 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T19:02:34.132 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_fingerprint": "c218759a-49a7-46bd-b60a-88585e0ea322", 2021-01-15T19:02:34.133 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "16.0.0-9031-g892c51dd", 2021-01-15T19:02:34.133 INFO:tasks.workunit.client.0.smithi145.stdout: "commit": "892c51dd3c5f7108e766bea30cd5e0d801b0abd3", 2021-01-15T19:02:34.133 INFO:tasks.workunit.client.0.smithi145.stdout: "timestamp": "2021-01-15T19:02:34.130755+0000", 2021-01-15T19:02:34.133 INFO:tasks.workunit.client.0.smithi145.stdout: "tag": "", 2021-01-15T19:02:34.134 INFO:tasks.workunit.client.0.smithi145.stdout: "health": { 2021-01-15T19:02:34.134 INFO:tasks.workunit.client.0.smithi145.stdout: "status": "HEALTH_ERR", 2021-01-15T19:02:34.134 INFO:tasks.workunit.client.0.smithi145.stdout: "checks": { 2021-01-15T19:02:34.134 INFO:tasks.workunit.client.0.smithi145.stdout: "OBJECT_UNFOUND": { 2021-01-15T19:02:34.134 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T19:02:34.135 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T19:02:34.135 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1/1 objects unfound (100.000%)", 2021-01-15T19:02:34.135 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T19:02:34.135 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.135 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T19:02:34.136 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.136 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 2.0 has 1 unfound objects" 2021-01-15T19:02:34.136 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.137 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.137 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T19:02:34.137 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.137 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DAMAGED": { 2021-01-15T19:02:34.137 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_ERR", 2021-01-15T19:02:34.138 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T19:02:34.138 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2021-01-15T19:02:34.138 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T19:02:34.138 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.138 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T19:02:34.139 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.139 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2021-01-15T19:02:34.139 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.139 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.140 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T19:02:34.140 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.140 INFO:tasks.workunit.client.0.smithi145.stdout: "PG_DEGRADED": { 2021-01-15T19:02:34.140 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T19:02:34.140 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T19:02:34.141 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2021-01-15T19:02:34.141 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T19:02:34.141 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.141 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T19:02:34.141 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.142 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2021-01-15T19:02:34.142 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.142 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.142 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T19:02:34.142 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.143 INFO:tasks.workunit.client.0.smithi145.stdout: "POOL_APP_NOT_ENABLED": { 2021-01-15T19:02:34.143 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T19:02:34.143 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T19:02:34.143 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 pool(s) do not have an application enabled", 2021-01-15T19:02:34.143 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T19:02:34.144 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.144 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [ 2021-01-15T19:02:34.144 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.144 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "application not enabled on pool 'pool-rep'" 2021-01-15T19:02:34.144 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.145 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.145 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2021-01-15T19:02:34.145 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.145 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.145 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T19:02:34.146 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.146 INFO:tasks.workunit.client.0.smithi145.stdout: "SLOW_OPS": { 2021-01-15T19:02:34.146 INFO:tasks.workunit.client.0.smithi145.stdout: "severity": "HEALTH_WARN", 2021-01-15T19:02:34.146 INFO:tasks.workunit.client.0.smithi145.stdout: "summary": { 2021-01-15T19:02:34.147 INFO:tasks.workunit.client.0.smithi145.stdout: "message": "1 slow ops, oldest one blocked for 113 sec, osd.3 has slow ops", 2021-01-15T19:02:34.147 INFO:tasks.workunit.client.0.smithi145.stdout: "count": 1 2021-01-15T19:02:34.147 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.147 INFO:tasks.workunit.client.0.smithi145.stdout: "detail": [], 2021-01-15T19:02:34.148 INFO:tasks.workunit.client.0.smithi145.stdout: "muted": false 2021-01-15T19:02:34.148 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.148 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.148 INFO:tasks.workunit.client.0.smithi145.stdout: "mutes": [] 2021-01-15T19:02:34.149 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.149 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_first_committed": 1, 2021-01-15T19:02:34.149 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap_last_committed": 1, 2021-01-15T19:02:34.149 INFO:tasks.workunit.client.0.smithi145.stdout: "monmap": { 2021-01-15T19:02:34.149 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T19:02:34.150 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e", 2021-01-15T19:02:34.150 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T18:59:12.245717Z", 2021-01-15T19:02:34.150 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T18:59:12.245717Z", 2021-01-15T19:02:34.150 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release": 16, 2021-01-15T19:02:34.150 INFO:tasks.workunit.client.0.smithi145.stdout: "min_mon_release_name": "pacific", 2021-01-15T19:02:34.151 INFO:tasks.workunit.client.0.smithi145.stdout: "election_strategy": 1, 2021-01-15T19:02:34.151 INFO:tasks.workunit.client.0.smithi145.stdout: "disallowed_leaders: ": "", 2021-01-15T19:02:34.151 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": false, 2021-01-15T19:02:34.151 INFO:tasks.workunit.client.0.smithi145.stdout: "features": { 2021-01-15T19:02:34.151 INFO:tasks.workunit.client.0.smithi145.stdout: "persistent": [ 2021-01-15T19:02:34.152 INFO:tasks.workunit.client.0.smithi145.stdout: "kraken", 2021-01-15T19:02:34.152 INFO:tasks.workunit.client.0.smithi145.stdout: "luminous", 2021-01-15T19:02:34.152 INFO:tasks.workunit.client.0.smithi145.stdout: "mimic", 2021-01-15T19:02:34.152 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap-prune", 2021-01-15T19:02:34.153 INFO:tasks.workunit.client.0.smithi145.stdout: "nautilus", 2021-01-15T19:02:34.153 INFO:tasks.workunit.client.0.smithi145.stdout: "octopus", 2021-01-15T19:02:34.153 INFO:tasks.workunit.client.0.smithi145.stdout: "pacific", 2021-01-15T19:02:34.153 INFO:tasks.workunit.client.0.smithi145.stdout: "elector-pinging" 2021-01-15T19:02:34.153 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.154 INFO:tasks.workunit.client.0.smithi145.stdout: "optional": [] 2021-01-15T19:02:34.154 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.154 INFO:tasks.workunit.client.0.smithi145.stdout: "mons": [ 2021-01-15T19:02:34.154 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.154 INFO:tasks.workunit.client.0.smithi145.stdout: "rank": 0, 2021-01-15T19:02:34.155 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "a", 2021-01-15T19:02:34.155 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T19:02:34.155 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.155 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.155 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.156 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7140", 2021-01-15T19:02:34.156 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 0 2021-01-15T19:02:34.156 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.156 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.156 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.157 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:7140/0", 2021-01-15T19:02:34.157 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:7140/0", 2021-01-15T19:02:34.157 INFO:tasks.workunit.client.0.smithi145.stdout: "priority": 0, 2021-01-15T19:02:34.158 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 0, 2021-01-15T19:02:34.158 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_location": "{}" 2021-01-15T19:02:34.158 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.158 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.158 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.159 INFO:tasks.workunit.client.0.smithi145.stdout: "quorum": [ 2021-01-15T19:02:34.159 INFO:tasks.workunit.client.0.smithi145.stdout: 0 2021-01-15T19:02:34.159 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.159 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap": { 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 33, 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "fsid": "8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e", 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "created": "2021-01-15T18:59:12.992514+0000", 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T18:59:57.859526+0000", 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "last_up_change": "2021-01-15T18:59:48.400628+0000", 2021-01-15T19:02:34.160 INFO:tasks.workunit.client.0.smithi145.stdout: "last_in_change": "2021-01-15T18:59:48.400628+0000", 2021-01-15T19:02:34.161 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-01-15T19:02:34.161 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_num": 5799936, 2021-01-15T19:02:34.161 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_set": [ 2021-01-15T19:02:34.161 INFO:tasks.workunit.client.0.smithi145.stdout: "pglog_hardlimit", 2021-01-15T19:02:34.161 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snapdirs", 2021-01-15T19:02:34.162 INFO:tasks.workunit.client.0.smithi145.stdout: "recovery_deletes", 2021-01-15T19:02:34.162 INFO:tasks.workunit.client.0.smithi145.stdout: "sortbitwise" 2021-01-15T19:02:34.162 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.162 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_version": 9, 2021-01-15T19:02:34.162 INFO:tasks.workunit.client.0.smithi145.stdout: "full_ratio": 0.99000000953674316, 2021-01-15T19:02:34.163 INFO:tasks.workunit.client.0.smithi145.stdout: "backfillfull_ratio": 0.99000000953674316, 2021-01-15T19:02:34.163 INFO:tasks.workunit.client.0.smithi145.stdout: "nearfull_ratio": 0.85000002384185791, 2021-01-15T19:02:34.163 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_snapshot": "", 2021-01-15T19:02:34.163 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_max": 2, 2021-01-15T19:02:34.163 INFO:tasks.workunit.client.0.smithi145.stdout: "max_osd": 4, 2021-01-15T19:02:34.164 INFO:tasks.workunit.client.0.smithi145.stdout: "require_min_compat_client": "luminous", 2021-01-15T19:02:34.164 INFO:tasks.workunit.client.0.smithi145.stdout: "min_compat_client": "jewel", 2021-01-15T19:02:34.164 INFO:tasks.workunit.client.0.smithi145.stdout: "require_osd_release": "pacific", 2021-01-15T19:02:34.164 INFO:tasks.workunit.client.0.smithi145.stdout: "pools": [ 2021-01-15T19:02:34.164 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.165 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T19:02:34.165 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "foo", 2021-01-15T19:02:34.165 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T18:59:13.997234+0000", 2021-01-15T19:02:34.165 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T19:02:34.165 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T19:02:34.166 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T19:02:34.166 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T19:02:34.166 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T19:02:34.166 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T19:02:34.166 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T19:02:34.167 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T19:02:34.167 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T19:02:34.167 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T19:02:34.168 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T19:02:34.168 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T19:02:34.168 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 8, 2021-01-15T19:02:34.168 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 8, 2021-01-15T19:02:34.168 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 8, 2021-01-15T19:02:34.169 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 8, 2021-01-15T19:02:34.169 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 8, 2021-01-15T19:02:34.169 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T19:02:34.169 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T19:02:34.169 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T19:02:34.170 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T19:02:34.170 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T19:02:34.170 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T19:02:34.170 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T19:02:34.170 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.171 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "8", 2021-01-15T19:02:34.171 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T19:02:34.171 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T19:02:34.171 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T19:02:34.171 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T19:02:34.172 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T19:02:34.172 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T19:02:34.172 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T19:02:34.172 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T19:02:34.172 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T19:02:34.173 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T19:02:34.173 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T19:02:34.173 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T19:02:34.173 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T19:02:34.173 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T19:02:34.174 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T19:02:34.174 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T19:02:34.174 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T19:02:34.174 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T19:02:34.174 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T19:02:34.175 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T19:02:34.176 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T19:02:34.176 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.176 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T19:02:34.176 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T19:02:34.176 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T19:02:34.177 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T19:02:34.177 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T19:02:34.177 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T19:02:34.177 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T19:02:34.178 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T19:02:34.178 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T19:02:34.178 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T19:02:34.178 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T19:02:34.179 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T19:02:34.179 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T19:02:34.179 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.179 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.179 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 2, 2021-01-15T19:02:34.180 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_name": "pool-rep", 2021-01-15T19:02:34.180 INFO:tasks.workunit.client.0.smithi145.stdout: "create_time": "2021-01-15T18:59:55.795084+0000", 2021-01-15T19:02:34.180 INFO:tasks.workunit.client.0.smithi145.stdout: "flags": 1, 2021-01-15T19:02:34.180 INFO:tasks.workunit.client.0.smithi145.stdout: "flags_names": "hashpspool", 2021-01-15T19:02:34.180 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T19:02:34.181 INFO:tasks.workunit.client.0.smithi145.stdout: "size": 3, 2021-01-15T19:02:34.181 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 2, 2021-01-15T19:02:34.181 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_rule": 0, 2021-01-15T19:02:34.181 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_count": 0, 2021-01-15T19:02:34.182 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_target": 0, 2021-01-15T19:02:34.182 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_barrier": 0, 2021-01-15T19:02:34.182 INFO:tasks.workunit.client.0.smithi145.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-01-15T19:02:34.182 INFO:tasks.workunit.client.0.smithi145.stdout: "object_hash": 2, 2021-01-15T19:02:34.182 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_autoscale_mode": "off", 2021-01-15T19:02:34.183 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num": 1, 2021-01-15T19:02:34.183 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num": 1, 2021-01-15T19:02:34.183 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_placement_num_target": 1, 2021-01-15T19:02:34.183 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_target": 1, 2021-01-15T19:02:34.183 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_num_pending": 1, 2021-01-15T19:02:34.184 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pg_merge_meta": { 2021-01-15T19:02:34.184 INFO:tasks.workunit.client.0.smithi145.stdout: "source_pgid": "0.0", 2021-01-15T19:02:34.184 INFO:tasks.workunit.client.0.smithi145.stdout: "ready_epoch": 0, 2021-01-15T19:02:34.184 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_started": 0, 2021-01-15T19:02:34.184 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 0, 2021-01-15T19:02:34.185 INFO:tasks.workunit.client.0.smithi145.stdout: "source_version": "0'0", 2021-01-15T19:02:34.185 INFO:tasks.workunit.client.0.smithi145.stdout: "target_version": "0'0" 2021-01-15T19:02:34.185 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.185 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "33", 2021-01-15T19:02:34.185 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend": "0", 2021-01-15T19:02:34.186 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_prenautilus": "0", 2021-01-15T19:02:34.186 INFO:tasks.workunit.client.0.smithi145.stdout: "last_force_op_resend_preluminous": "0", 2021-01-15T19:02:34.186 INFO:tasks.workunit.client.0.smithi145.stdout: "auid": 0, 2021-01-15T19:02:34.186 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_mode": "selfmanaged", 2021-01-15T19:02:34.186 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_seq": 0, 2021-01-15T19:02:34.187 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_epoch": 0, 2021-01-15T19:02:34.187 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_snaps": [], 2021-01-15T19:02:34.187 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps": "[]", 2021-01-15T19:02:34.187 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_bytes": 0, 2021-01-15T19:02:34.187 INFO:tasks.workunit.client.0.smithi145.stdout: "quota_max_objects": 0, 2021-01-15T19:02:34.188 INFO:tasks.workunit.client.0.smithi145.stdout: "tiers": [], 2021-01-15T19:02:34.188 INFO:tasks.workunit.client.0.smithi145.stdout: "tier_of": -1, 2021-01-15T19:02:34.188 INFO:tasks.workunit.client.0.smithi145.stdout: "read_tier": -1, 2021-01-15T19:02:34.188 INFO:tasks.workunit.client.0.smithi145.stdout: "write_tier": -1, 2021-01-15T19:02:34.189 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_mode": "none", 2021-01-15T19:02:34.189 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_bytes": 0, 2021-01-15T19:02:34.189 INFO:tasks.workunit.client.0.smithi145.stdout: "target_max_objects": 0, 2021-01-15T19:02:34.189 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-01-15T19:02:34.189 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-01-15T19:02:34.190 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_target_full_ratio_micro": 800000, 2021-01-15T19:02:34.190 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_flush_age": 0, 2021-01-15T19:02:34.190 INFO:tasks.workunit.client.0.smithi145.stdout: "cache_min_evict_age": 0, 2021-01-15T19:02:34.190 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profile": "", 2021-01-15T19:02:34.190 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_params": { 2021-01-15T19:02:34.191 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "none" 2021-01-15T19:02:34.191 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.191 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_period": 0, 2021-01-15T19:02:34.191 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_count": 0, 2021-01-15T19:02:34.191 INFO:tasks.workunit.client.0.smithi145.stdout: "use_gmt_hitset": true, 2021-01-15T19:02:34.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_read_recency_for_promote": 0, 2021-01-15T19:02:34.192 INFO:tasks.workunit.client.0.smithi145.stdout: "min_write_recency_for_promote": 0, 2021-01-15T19:02:34.192 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_grade_decay_rate": 0, 2021-01-15T19:02:34.192 INFO:tasks.workunit.client.0.smithi145.stdout: "hit_set_search_last_n": 0, 2021-01-15T19:02:34.193 INFO:tasks.workunit.client.0.smithi145.stdout: "grade_table": [], 2021-01-15T19:02:34.193 INFO:tasks.workunit.client.0.smithi145.stdout: "stripe_width": 0, 2021-01-15T19:02:34.193 INFO:tasks.workunit.client.0.smithi145.stdout: "expected_num_objects": 0, 2021-01-15T19:02:34.193 INFO:tasks.workunit.client.0.smithi145.stdout: "fast_read": false, 2021-01-15T19:02:34.193 INFO:tasks.workunit.client.0.smithi145.stdout: "options": {}, 2021-01-15T19:02:34.194 INFO:tasks.workunit.client.0.smithi145.stdout: "application_metadata": {} 2021-01-15T19:02:34.194 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.194 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.194 INFO:tasks.workunit.client.0.smithi145.stdout: "osds": [ 2021-01-15T19:02:34.194 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.195 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T19:02:34.195 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "46cd3dd3-fc19-491b-98a7-b929296b20b3", 2021-01-15T19:02:34.195 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T19:02:34.195 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T19:02:34.195 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T19:02:34.196 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T19:02:34.196 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T19:02:34.196 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T19:02:34.196 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 6, 2021-01-15T19:02:34.196 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 27, 2021-01-15T19:02:34.197 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 0, 2021-01-15T19:02:34.197 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T19:02:34.197 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T19:02:34.197 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.197 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.198 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.198 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6802", 2021-01-15T19:02:34.198 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.198 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.198 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.199 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.199 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6803", 2021-01-15T19:02:34.199 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.199 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.199 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.200 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6804", 2021-01-15T19:02:34.201 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.201 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.201 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.201 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.201 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6805", 2021-01-15T19:02:34.202 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.202 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.202 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.202 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.202 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T19:02:34.203 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.203 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.203 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.203 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6808", 2021-01-15T19:02:34.203 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.204 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.204 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.204 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.204 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6809", 2021-01-15T19:02:34.204 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.205 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.205 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.205 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.205 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T19:02:34.205 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.206 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.206 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.206 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6806", 2021-01-15T19:02:34.206 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.206 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.207 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.207 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.207 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6807", 2021-01-15T19:02:34.207 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 938547 2021-01-15T19:02:34.207 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.208 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.208 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.208 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6803/938547", 2021-01-15T19:02:34.208 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6805/938547", 2021-01-15T19:02:34.208 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6809/938547", 2021-01-15T19:02:34.209 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6807/938547", 2021-01-15T19:02:34.209 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T19:02:34.209 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T19:02:34.209 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T19:02:34.209 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.210 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.210 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.210 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T19:02:34.210 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "87bd569d-ebec-49a1-8bcc-aafe44544944", 2021-01-15T19:02:34.210 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T19:02:34.211 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T19:02:34.211 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T19:02:34.211 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T19:02:34.211 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T19:02:34.211 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T19:02:34.212 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 12, 2021-01-15T19:02:34.212 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 27, 2021-01-15T19:02:34.212 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 0, 2021-01-15T19:02:34.212 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T19:02:34.212 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T19:02:34.213 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.213 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.213 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.213 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6810", 2021-01-15T19:02:34.213 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.214 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.214 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.214 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.214 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6811", 2021-01-15T19:02:34.214 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.215 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.215 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.215 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.215 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T19:02:34.215 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.216 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.216 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.216 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6812", 2021-01-15T19:02:34.216 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.216 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.217 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.217 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.217 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6813", 2021-01-15T19:02:34.217 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.217 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.218 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.218 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.218 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T19:02:34.218 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.218 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.219 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.219 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6816", 2021-01-15T19:02:34.219 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.219 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.219 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.220 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.220 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6817", 2021-01-15T19:02:34.220 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.220 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.220 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.221 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.221 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T19:02:34.221 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.221 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.221 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.222 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6814", 2021-01-15T19:02:34.222 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.222 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.222 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.222 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.223 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6815", 2021-01-15T19:02:34.223 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 939811 2021-01-15T19:02:34.223 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.223 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.223 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.224 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6811/939811", 2021-01-15T19:02:34.224 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6813/939811", 2021-01-15T19:02:34.224 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6817/939811", 2021-01-15T19:02:34.224 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6815/939811", 2021-01-15T19:02:34.224 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T19:02:34.225 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T19:02:34.225 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T19:02:34.225 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.225 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.225 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.226 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T19:02:34.226 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "d3180277-0ce9-44ef-a659-588b7fea3280", 2021-01-15T19:02:34.226 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T19:02:34.226 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T19:02:34.226 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T19:02:34.227 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T19:02:34.227 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T19:02:34.227 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T19:02:34.227 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 19, 2021-01-15T19:02:34.227 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 27, 2021-01-15T19:02:34.228 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 0, 2021-01-15T19:02:34.228 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T19:02:34.228 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T19:02:34.228 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.228 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.229 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.229 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6818", 2021-01-15T19:02:34.229 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.229 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.229 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.230 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.230 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6819", 2021-01-15T19:02:34.230 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.230 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.230 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.231 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.231 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T19:02:34.231 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.231 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.231 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.232 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6820", 2021-01-15T19:02:34.232 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.232 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.232 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.232 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.233 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6821", 2021-01-15T19:02:34.233 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.233 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.233 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.233 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.234 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T19:02:34.234 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.234 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.234 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.234 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6824", 2021-01-15T19:02:34.235 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.235 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.235 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.235 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.235 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6825", 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T19:02:34.236 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.237 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.237 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.237 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6822", 2021-01-15T19:02:34.237 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.237 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.238 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.238 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.238 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6823", 2021-01-15T19:02:34.238 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 941083 2021-01-15T19:02:34.238 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.239 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.239 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.239 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6819/941083", 2021-01-15T19:02:34.239 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6821/941083", 2021-01-15T19:02:34.239 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6825/941083", 2021-01-15T19:02:34.240 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6823/941083", 2021-01-15T19:02:34.240 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T19:02:34.240 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T19:02:34.240 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T19:02:34.240 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.241 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.241 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.241 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 3, 2021-01-15T19:02:34.241 INFO:tasks.workunit.client.0.smithi145.stdout: "uuid": "349bd717-875d-417c-9384-2e5329e121c2", 2021-01-15T19:02:34.241 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 1, 2021-01-15T19:02:34.242 INFO:tasks.workunit.client.0.smithi145.stdout: "in": 1, 2021-01-15T19:02:34.242 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 1, 2021-01-15T19:02:34.242 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_affinity": 1, 2021-01-15T19:02:34.242 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_begin": 0, 2021-01-15T19:02:34.242 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_end": 0, 2021-01-15T19:02:34.243 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 27, 2021-01-15T19:02:34.243 INFO:tasks.workunit.client.0.smithi145.stdout: "up_thru": 31, 2021-01-15T19:02:34.243 INFO:tasks.workunit.client.0.smithi145.stdout: "down_at": 0, 2021-01-15T19:02:34.243 INFO:tasks.workunit.client.0.smithi145.stdout: "lost_at": 0, 2021-01-15T19:02:34.243 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addrs": { 2021-01-15T19:02:34.244 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.244 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.244 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.244 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6826", 2021-01-15T19:02:34.244 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6827", 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.245 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.246 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.246 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.246 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addrs": { 2021-01-15T19:02:34.246 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.246 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.247 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.247 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6828", 2021-01-15T19:02:34.247 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.247 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.247 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.248 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.248 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6829", 2021-01-15T19:02:34.248 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.248 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.248 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.249 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.249 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addrs": { 2021-01-15T19:02:34.249 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.249 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.249 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.250 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6832", 2021-01-15T19:02:34.250 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.250 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.250 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.250 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.251 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6833", 2021-01-15T19:02:34.251 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.251 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.251 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.251 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.252 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addrs": { 2021-01-15T19:02:34.252 INFO:tasks.workunit.client.0.smithi145.stdout: "addrvec": [ 2021-01-15T19:02:34.252 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.252 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v2", 2021-01-15T19:02:34.252 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6830", 2021-01-15T19:02:34.253 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.253 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.253 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.253 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "v1", 2021-01-15T19:02:34.253 INFO:tasks.workunit.client.0.smithi145.stdout: "addr": "127.0.0.1:6831", 2021-01-15T19:02:34.254 INFO:tasks.workunit.client.0.smithi145.stdout: "nonce": 942373 2021-01-15T19:02:34.254 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.254 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.254 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.254 INFO:tasks.workunit.client.0.smithi145.stdout: "public_addr": "127.0.0.1:6827/942373", 2021-01-15T19:02:34.255 INFO:tasks.workunit.client.0.smithi145.stdout: "cluster_addr": "127.0.0.1:6829/942373", 2021-01-15T19:02:34.255 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_back_addr": "127.0.0.1:6833/942373", 2021-01-15T19:02:34.255 INFO:tasks.workunit.client.0.smithi145.stdout: "heartbeat_front_addr": "127.0.0.1:6831/942373", 2021-01-15T19:02:34.255 INFO:tasks.workunit.client.0.smithi145.stdout: "state": [ 2021-01-15T19:02:34.255 INFO:tasks.workunit.client.0.smithi145.stdout: "exists", 2021-01-15T19:02:34.256 INFO:tasks.workunit.client.0.smithi145.stdout: "up" 2021-01-15T19:02:34.256 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.256 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.256 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.256 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_xinfo": [ 2021-01-15T19:02:34.257 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.257 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T19:02:34.257 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "0.000000", 2021-01-15T19:02:34.257 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T19:02:34.257 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T19:02:34.258 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T19:02:34.258 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T19:02:34.258 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T18:59:22.007354+0000", 2021-01-15T19:02:34.258 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T19:02:34.258 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.259 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.259 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T19:02:34.259 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "0.000000", 2021-01-15T19:02:34.259 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T19:02:34.259 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T19:02:34.260 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T19:02:34.260 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T19:02:34.260 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T18:59:30.426829+0000", 2021-01-15T19:02:34.260 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T19:02:34.260 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.261 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.261 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T19:02:34.261 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "0.000000", 2021-01-15T19:02:34.261 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T19:02:34.261 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T19:02:34.262 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T19:02:34.262 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T19:02:34.262 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T18:59:38.912258+0000", 2021-01-15T19:02:34.262 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T19:02:34.262 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.263 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.263 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 3, 2021-01-15T19:02:34.263 INFO:tasks.workunit.client.0.smithi145.stdout: "down_stamp": "0.000000", 2021-01-15T19:02:34.263 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_probability": 0, 2021-01-15T19:02:34.263 INFO:tasks.workunit.client.0.smithi145.stdout: "laggy_interval": 0, 2021-01-15T19:02:34.264 INFO:tasks.workunit.client.0.smithi145.stdout: "features": 4540138297136906239, 2021-01-15T19:02:34.264 INFO:tasks.workunit.client.0.smithi145.stdout: "old_weight": 0, 2021-01-15T19:02:34.264 INFO:tasks.workunit.client.0.smithi145.stdout: "last_purged_snaps_scrub": "2021-01-15T18:59:48.528274+0000", 2021-01-15T19:02:34.264 INFO:tasks.workunit.client.0.smithi145.stdout: "dead_epoch": 0 2021-01-15T19:02:34.264 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap": [], 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_upmap_items": [], 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_temp": [], 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: "primary_temp": [], 2021-01-15T19:02:34.265 INFO:tasks.workunit.client.0.smithi145.stdout: "blocklist": {}, 2021-01-15T19:02:34.266 INFO:tasks.workunit.client.0.smithi145.stdout: "erasure_code_profiles": { 2021-01-15T19:02:34.266 INFO:tasks.workunit.client.0.smithi145.stdout: "default": { 2021-01-15T19:02:34.266 INFO:tasks.workunit.client.0.smithi145.stdout: "k": "2", 2021-01-15T19:02:34.266 INFO:tasks.workunit.client.0.smithi145.stdout: "m": "2", 2021-01-15T19:02:34.266 INFO:tasks.workunit.client.0.smithi145.stdout: "plugin": "jerasure", 2021-01-15T19:02:34.267 INFO:tasks.workunit.client.0.smithi145.stdout: "technique": "reed_sol_van" 2021-01-15T19:02:34.267 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.267 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.267 INFO:tasks.workunit.client.0.smithi145.stdout: "removed_snaps_queue": [], 2021-01-15T19:02:34.267 INFO:tasks.workunit.client.0.smithi145.stdout: "new_removed_snaps": [], 2021-01-15T19:02:34.268 INFO:tasks.workunit.client.0.smithi145.stdout: "new_purged_snaps": [], 2021-01-15T19:02:34.268 INFO:tasks.workunit.client.0.smithi145.stdout: "crush_node_flags": {}, 2021-01-15T19:02:34.268 INFO:tasks.workunit.client.0.smithi145.stdout: "device_class_flags": {}, 2021-01-15T19:02:34.268 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode": { 2021-01-15T19:02:34.268 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_enabled": false, 2021-01-15T19:02:34.269 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_bucket_count": 0, 2021-01-15T19:02:34.269 INFO:tasks.workunit.client.0.smithi145.stdout: "degraded_stretch_mode": 0, 2021-01-15T19:02:34.269 INFO:tasks.workunit.client.0.smithi145.stdout: "recovering_stretch_mode": 0, 2021-01-15T19:02:34.269 INFO:tasks.workunit.client.0.smithi145.stdout: "stretch_mode_bucket": 0 2021-01-15T19:02:34.269 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.270 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.270 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_metadata": [ 2021-01-15T19:02:34.270 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.270 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T19:02:34.270 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T19:02:34.271 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6804/938547,v1:127.0.0.1:6805/938547]", 2021-01-15T19:02:34.271 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T19:02:34.271 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T19:02:34.271 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T19:02:34.271 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T19:02:34.272 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T19:02:34.272 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T19:02:34.272 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T19:02:34.272 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T19:02:34.272 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T19:02:34.273 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2021-01-15T19:02:34.273 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T19:02:34.273 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T19:02:34.273 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T19:02:34.273 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T19:02:34.274 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T19:02:34.274 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T19:02:34.274 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T19:02:34.274 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T19:02:34.274 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T19:02:34.275 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T19:02:34.275 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T19:02:34.275 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T19:02:34.275 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T19:02:34.275 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T19:02:34.276 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T19:02:34.276 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6802/938547,v1:127.0.0.1:6803/938547]", 2021-01-15T19:02:34.276 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T19:02:34.276 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/938547,v1:127.0.0.1:6809/938547]", 2021-01-15T19:02:34.276 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/938547,v1:127.0.0.1:6807/938547]", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T19:02:34.277 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T19:02:34.278 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T19:02:34.278 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T19:02:34.278 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T19:02:34.278 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2021-01-15T19:02:34.278 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T19:02:34.279 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T19:02:34.279 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T19:02:34.279 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.279 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.279 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T19:02:34.280 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T19:02:34.280 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6812/939811,v1:127.0.0.1:6813/939811]", 2021-01-15T19:02:34.280 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T19:02:34.280 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T19:02:34.280 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T19:02:34.281 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T19:02:34.281 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T19:02:34.281 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T19:02:34.281 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T19:02:34.281 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T19:02:34.282 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T19:02:34.282 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2021-01-15T19:02:34.282 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T19:02:34.282 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T19:02:34.282 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T19:02:34.283 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T19:02:34.283 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T19:02:34.283 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T19:02:34.283 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T19:02:34.283 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T19:02:34.284 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T19:02:34.284 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T19:02:34.284 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T19:02:34.284 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T19:02:34.284 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T19:02:34.285 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T19:02:34.285 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T19:02:34.285 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6810/939811,v1:127.0.0.1:6811/939811]", 2021-01-15T19:02:34.285 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T19:02:34.285 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/939811,v1:127.0.0.1:6817/939811]", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/939811,v1:127.0.0.1:6815/939811]", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T19:02:34.286 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T19:02:34.287 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T19:02:34.287 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T19:02:34.287 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T19:02:34.287 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T19:02:34.287 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2021-01-15T19:02:34.288 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T19:02:34.288 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T19:02:34.288 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T19:02:34.288 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.288 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.289 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T19:02:34.289 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T19:02:34.289 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6820/941083,v1:127.0.0.1:6821/941083]", 2021-01-15T19:02:34.289 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T19:02:34.289 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T19:02:34.290 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T19:02:34.290 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T19:02:34.290 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T19:02:34.290 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T19:02:34.290 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T19:02:34.291 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T19:02:34.291 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T19:02:34.291 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2021-01-15T19:02:34.291 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T19:02:34.291 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T19:02:34.292 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T19:02:34.292 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T19:02:34.292 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T19:02:34.292 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T19:02:34.292 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T19:02:34.293 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T19:02:34.293 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T19:02:34.293 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T19:02:34.293 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T19:02:34.293 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T19:02:34.294 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T19:02:34.294 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T19:02:34.294 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T19:02:34.294 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6818/941083,v1:127.0.0.1:6819/941083]", 2021-01-15T19:02:34.294 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T19:02:34.295 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/941083,v1:127.0.0.1:6825/941083]", 2021-01-15T19:02:34.295 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/941083,v1:127.0.0.1:6823/941083]", 2021-01-15T19:02:34.295 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T19:02:34.295 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T19:02:34.295 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T19:02:34.296 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T19:02:34.297 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2021-01-15T19:02:34.297 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T19:02:34.297 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T19:02:34.297 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T19:02:34.297 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.298 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.298 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 3, 2021-01-15T19:02:34.298 INFO:tasks.workunit.client.0.smithi145.stdout: "arch": "x86_64", 2021-01-15T19:02:34.298 INFO:tasks.workunit.client.0.smithi145.stdout: "back_addr": "[v2:127.0.0.1:6828/942373,v1:127.0.0.1:6829/942373]", 2021-01-15T19:02:34.298 INFO:tasks.workunit.client.0.smithi145.stdout: "back_iface": "", 2021-01-15T19:02:34.299 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs": "1", 2021-01-15T19:02:34.299 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_db": "0", 2021-01-15T19:02:34.299 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_dedicated_wal": "0", 2021-01-15T19:02:34.299 INFO:tasks.workunit.client.0.smithi145.stdout: "bluefs_single_shared_device": "1", 2021-01-15T19:02:34.299 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_access_mode": "file", 2021-01-15T19:02:34.300 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_block_size": "4096", 2021-01-15T19:02:34.300 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_devices": "sda", 2021-01-15T19:02:34.300 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_driver": "KernelDevice", 2021-01-15T19:02:34.300 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2021-01-15T19:02:34.300 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_rotational": "1", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_size": "107374182400", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_support_discard": "0", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "bluestore_bdev_type": "hdd", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_release": "pacific", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version": "ceph version 16.0.0-9031-g892c51dd (892c51dd3c5f7108e766bea30cd5e0d801b0abd3) pacific (dev)", 2021-01-15T19:02:34.301 INFO:tasks.workunit.client.0.smithi145.stdout: "ceph_version_short": "16.0.0-9031-g892c51dd", 2021-01-15T19:02:34.302 INFO:tasks.workunit.client.0.smithi145.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2021-01-15T19:02:34.302 INFO:tasks.workunit.client.0.smithi145.stdout: "default_device_class": "hdd", 2021-01-15T19:02:34.302 INFO:tasks.workunit.client.0.smithi145.stdout: "device_ids": "sda=ST1000NM0033-9ZM_Z1W5XLDP", 2021-01-15T19:02:34.302 INFO:tasks.workunit.client.0.smithi145.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2021-01-15T19:02:34.303 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": "sda", 2021-01-15T19:02:34.303 INFO:tasks.workunit.client.0.smithi145.stdout: "distro": "centos", 2021-01-15T19:02:34.303 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_description": "CentOS Linux 8 (Core)", 2021-01-15T19:02:34.303 INFO:tasks.workunit.client.0.smithi145.stdout: "distro_version": "8", 2021-01-15T19:02:34.303 INFO:tasks.workunit.client.0.smithi145.stdout: "front_addr": "[v2:127.0.0.1:6826/942373,v1:127.0.0.1:6827/942373]", 2021-01-15T19:02:34.304 INFO:tasks.workunit.client.0.smithi145.stdout: "front_iface": "", 2021-01-15T19:02:34.304 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/942373,v1:127.0.0.1:6833/942373]", 2021-01-15T19:02:34.304 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/942373,v1:127.0.0.1:6831/942373]", 2021-01-15T19:02:34.304 INFO:tasks.workunit.client.0.smithi145.stdout: "hostname": "smithi145", 2021-01-15T19:02:34.304 INFO:tasks.workunit.client.0.smithi145.stdout: "journal_rotational": "1", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_description": "#1 SMP Thu Nov 19 17:20:08 UTC 2020", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "kernel_version": "4.18.0-240.1.1.el8_3.x86_64", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_swap_kb": "0", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "mem_total_kb": "32609904", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2021-01-15T19:02:34.305 INFO:tasks.workunit.client.0.smithi145.stdout: "objectstore_numa_unknown_devices": "sda", 2021-01-15T19:02:34.306 INFO:tasks.workunit.client.0.smithi145.stdout: "os": "Linux", 2021-01-15T19:02:34.306 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2021-01-15T19:02:34.306 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_objectstore": "bluestore", 2021-01-15T19:02:34.306 INFO:tasks.workunit.client.0.smithi145.stdout: "osdspec_affinity": "", 2021-01-15T19:02:34.306 INFO:tasks.workunit.client.0.smithi145.stdout: "rotational": "1" 2021-01-15T19:02:34.307 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.307 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.307 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_clean_epochs": { 2021-01-15T19:02:34.307 INFO:tasks.workunit.client.0.smithi145.stdout: "min_last_epoch_clean": 0, 2021-01-15T19:02:34.307 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": { 2021-01-15T19:02:34.308 INFO:tasks.workunit.client.0.smithi145.stdout: "per_pool": [ 2021-01-15T19:02:34.308 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.308 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T19:02:34.308 INFO:tasks.workunit.client.0.smithi145.stdout: "floor": 4294967295 2021-01-15T19:02:34.308 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.309 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.309 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.309 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_epochs": [ 2021-01-15T19:02:34.309 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.309 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T19:02:34.310 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 7 2021-01-15T19:02:34.310 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.310 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.310 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T19:02:34.310 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 12 2021-01-15T19:02:34.311 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.311 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.311 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T19:02:34.311 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 21 2021-01-15T19:02:34.311 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.312 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.312 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 3, 2021-01-15T19:02:34.312 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 28 2021-01-15T19:02:34.312 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.312 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_first_committed": 1, 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: "osdmap_last_committed": 33, 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: "crushmap": { 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: "devices": [ 2021-01-15T19:02:34.313 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.314 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T19:02:34.314 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.0", 2021-01-15T19:02:34.314 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T19:02:34.314 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.315 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.315 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T19:02:34.315 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.1", 2021-01-15T19:02:34.315 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T19:02:34.315 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.316 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.316 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T19:02:34.316 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.2", 2021-01-15T19:02:34.316 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T19:02:34.316 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.317 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.317 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 3, 2021-01-15T19:02:34.317 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd.3", 2021-01-15T19:02:34.317 INFO:tasks.workunit.client.0.smithi145.stdout: "class": "hdd" 2021-01-15T19:02:34.317 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: "types": [ 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 0, 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "osd" 2021-01-15T19:02:34.318 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.319 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.319 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T19:02:34.319 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "host" 2021-01-15T19:02:34.319 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.319 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.320 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 2, 2021-01-15T19:02:34.320 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "chassis" 2021-01-15T19:02:34.320 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.320 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.320 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 3, 2021-01-15T19:02:34.321 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "rack" 2021-01-15T19:02:34.321 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.321 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.321 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 4, 2021-01-15T19:02:34.321 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "row" 2021-01-15T19:02:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 5, 2021-01-15T19:02:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pdu" 2021-01-15T19:02:34.322 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 6, 2021-01-15T19:02:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "pod" 2021-01-15T19:02:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.323 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 7, 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "room" 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 8, 2021-01-15T19:02:34.324 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "datacenter" 2021-01-15T19:02:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 9, 2021-01-15T19:02:34.325 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "zone" 2021-01-15T19:02:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 10, 2021-01-15T19:02:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "region" 2021-01-15T19:02:34.326 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.327 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.327 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T19:02:34.327 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "root" 2021-01-15T19:02:34.327 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.327 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.328 INFO:tasks.workunit.client.0.smithi145.stdout: "buckets": [ 2021-01-15T19:02:34.328 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.328 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -1, 2021-01-15T19:02:34.328 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default", 2021-01-15T19:02:34.328 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T19:02:34.329 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T19:02:34.329 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.329 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T19:02:34.329 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T19:02:34.329 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T19:02:34.330 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.330 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T19:02:34.330 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.330 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T19:02:34.330 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.331 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.331 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.331 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.331 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -2, 2021-01-15T19:02:34.331 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "default~hdd", 2021-01-15T19:02:34.332 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 11, 2021-01-15T19:02:34.332 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "root", 2021-01-15T19:02:34.332 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.332 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T19:02:34.332 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T19:02:34.333 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T19:02:34.333 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.333 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T19:02:34.333 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.333 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T19:02:34.334 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.334 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.334 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.334 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.334 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -3, 2021-01-15T19:02:34.335 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145", 2021-01-15T19:02:34.335 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T19:02:34.335 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T19:02:34.335 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.335 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T19:02:34.336 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T19:02:34.336 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T19:02:34.336 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.336 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T19:02:34.337 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.337 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T19:02:34.337 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.337 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.338 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T19:02:34.338 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.338 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T19:02:34.338 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.338 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.339 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T19:02:34.339 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.339 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T19:02:34.339 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.340 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.340 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 3, 2021-01-15T19:02:34.340 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.340 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 3 2021-01-15T19:02:34.341 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.341 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.341 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.341 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.341 INFO:tasks.workunit.client.0.smithi145.stdout: "id": -4, 2021-01-15T19:02:34.342 INFO:tasks.workunit.client.0.smithi145.stdout: "name": "smithi145~hdd", 2021-01-15T19:02:34.342 INFO:tasks.workunit.client.0.smithi145.stdout: "type_id": 1, 2021-01-15T19:02:34.342 INFO:tasks.workunit.client.0.smithi145.stdout: "type_name": "host", 2021-01-15T19:02:34.342 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 25608, 2021-01-15T19:02:34.342 INFO:tasks.workunit.client.0.smithi145.stdout: "alg": "straw2", 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: "hash": "rjenkins1", 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: "items": [ 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 0, 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.343 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 0 2021-01-15T19:02:34.344 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.344 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.344 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 1, 2021-01-15T19:02:34.344 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.344 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 1 2021-01-15T19:02:34.345 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.345 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.345 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 2, 2021-01-15T19:02:34.345 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.345 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 2 2021-01-15T19:02:34.346 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.346 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.346 INFO:tasks.workunit.client.0.smithi145.stdout: "id": 3, 2021-01-15T19:02:34.346 INFO:tasks.workunit.client.0.smithi145.stdout: "weight": 6402, 2021-01-15T19:02:34.346 INFO:tasks.workunit.client.0.smithi145.stdout: "pos": 3 2021-01-15T19:02:34.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.347 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.347 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.347 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.347 INFO:tasks.workunit.client.0.smithi145.stdout: "rules": [ 2021-01-15T19:02:34.348 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.348 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_id": 0, 2021-01-15T19:02:34.348 INFO:tasks.workunit.client.0.smithi145.stdout: "rule_name": "replicated_rule", 2021-01-15T19:02:34.348 INFO:tasks.workunit.client.0.smithi145.stdout: "ruleset": 0, 2021-01-15T19:02:34.348 INFO:tasks.workunit.client.0.smithi145.stdout: "type": 1, 2021-01-15T19:02:34.349 INFO:tasks.workunit.client.0.smithi145.stdout: "min_size": 1, 2021-01-15T19:02:34.349 INFO:tasks.workunit.client.0.smithi145.stdout: "max_size": 10, 2021-01-15T19:02:34.349 INFO:tasks.workunit.client.0.smithi145.stdout: "steps": [ 2021-01-15T19:02:34.349 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.349 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "take", 2021-01-15T19:02:34.350 INFO:tasks.workunit.client.0.smithi145.stdout: "item": -1, 2021-01-15T19:02:34.350 INFO:tasks.workunit.client.0.smithi145.stdout: "item_name": "default" 2021-01-15T19:02:34.350 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.350 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.350 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "choose_firstn", 2021-01-15T19:02:34.351 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 0, 2021-01-15T19:02:34.351 INFO:tasks.workunit.client.0.smithi145.stdout: "type": "osd" 2021-01-15T19:02:34.351 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.351 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.351 INFO:tasks.workunit.client.0.smithi145.stdout: "op": "emit" 2021-01-15T19:02:34.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.352 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:02:34.352 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.352 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.352 INFO:tasks.workunit.client.0.smithi145.stdout: "tunables": { 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_tries": 0, 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_local_fallback_tries": 0, 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_total_tries": 50, 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_descend_once": 1, 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_vary_r": 1, 2021-01-15T19:02:34.353 INFO:tasks.workunit.client.0.smithi145.stdout: "chooseleaf_stable": 1, 2021-01-15T19:02:34.354 INFO:tasks.workunit.client.0.smithi145.stdout: "straw_calc_version": 1, 2021-01-15T19:02:34.354 INFO:tasks.workunit.client.0.smithi145.stdout: "allowed_bucket_algs": 54, 2021-01-15T19:02:34.354 INFO:tasks.workunit.client.0.smithi145.stdout: "profile": "jewel", 2021-01-15T19:02:34.354 INFO:tasks.workunit.client.0.smithi145.stdout: "optimal_tunables": 1, 2021-01-15T19:02:34.354 INFO:tasks.workunit.client.0.smithi145.stdout: "legacy_tunables": 0, 2021-01-15T19:02:34.355 INFO:tasks.workunit.client.0.smithi145.stdout: "minimum_required_version": "jewel", 2021-01-15T19:02:34.355 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables": 1, 2021-01-15T19:02:34.355 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables2": 1, 2021-01-15T19:02:34.355 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v2_rules": 0, 2021-01-15T19:02:34.355 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables3": 1, 2021-01-15T19:02:34.356 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v3_rules": 0, 2021-01-15T19:02:34.356 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v4_buckets": 1, 2021-01-15T19:02:34.356 INFO:tasks.workunit.client.0.smithi145.stdout: "require_feature_tunables5": 1, 2021-01-15T19:02:34.356 INFO:tasks.workunit.client.0.smithi145.stdout: "has_v5_rules": 0 2021-01-15T19:02:34.356 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.357 INFO:tasks.workunit.client.0.smithi145.stdout: "choose_args": {} 2021-01-15T19:02:34.357 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.357 INFO:tasks.workunit.client.0.smithi145.stdout: "fsmap": { 2021-01-15T19:02:34.357 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 1, 2021-01-15T19:02:34.357 INFO:tasks.workunit.client.0.smithi145.stdout: "default_fscid": -1, 2021-01-15T19:02:34.358 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": { 2021-01-15T19:02:34.358 INFO:tasks.workunit.client.0.smithi145.stdout: "compat": {}, 2021-01-15T19:02:34.358 INFO:tasks.workunit.client.0.smithi145.stdout: "ro_compat": {}, 2021-01-15T19:02:34.358 INFO:tasks.workunit.client.0.smithi145.stdout: "incompat": { 2021-01-15T19:02:34.358 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_1": "base v0.20", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_2": "client writeable ranges", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_3": "default file layouts on dirs", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_4": "dir inode in separate object", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_5": "mds uses versioned encoding", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_6": "dirfrag is stored in omap", 2021-01-15T19:02:34.359 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_8": "no anchor table", 2021-01-15T19:02:34.360 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_9": "file layout v2", 2021-01-15T19:02:34.360 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_10": "snaprealm v2" 2021-01-15T19:02:34.360 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.360 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.361 INFO:tasks.workunit.client.0.smithi145.stdout: "feature_flags": { 2021-01-15T19:02:34.361 INFO:tasks.workunit.client.0.smithi145.stdout: "enable_multiple": true, 2021-01-15T19:02:34.361 INFO:tasks.workunit.client.0.smithi145.stdout: "ever_enabled_multiple": true 2021-01-15T19:02:34.361 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.361 INFO:tasks.workunit.client.0.smithi145.stdout: "standbys": [], 2021-01-15T19:02:34.362 INFO:tasks.workunit.client.0.smithi145.stdout: "filesystems": [] 2021-01-15T19:02:34.362 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.362 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_first_committed": 1, 2021-01-15T19:02:34.362 INFO:tasks.workunit.client.0.smithi145.stdout: "mdsmap_last_committed": 1, 2021-01-15T19:02:34.362 INFO:tasks.workunit.client.0.smithi145.stdout: "auth": { 2021-01-15T19:02:34.363 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T19:02:34.363 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 6, 2021-01-15T19:02:34.363 INFO:tasks.workunit.client.0.smithi145.stdout: "num_secrets": 4 2021-01-15T19:02:34.363 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.363 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 9, 2021-01-15T19:02:34.364 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_active": 9, 2021-01-15T19:02:34.364 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_unknown": 0, 2021-01-15T19:02:34.364 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osd": 4, 2021-01-15T19:02:34.364 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_sum": { 2021-01-15T19:02:34.365 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T19:02:34.365 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 4096, 2021-01-15T19:02:34.365 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 1, 2021-01-15T19:02:34.365 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T19:02:34.365 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 3, 2021-01-15T19:02:34.366 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 1, 2021-01-15T19:02:34.366 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T19:02:34.366 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 3, 2021-01-15T19:02:34.366 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T19:02:34.366 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 1, 2021-01-15T19:02:34.367 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 1, 2021-01-15T19:02:34.367 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T19:02:34.367 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 3, 2021-01-15T19:02:34.367 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 12, 2021-01-15T19:02:34.367 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 1, 2021-01-15T19:02:34.368 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 4, 2021-01-15T19:02:34.368 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T19:02:34.368 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T19:02:34.368 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T19:02:34.368 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 3, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 12288, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T19:02:34.369 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T19:02:34.370 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T19:02:34.370 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T19:02:34.370 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T19:02:34.370 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T19:02:34.371 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T19:02:34.372 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T19:02:34.372 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T19:02:34.372 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T19:02:34.372 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T19:02:34.372 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 3 2021-01-15T19:02:34.373 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.373 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T19:02:34.373 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T19:02:34.373 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T19:02:34.373 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T19:02:34.374 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T19:02:34.374 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T19:02:34.374 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T19:02:34.374 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T19:02:34.374 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T19:02:34.375 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T19:02:34.375 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T19:02:34.375 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.375 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 1, 2021-01-15T19:02:34.375 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 1, 2021-01-15T19:02:34.376 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 27, 2021-01-15T19:02:34.376 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 27, 2021-01-15T19:02:34.376 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 0 2021-01-15T19:02:34.376 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.376 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum": { 2021-01-15T19:02:34.377 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T19:02:34.377 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T19:02:34.377 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 27, 2021-01-15T19:02:34.377 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 4, 2021-01-15T19:02:34.377 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 4, 2021-01-15T19:02:34.378 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 4, 2021-01-15T19:02:34.378 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 419430400, 2021-01-15T19:02:34.378 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 22064, 2021-01-15T19:02:34.378 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 1100, 2021-01-15T19:02:34.378 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 0, 2021-01-15T19:02:34.379 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 20928, 2021-01-15T19:02:34.379 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 419408336, 2021-01-15T19:02:34.379 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T19:02:34.379 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 429496729600, 2021-01-15T19:02:34.379 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 429474136064, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1126400, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 369004, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T19:02:34.380 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T19:02:34.381 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T19:02:34.381 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 21430272 2021-01-15T19:02:34.381 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.381 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T19:02:34.382 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T19:02:34.382 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T19:02:34.382 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 4, 2021-01-15T19:02:34.382 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T19:02:34.382 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [ 2021-01-15T19:02:34.383 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.383 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.383 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.383 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.383 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.384 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.384 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.384 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.384 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.384 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.385 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.385 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.385 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.385 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.385 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: 1 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 262144 2021-01-15T19:02:34.386 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.387 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T19:02:34.387 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T19:02:34.387 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T19:02:34.387 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T19:02:34.387 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T19:02:34.388 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.388 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T19:02:34.388 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T19:02:34.388 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.388 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_sum_by_class": { 2021-01-15T19:02:34.389 INFO:tasks.workunit.client.0.smithi145.stdout: "hdd": { 2021-01-15T19:02:34.389 INFO:tasks.workunit.client.0.smithi145.stdout: "up_from": 0, 2021-01-15T19:02:34.389 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 0, 2021-01-15T19:02:34.389 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pgs": 27, 2021-01-15T19:02:34.389 INFO:tasks.workunit.client.0.smithi145.stdout: "num_osds": 4, 2021-01-15T19:02:34.390 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_osds": 4, 2021-01-15T19:02:34.390 INFO:tasks.workunit.client.0.smithi145.stdout: "num_per_pool_omap_osds": 4, 2021-01-15T19:02:34.390 INFO:tasks.workunit.client.0.smithi145.stdout: "kb": 419430400, 2021-01-15T19:02:34.390 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used": 22064, 2021-01-15T19:02:34.390 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_data": 1100, 2021-01-15T19:02:34.391 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_omap": 0, 2021-01-15T19:02:34.391 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_used_meta": 20928, 2021-01-15T19:02:34.391 INFO:tasks.workunit.client.0.smithi145.stdout: "kb_avail": 419408336, 2021-01-15T19:02:34.391 INFO:tasks.workunit.client.0.smithi145.stdout: "statfs": { 2021-01-15T19:02:34.391 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 429496729600, 2021-01-15T19:02:34.392 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 429474136064, 2021-01-15T19:02:34.392 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T19:02:34.392 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 1126400, 2021-01-15T19:02:34.392 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 369004, 2021-01-15T19:02:34.392 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T19:02:34.393 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T19:02:34.393 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T19:02:34.393 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T19:02:34.393 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 21430272 2021-01-15T19:02:34.393 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.394 INFO:tasks.workunit.client.0.smithi145.stdout: "hb_peers": [], 2021-01-15T19:02:34.394 INFO:tasks.workunit.client.0.smithi145.stdout: "snap_trim_queue_len": 0, 2021-01-15T19:02:34.394 INFO:tasks.workunit.client.0.smithi145.stdout: "num_snap_trimming": 0, 2021-01-15T19:02:34.394 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shards_repaired": 4, 2021-01-15T19:02:34.394 INFO:tasks.workunit.client.0.smithi145.stdout: "op_queue_age_hist": { 2021-01-15T19:02:34.395 INFO:tasks.workunit.client.0.smithi145.stdout: "histogram": [ 2021-01-15T19:02:34.395 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.395 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.395 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.395 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.396 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.397 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.397 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.397 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.397 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.397 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.398 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.398 INFO:tasks.workunit.client.0.smithi145.stdout: 0, 2021-01-15T19:02:34.398 INFO:tasks.workunit.client.0.smithi145.stdout: 1 2021-01-15T19:02:34.398 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.398 INFO:tasks.workunit.client.0.smithi145.stdout: "upper_bound": 262144 2021-01-15T19:02:34.399 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.399 INFO:tasks.workunit.client.0.smithi145.stdout: "perf_stat": { 2021-01-15T19:02:34.399 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ms": 0, 2021-01-15T19:02:34.399 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ms": 0, 2021-01-15T19:02:34.399 INFO:tasks.workunit.client.0.smithi145.stdout: "commit_latency_ns": 0, 2021-01-15T19:02:34.400 INFO:tasks.workunit.client.0.smithi145.stdout: "apply_latency_ns": 0 2021-01-15T19:02:34.400 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.400 INFO:tasks.workunit.client.0.smithi145.stdout: "alerts": [], 2021-01-15T19:02:34.400 INFO:tasks.workunit.client.0.smithi145.stdout: "network_ping_times": [] 2021-01-15T19:02:34.400 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.401 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.401 INFO:tasks.workunit.client.0.smithi145.stdout: "pool_stats": [ 2021-01-15T19:02:34.401 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.401 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 2, 2021-01-15T19:02:34.401 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 1, 2021-01-15T19:02:34.402 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T19:02:34.402 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 4096, 2021-01-15T19:02:34.402 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 1, 2021-01-15T19:02:34.402 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T19:02:34.402 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 3, 2021-01-15T19:02:34.403 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 1, 2021-01-15T19:02:34.403 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T19:02:34.403 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 3, 2021-01-15T19:02:34.403 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T19:02:34.403 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 1, 2021-01-15T19:02:34.404 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 1, 2021-01-15T19:02:34.404 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T19:02:34.404 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 3, 2021-01-15T19:02:34.404 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 12, 2021-01-15T19:02:34.404 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 1, 2021-01-15T19:02:34.405 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 4, 2021-01-15T19:02:34.405 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T19:02:34.405 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T19:02:34.405 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T19:02:34.405 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 3, 2021-01-15T19:02:34.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 12288, 2021-01-15T19:02:34.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T19:02:34.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T19:02:34.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T19:02:34.406 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T19:02:34.407 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T19:02:34.408 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T19:02:34.408 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T19:02:34.408 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T19:02:34.408 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T19:02:34.408 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T19:02:34.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T19:02:34.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T19:02:34.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T19:02:34.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T19:02:34.409 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 3 2021-01-15T19:02:34.410 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.410 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T19:02:34.410 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T19:02:34.410 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T19:02:34.410 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T19:02:34.411 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 12288, 2021-01-15T19:02:34.411 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 12288, 2021-01-15T19:02:34.411 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T19:02:34.411 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T19:02:34.411 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T19:02:34.412 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T19:02:34.412 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T19:02:34.412 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.412 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 1, 2021-01-15T19:02:34.412 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 1, 2021-01-15T19:02:34.413 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 3, 2021-01-15T19:02:34.413 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 3, 2021-01-15T19:02:34.413 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 3 2021-01-15T19:02:34.413 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.413 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.414 INFO:tasks.workunit.client.0.smithi145.stdout: "poolid": 1, 2021-01-15T19:02:34.414 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg": 8, 2021-01-15T19:02:34.414 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T19:02:34.414 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 0, 2021-01-15T19:02:34.414 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 0, 2021-01-15T19:02:34.415 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T19:02:34.415 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 0, 2021-01-15T19:02:34.415 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 0, 2021-01-15T19:02:34.415 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T19:02:34.415 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 0, 2021-01-15T19:02:34.416 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T19:02:34.416 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 0, 2021-01-15T19:02:34.416 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 0, 2021-01-15T19:02:34.416 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T19:02:34.416 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T19:02:34.417 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T19:02:34.417 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 0, 2021-01-15T19:02:34.417 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 0, 2021-01-15T19:02:34.417 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T19:02:34.417 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T19:02:34.418 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T19:02:34.418 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T19:02:34.418 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T19:02:34.418 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T19:02:34.418 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T19:02:34.419 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T19:02:34.419 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T19:02:34.419 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T19:02:34.419 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T19:02:34.419 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T19:02:34.420 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T19:02:34.420 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T19:02:34.420 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T19:02:34.420 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T19:02:34.420 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T19:02:34.421 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T19:02:34.421 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T19:02:34.421 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T19:02:34.421 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T19:02:34.421 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T19:02:34.422 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T19:02:34.422 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T19:02:34.422 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T19:02:34.422 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.422 INFO:tasks.workunit.client.0.smithi145.stdout: "store_stats": { 2021-01-15T19:02:34.423 INFO:tasks.workunit.client.0.smithi145.stdout: "total": 0, 2021-01-15T19:02:34.423 INFO:tasks.workunit.client.0.smithi145.stdout: "available": 0, 2021-01-15T19:02:34.423 INFO:tasks.workunit.client.0.smithi145.stdout: "internally_reserved": 0, 2021-01-15T19:02:34.423 INFO:tasks.workunit.client.0.smithi145.stdout: "allocated": 0, 2021-01-15T19:02:34.423 INFO:tasks.workunit.client.0.smithi145.stdout: "data_stored": 0, 2021-01-15T19:02:34.424 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed": 0, 2021-01-15T19:02:34.424 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_allocated": 0, 2021-01-15T19:02:34.424 INFO:tasks.workunit.client.0.smithi145.stdout: "data_compressed_original": 0, 2021-01-15T19:02:34.424 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_allocated": 0, 2021-01-15T19:02:34.424 INFO:tasks.workunit.client.0.smithi145.stdout: "internal_metadata": 0 2021-01-15T19:02:34.425 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.425 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 0, 2021-01-15T19:02:34.425 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 0, 2021-01-15T19:02:34.425 INFO:tasks.workunit.client.0.smithi145.stdout: "up": 24, 2021-01-15T19:02:34.425 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": 24, 2021-01-15T19:02:34.426 INFO:tasks.workunit.client.0.smithi145.stdout: "num_store_stats": 4 2021-01-15T19:02:34.426 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.426 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.426 INFO:tasks.workunit.client.0.smithi145.stdout: "osd_stats": [ 2021-01-15T19:02:34.426 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.427 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T19:02:34.427 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 25769803831 2021-01-15T19:02:34.427 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.427 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.427 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T19:02:34.428 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 51539607606 2021-01-15T19:02:34.428 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.428 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.428 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T19:02:34.428 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 81604378675 2021-01-15T19:02:34.429 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.429 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.429 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 3, 2021-01-15T19:02:34.429 INFO:tasks.workunit.client.0.smithi145.stdout: "seq": 115964117041 2021-01-15T19:02:34.429 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.430 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_state": [ 2021-01-15T19:02:34.430 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.430 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+recovery_unfound+degraded", 2021-01-15T19:02:34.430 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 1 2021-01-15T19:02:34.431 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.431 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.431 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+clean", 2021-01-15T19:02:34.431 INFO:tasks.workunit.client.0.smithi145.stdout: "num": 8 2021-01-15T19:02:34.431 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_pg_by_osd": [ 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 3, 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 4, 2021-01-15T19:02:34.432 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 7, 2021-01-15T19:02:34.433 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T19:02:34.433 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.433 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.433 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 2, 2021-01-15T19:02:34.434 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T19:02:34.434 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 5, 2021-01-15T19:02:34.434 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T19:02:34.434 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.434 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.435 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 0, 2021-01-15T19:02:34.435 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 1, 2021-01-15T19:02:34.435 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 9, 2021-01-15T19:02:34.435 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T19:02:34.435 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.436 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.436 INFO:tasks.workunit.client.0.smithi145.stdout: "osd": 1, 2021-01-15T19:02:34.436 INFO:tasks.workunit.client.0.smithi145.stdout: "num_primary_pg": 3, 2021-01-15T19:02:34.436 INFO:tasks.workunit.client.0.smithi145.stdout: "num_acting_pg": 6, 2021-01-15T19:02:34.436 INFO:tasks.workunit.client.0.smithi145.stdout: "num_up_not_acting_pg": 0 2021-01-15T19:02:34.437 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.437 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.437 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [ 2021-01-15T19:02:34.437 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.437 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 1, 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: "pool": 2, 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": {} 2021-01-15T19:02:34.438 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.439 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:02:34.439 INFO:tasks.workunit.client.0.smithi145.stdout: "servicemap": { 2021-01-15T19:02:34.439 INFO:tasks.workunit.client.0.smithi145.stdout: "epoch": 6, 2021-01-15T19:02:34.439 INFO:tasks.workunit.client.0.smithi145.stdout: "modified": "2021-01-15T19:00:49.598636+0000", 2021-01-15T19:02:34.439 INFO:tasks.workunit.client.0.smithi145.stdout: "services": {} 2021-01-15T19:02:34.440 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:02:34.440 INFO:tasks.workunit.client.0.smithi145.stdout: "paxos": { 2021-01-15T19:02:34.440 INFO:tasks.workunit.client.0.smithi145.stdout: "first_committed": 1, 2021-01-15T19:02:34.440 INFO:tasks.workunit.client.0.smithi145.stdout: "last_committed": 636, 2021-01-15T19:02:34.440 INFO:tasks.workunit.client.0.smithi145.stdout: "last_pn": 0, 2021-01-15T19:02:34.441 INFO:tasks.workunit.client.0.smithi145.stdout: "accepted_pn": 0 2021-01-15T19:02:34.441 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:02:34.441 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T19:02:34.442 INFO:tasks.workunit.client.0.smithi145.stderr:report 539213905 2021-01-15T19:02:34.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1604: wait_for_clean: return 1 2021-01-15T19:02:34.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2021-01-15T19:02:34.443 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2021-01-15T19:02:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2021-01-15T19:02:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2021-01-15T19:02:34.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2021-01-15T19:02:34.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:02:34.444 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:02:34.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:02:34.693 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:02:34.693 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:02:34.693 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:02:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:02:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:02:34.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:02:34.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803833 2021-01-15T19:02:34.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803833 2021-01-15T19:02:34.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803833' 2021-01-15T19:02:34.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:02:34.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:02:34.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607608 2021-01-15T19:02:34.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607608 2021-01-15T19:02:34.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803833 1-51539607608' 2021-01-15T19:02:34.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:02:34.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:02:35.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378678 2021-01-15T19:02:35.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378678 2021-01-15T19:02:35.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803833 1-51539607608 2-81604378678' 2021-01-15T19:02:35.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:02:35.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:02:35.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=115964117044 2021-01-15T19:02:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 115964117044 2021-01-15T19:02:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803833 1-51539607608 2-81604378678 3-115964117044' 2021-01-15T19:02:35.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:02:35.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803833 2021-01-15T19:02:35.237 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:02:35.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:02:35.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803833 2021-01-15T19:02:35.238 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:02:35.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803833 2021-01-15T19:02:35.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2021-01-15T19:02:35.239 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803833 2021-01-15T19:02:35.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:02:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803831 -lt 25769803833 2021-01-15T19:02:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:02:36.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:02:36.624 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:02:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803833 -lt 25769803833 2021-01-15T19:02:37.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:02:37.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607608 2021-01-15T19:02:37.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:02:37.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:02:37.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607608 2021-01-15T19:02:37.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:02:37.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607608 2021-01-15T19:02:37.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607608' 2021-01-15T19:02:37.027 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607608 2021-01-15T19:02:37.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:02:37.431 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607608 -lt 51539607608 2021-01-15T19:02:37.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:02:37.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378678 2021-01-15T19:02:37.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:02:37.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:02:37.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378678 2021-01-15T19:02:37.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:02:37.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378678 2021-01-15T19:02:37.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378678' 2021-01-15T19:02:37.437 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378678 2021-01-15T19:02:37.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:02:37.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378678 -lt 81604378678 2021-01-15T19:02:37.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:02:37.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-115964117044 2021-01-15T19:02:37.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:02:37.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:02:37.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-115964117044 2021-01-15T19:02:37.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:02:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=115964117044 2021-01-15T19:02:37.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 115964117044' 2021-01-15T19:02:37.842 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 115964117044 2021-01-15T19:02:37.842 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:02:38.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 115964117044 -lt 115964117044 2021-01-15T19:02:38.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2021-01-15T19:02:38.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-01-15T19:02:38.614 INFO:tasks.workunit.client.0.smithi145.stderr:dumped all 2021-01-15T19:02:38.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2021-01-15T19:02:38.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2021-01-15T19:02:38.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2021-01-15T19:02:38.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-rep 2021-01-15T19:02:38.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2021-01-15T19:02:39.173 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'pool-rep' does not exist 2021-01-15T19:02:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:02:39.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:02:39.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:02:39.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:02:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:02:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:02:39.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:02:39.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:02:39.292 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:02:39.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:02:39.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:02:39.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:02:39.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:02:39.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:02:39.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:02:39.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:02:39.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:02:39.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:02:39.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:02:39.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:02:39.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:02:39.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:02:39.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:02:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:02:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2021-01-15T19:02:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:02:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:02:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:02:39.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:02:39.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:02:39.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:02:39.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:02:39.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:02:39.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:02:39.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:02:39.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:02:39.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:02:39.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:02:39.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:02:39.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:02:39.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:02:39.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:02:39.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:02:39.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:02:39.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:02:39.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:02:39.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:02:39.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:02:39.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:02:39.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:02:39.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.937479 2021-01-15T19:02:39.325 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:02:39.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T19:02:39.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:02:39.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2021-01-15T19:02:39.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2021-01-15T19:02:39.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:39.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:02:39.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:02:39.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:02:39.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2021-01-15T19:02:39.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2021-01-15T19:02:39.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:02:39.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:02:39.637 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:02:39.637 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:02:39.637 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.637 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:39.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon_osd_warn_num_repaired=10 2021-01-15T19:02:39.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:02:39.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:02:39.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:02:39.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:02:39.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:02:39.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:02:39.814 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:02:39.815 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:02:39.815 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:02:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:02:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:02:39.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:02:39.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get fsid 2021-01-15T19:02:39.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:02:39.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:02:39.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:02:39.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:02:39.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:02:39.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:02:39.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:02:39.919 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:02:39.919 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:02:39.919 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:39.919 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:39.920 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:02:39.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:02:39.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get mon_host 2021-01-15T19:02:40.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2021-01-15T19:02:40.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:40.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:02:40.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:02:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:02:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2021-01-15T19:02:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:02:40.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:02:40.548 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:02:40.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:02:40.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:02:40.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:40.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:40.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:40.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:02:40.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:02:40.569 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2021-01-15T19:02:40.906 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:02:40.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2021-01-15T19:02:40.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:40.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2021-01-15T19:02:40.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2021-01-15T19:02:40.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2021-01-15T19:02:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2021-01-15T19:02:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2021-01-15T19:02:40.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2021-01-15T19:02:40.918 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2021-01-15T19:02:40.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2021-01-15T19:02:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:02:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2021-01-15T19:02:40.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:02:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:02:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:02:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:02:40.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T19:02:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:02:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:02:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:02:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:02:40.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:02:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:02:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:02:40.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:02:40.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:02:40.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:02:40.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:02:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:02:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:40.925 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:40.925 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:02:40.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:02:40.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:02:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:02:40.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:02:40.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:02:40.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=b81b33ad-3020-40fd-93ff-e43df16a8d23 2021-01-15T19:02:40.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 b81b33ad-3020-40fd-93ff-e43df16a8d23' 2021-01-15T19:02:40.929 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:02:40.930 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 b81b33ad-3020-40fd-93ff-e43df16a8d23 2021-01-15T19:02:40.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDQ5gFgAb+nNxAAdsgkwoK5I57lYdodOMwZhw== 2021-01-15T19:02:40.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDQ5gFgAb+nNxAAdsgkwoK5I57lYdodOMwZhw=="}' 2021-01-15T19:02:40.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new b81b33ad-3020-40fd-93ff-e43df16a8d23 -i td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:02:41.321 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:02:41.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:02:41.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDQ5gFgAb+nNxAAdsgkwoK5I57lYdodOMwZhw== --osd-uuid b81b33ad-3020-40fd-93ff-e43df16a8d23 2021-01-15T19:02:41.354 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.353+0000 7f395fb7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:41.356 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.355+0000 7f395fb7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:41.357 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.356+0000 7f395fb7f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.357+0000 7f395fb7f080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:02:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.357+0000 7f395fb7f080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:02:41.358 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.357+0000 7f395fb7f080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:02:41.387 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:41.386+0000 7f395fb7f080 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2021-01-15T19:02:45.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2021-01-15T19:02:45.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:02:45.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:02:45.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:02:45.930 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:02:46.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:02:46.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:02:46.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:02:46.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:02:46.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:02:46.455 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:02:46.473 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:46.472+0000 7f33c1347080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:46.475 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:46.474+0000 7f33c1347080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:46.477 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:46.476+0000 7f33c1347080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:46.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:02:46.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:02:46.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:02:46.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:02:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:02:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:02:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:02:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:02:46.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:02:46.848 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:02:46.998 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:46.997+0000 7f33c1347080 -1 Falling back to public interface 2021-01-15T19:02:47.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:02:47.820 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:47.819+0000 7f33c1347080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:02:48.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:02:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:02:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:02:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:02:48.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:02:48.259 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:02:48.697 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/949234,v1:127.0.0.1:6803/949234] [v2:127.0.0.1:6804/949234,v1:127.0.0.1:6805/949234] exists,up b81b33ad-3020-40fd-93ff-e43df16a8d23 2021-01-15T19:02:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:02:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:02:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:02:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:02:48.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2021-01-15T19:02:48.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:48.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:02:48.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:02:48.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:02:48.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:02:48.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T19:02:48.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:02:48.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:02:48.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:02:48.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:02:48.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:02:48.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:02:48.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:02:48.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:02:48.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:02:48.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:02:48.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:02:48.702 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:02:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:48.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:48.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:02:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:02:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:02:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:02:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:02:48.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:02:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:02:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:02:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:02:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:02:48.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:02:48.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:02:48.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=01a88c23-419c-40e9-86f3-1ebcafe8c8ab 2021-01-15T19:02:48.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 01a88c23-419c-40e9-86f3-1ebcafe8c8ab' 2021-01-15T19:02:48.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:02:48.707 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 01a88c23-419c-40e9-86f3-1ebcafe8c8ab 2021-01-15T19:02:48.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDY5gFg3w1UKhAAFNu0cLrq/MWE2MKRqrBXEQ== 2021-01-15T19:02:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDY5gFg3w1UKhAAFNu0cLrq/MWE2MKRqrBXEQ=="}' 2021-01-15T19:02:48.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 01a88c23-419c-40e9-86f3-1ebcafe8c8ab -i td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:02:49.336 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:02:49.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:02:49.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDY5gFg3w1UKhAAFNu0cLrq/MWE2MKRqrBXEQ== --osd-uuid 01a88c23-419c-40e9-86f3-1ebcafe8c8ab 2021-01-15T19:02:49.370 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.369+0000 7f482aec6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:49.403 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.402+0000 7f482aec6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:49.404 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.403+0000 7f482aec6080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:49.405 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.404+0000 7f482aec6080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:02:49.405 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.404+0000 7f482aec6080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:02:49.405 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.404+0000 7f482aec6080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:02:49.486 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:49.485+0000 7f482aec6080 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2021-01-15T19:02:54.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2021-01-15T19:02:54.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:02:54.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:02:54.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:02:54.229 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:02:54.753 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:02:54.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:02:54.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:02:54.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:02:54.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:02:54.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:02:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:54.771+0000 7fa047d59080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:54.774 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:54.773+0000 7fa047d59080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:54.774+0000 7fa047d59080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:55.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:02:55.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:02:55.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:02:55.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:02:55.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:02:55.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:02:55.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:02:55.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:02:55.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:02:55.145 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:02:55.292 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:55.291+0000 7fa047d59080 -1 Falling back to public interface 2021-01-15T19:02:55.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:02:56.086 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:56.084+0000 7fa047d59080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:02:56.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:02:56.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:02:56.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:02:56.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:02:56.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:02:56.534 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:02:56.939 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/950498,v1:127.0.0.1:6811/950498] [v2:127.0.0.1:6812/950498,v1:127.0.0.1:6813/950498] exists,up 01a88c23-419c-40e9-86f3-1ebcafe8c8ab 2021-01-15T19:02:56.940 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 4623e2a9-9c61-4dfa-be58-85b263b506cc 2021-01-15T19:02:56.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:02:56.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:02:56.940 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:02:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:02:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2021-01-15T19:02:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:02:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:02:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T19:02:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:02:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:02:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T19:02:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:02:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:02:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:02:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T19:02:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T19:02:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:02:56.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:02:56.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:02:56.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:02:56.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:02:56.944 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:02:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:02:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:02:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:02:56.945 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:56.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:02:56.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:02:56.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:02:56.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:02:56.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:02:56.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:02:56.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:02:56.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:02:56.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:02:56.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:02:56.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:02:56.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T19:02:56.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:02:56.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4623e2a9-9c61-4dfa-be58-85b263b506cc 2021-01-15T19:02:56.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 4623e2a9-9c61-4dfa-be58-85b263b506cc' 2021-01-15T19:02:56.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:02:56.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDg5gFgo3m2OBAAH/fVm7mRmeBxbUoUcad4PQ== 2021-01-15T19:02:56.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDg5gFgo3m2OBAAH/fVm7mRmeBxbUoUcad4PQ=="}' 2021-01-15T19:02:56.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4623e2a9-9c61-4dfa-be58-85b263b506cc -i td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:02:57.682 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:02:57.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:02:57.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDg5gFgo3m2OBAAH/fVm7mRmeBxbUoUcad4PQ== --osd-uuid 4623e2a9-9c61-4dfa-be58-85b263b506cc 2021-01-15T19:02:57.717 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.716+0000 7fa26c710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:57.719 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.718+0000 7fa26c710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:57.720 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.719+0000 7fa26c710080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:02:57.721 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.720+0000 7fa26c710080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:02:57.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.720+0000 7fa26c710080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:02:57.722 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.720+0000 7fa26c710080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:02:57.802 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:02:57.801+0000 7fa26c710080 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2021-01-15T19:03:02.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2021-01-15T19:03:02.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:03:02.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T19:03:02.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:03:02.546 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T19:03:03.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T19:03:03.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:03:03.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:03:03.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:03:03.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:03:03.072 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:03:03.090 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:03.089+0000 7f1d8d0f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:03.092 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:03.091+0000 7f1d8d0f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:03.093 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:03.092+0000 7f1d8d0f2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:03.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T19:03:03.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:03:03.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:03:03.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:03:03.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:03:03.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:03:03.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:03:03.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:03:03.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:03:03.464 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:03:03.612 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:03.611+0000 7f1d8d0f2080 -1 Falling back to public interface 2021-01-15T19:03:03.881 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:03:04.455 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:04.454+0000 7f1d8d0f2080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T19:03:04.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:03:04.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:03:04.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:03:04.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:03:04.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:03:04.884 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:03:05.332 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/951762,v1:127.0.0.1:6819/951762] [v2:127.0.0.1:6820/951762,v1:127.0.0.1:6821/951762] exists,up 4623e2a9-9c61-4dfa-be58-85b263b506cc 2021-01-15T19:03:05.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:03:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:03:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:03:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:03:05.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2021-01-15T19:03:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:03:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=3 2021-01-15T19:03:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:03:05.334 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2021-01-15T19:03:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2021-01-15T19:03:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:03:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:03:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:03:05.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2021-01-15T19:03:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2021-01-15T19:03:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:03:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:03:05.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:03:05.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:03:05.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:03:05.337 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:03:05.337 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:03:05.337 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:05.337 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:05.338 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:03:05.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:03:05.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:03:05.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:03:05.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:03:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:03:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:03:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:03:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:03:05.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:03:05.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:03:05.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:03:05.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2021-01-15T19:03:05.340 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:03:05.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=54c459f7-bd9a-4e3d-a31c-10baaa11af1b 2021-01-15T19:03:05.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd3 54c459f7-bd9a-4e3d-a31c-10baaa11af1b' 2021-01-15T19:03:05.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:03:05.342 INFO:tasks.workunit.client.0.smithi145.stdout:add osd3 54c459f7-bd9a-4e3d-a31c-10baaa11af1b 2021-01-15T19:03:05.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDp5gFgYlGPFBAATHMYOQWipVZ8XgXby+TdMw== 2021-01-15T19:03:05.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDp5gFgYlGPFBAATHMYOQWipVZ8XgXby+TdMw=="}' 2021-01-15T19:03:05.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 54c459f7-bd9a-4e3d-a31c-10baaa11af1b -i td/osd-rep-recov-eio.sh/3/new.json 2021-01-15T19:03:06.236 INFO:tasks.workunit.client.0.smithi145.stdout:3 2021-01-15T19:03:06.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2021-01-15T19:03:06.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 3 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDp5gFgYlGPFBAATHMYOQWipVZ8XgXby+TdMw== --osd-uuid 54c459f7-bd9a-4e3d-a31c-10baaa11af1b 2021-01-15T19:03:06.269 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.268+0000 7f6698d18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:06.271 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.270+0000 7f6698d18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:06.272 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.272+0000 7f6698d18080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:06.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.272+0000 7f6698d18080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T19:03:06.273 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.272+0000 7f6698d18080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T19:03:06.274 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.272+0000 7f6698d18080 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2021-01-15T19:03:06.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:06.315+0000 7f6698d18080 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2021-01-15T19:03:11.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2021-01-15T19:03:11.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:03:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd3 key to auth repository 2021-01-15T19:03:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:03:11.210 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd3 key to auth repository 2021-01-15T19:03:11.743 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.3 2021-01-15T19:03:11.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.3 2021-01-15T19:03:11.743 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 3 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:03:11.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:03:11.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:03:11.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:03:11.762 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:11.761+0000 7ffb92649080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:11.764 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:11.763+0000 7ffb92649080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:11.766 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:11.765+0000 7ffb92649080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:03:12.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 3 2021-01-15T19:03:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:03:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=3 2021-01-15T19:03:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:03:12.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:03:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:03:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:03:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:03:12.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T19:03:12.137 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:03:12.286 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:12.284+0000 7ffb92649080 -1 Falling back to public interface 2021-01-15T19:03:12.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:03:13.101 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:03:13.100+0000 7ffb92649080 -1 osd.3 0 log_to_monitors {default=true} 2021-01-15T19:03:13.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:03:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:03:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:03:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:03:13.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T19:03:13.548 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:03:14.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:03:15.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:03:15.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:03:15.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T19:03:15.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:03:15.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.3 up' 2021-01-15T19:03:15.004 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:03:15.401 INFO:tasks.workunit.client.0.smithi145.stdout:osd.3 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/953026,v1:127.0.0.1:6827/953026] [v2:127.0.0.1:6828/953026,v1:127.0.0.1:6829/953026] exists,up 54c459f7-bd9a-4e3d-a31c-10baaa11af1b 2021-01-15T19:03:15.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:03:15.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:03:15.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:03:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2021-01-15T19:03:15.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:03:15.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:03:15.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:03:15.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:03:15.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:03:15.407 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:03:15.407 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:03:15.408 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:03:15.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:03:15.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:03:15.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:03:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:03:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:03:15.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:03:15.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:03:15.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:03:15.884 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:15.884 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:03:15.884 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:03:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:03:15.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:15.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:03:16.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803784 2021-01-15T19:03:16.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803784 2021-01-15T19:03:16.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784' 2021-01-15T19:03:16.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:16.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:03:16.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574854 2021-01-15T19:03:16.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574854 2021-01-15T19:03:16.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854' 2021-01-15T19:03:16.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:16.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:03:16.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378629 2021-01-15T19:03:16.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378629 2021-01-15T19:03:16.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-81604378629' 2021-01-15T19:03:16.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:16.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:03:16.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149700 2021-01-15T19:03:16.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149700 2021-01-15T19:03:16.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-55834574854 2-81604378629 3-111669149700' 2021-01-15T19:03:16.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:16.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803784 2021-01-15T19:03:16.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:16.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:03:16.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803784 2021-01-15T19:03:16.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:16.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803784 2021-01-15T19:03:16.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2021-01-15T19:03:16.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:03:16.421 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803784 2021-01-15T19:03:16.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803784 2021-01-15T19:03:16.833 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:16.833 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574854 2021-01-15T19:03:16.834 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:16.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:03:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574854 2021-01-15T19:03:16.836 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574854 2021-01-15T19:03:16.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574854' 2021-01-15T19:03:16.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:03:16.838 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574854 2021-01-15T19:03:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574854 -lt 55834574854 2021-01-15T19:03:17.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:17.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378629 2021-01-15T19:03:17.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:17.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:03:17.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378629 2021-01-15T19:03:17.221 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378629 2021-01-15T19:03:17.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378629' 2021-01-15T19:03:17.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:03:17.223 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378629 2021-01-15T19:03:17.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378629 2021-01-15T19:03:17.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:17.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149700 2021-01-15T19:03:17.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:17.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:03:17.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149700 2021-01-15T19:03:17.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:17.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149700 2021-01-15T19:03:17.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149700' 2021-01-15T19:03:17.621 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:17.622 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149700 2021-01-15T19:03:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149698 -lt 111669149700 2021-01-15T19:03:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:03:19.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:03:19.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:19.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149700 -lt 111669149700 2021-01-15T19:03:19.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:03:19.415 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:19.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T19:03:19.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:03:19.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:03:19.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:03:19.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:03:19.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:03:19.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:03:19.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:03:20.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:03:20.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:03:20.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:20.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:20.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T19:03:20.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:03:20.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:03:20.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2021-01-15T19:03:20.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2021-01-15T19:03:20.843 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create pool-rep 1 1 2021-01-15T19:03:21.380 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'pool-rep' already exists 2021-01-15T19:03:21.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T19:03:22.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2021-01-15T19:03:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:03:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:03:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:03:22.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:03:22.393 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:03:22.393 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:03:22.394 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:03:22.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:03:22.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:03:22.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:03:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:03:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:03:22.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:03:22.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:03:22.496 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:03:22.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:03:22.900 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:22.900 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:03:22.900 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:03:22.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:03:22.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:22.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:03:23.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803787 2021-01-15T19:03:23.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803787 2021-01-15T19:03:23.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787' 2021-01-15T19:03:23.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:23.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:03:23.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574858 2021-01-15T19:03:23.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574858 2021-01-15T19:03:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858' 2021-01-15T19:03:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:23.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:03:23.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378632 2021-01-15T19:03:23.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378632 2021-01-15T19:03:23.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-81604378632' 2021-01-15T19:03:23.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:23.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:03:23.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149703 2021-01-15T19:03:23.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149703 2021-01-15T19:03:23.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803787 1-55834574858 2-81604378632 3-111669149703' 2021-01-15T19:03:23.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:23.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803787 2021-01-15T19:03:23.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:23.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:03:23.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803787 2021-01-15T19:03:23.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803787 2021-01-15T19:03:23.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803787' 2021-01-15T19:03:23.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:03:23.522 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803787 2021-01-15T19:03:23.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803785 -lt 25769803787 2021-01-15T19:03:23.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:03:24.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:03:24.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:03:25.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803787 2021-01-15T19:03:25.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:25.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574858 2021-01-15T19:03:25.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:25.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:03:25.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574858 2021-01-15T19:03:25.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:25.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574858 2021-01-15T19:03:25.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574858' 2021-01-15T19:03:25.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:03:25.316 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574858 2021-01-15T19:03:25.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574858 -lt 55834574858 2021-01-15T19:03:25.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:25.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378632 2021-01-15T19:03:25.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:25.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:03:25.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378632 2021-01-15T19:03:25.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:25.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378632 2021-01-15T19:03:25.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378632' 2021-01-15T19:03:25.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:03:25.708 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378632 2021-01-15T19:03:26.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378632 -lt 81604378632 2021-01-15T19:03:26.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:26.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149703 2021-01-15T19:03:26.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:26.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:03:26.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149703 2021-01-15T19:03:26.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:26.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149703 2021-01-15T19:03:26.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149703' 2021-01-15T19:03:26.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:26.095 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149703 2021-01-15T19:03:26.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149704 -lt 111669149703 2021-01-15T19:03:26.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:03:26.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:26.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:27.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:03:27.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:03:27.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:03:27.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:03:27.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:03:27.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:03:27.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:03:27.035 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:03:27.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:03:27.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:03:27.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:27.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:27.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:03:27.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:03:27.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:03:27.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2021-01-15T19:03:27.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2021-01-15T19:03:27.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2021-01-15T19:03:27.924 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2021-01-15T19:03:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2021-01-15T19:03:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:27.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2021-01-15T19:03:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:27.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:27.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:27.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:27.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:27.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:27.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:28.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:28.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:28.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:28.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-1 2021-01-15T19:03:28.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:28.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:28.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-1 2021-01-15T19:03:28.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:28.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-1 2021-01-15T19:03:28.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-1 2021-01-15T19:03:28.068 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:28.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:28.467 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:28.467 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:28.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:28.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:28.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:28.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:28.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:28.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:28.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:28.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:28.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:28.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:28.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:28.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:28.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:28.476 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:28.476 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:28.477 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:28.478 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:28.478 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:28.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:28.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:28.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:28.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:28.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:28.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:28.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:28.581 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:28.582 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:28.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:28.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:28.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2021-01-15T19:03:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2021-01-15T19:03:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:28.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2021-01-15T19:03:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:28.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:28.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:28.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:28.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2021-01-15T19:03:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2021-01-15T19:03:28.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:28.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:28.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:28.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:28.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:28.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:28.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:28.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:28.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:28.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:28.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:28.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:28.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-2 2021-01-15T19:03:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:28.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:28.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:28.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:28.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-2 2021-01-15T19:03:28.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:28.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-2 2021-01-15T19:03:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-2 2021-01-15T19:03:28.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:29.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:29.379 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:29.379 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:29.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:29.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:29.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:29.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:29.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:29.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:29.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:29.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:29.384 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:29.385 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:29.386 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:29.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:29.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:29.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:29.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:29.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:29.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:29.492 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:29.492 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:29.492 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:29.493 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:29.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:29.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:29.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2021-01-15T19:03:29.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2021-01-15T19:03:29.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:29.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:29.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2021-01-15T19:03:29.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:29.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:29.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:29.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:29.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:29.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:29.727 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2021-01-15T19:03:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:29.728 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:29.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2021-01-15T19:03:29.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:29.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:29.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:29.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:29.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:29.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:29.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:29.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:29.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-3 2021-01-15T19:03:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:29.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:29.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:29.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:29.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-3 2021-01-15T19:03:29.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:29.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-3 2021-01-15T19:03:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-3 2021-01-15T19:03:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:30.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:30.335 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:30.335 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:30.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:30.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:30.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:30.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:30.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:30.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:30.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:30.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:30.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:30.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:30.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:30.339 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:30.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:30.341 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:30.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:30.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:30.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:30.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:30.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:30.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:30.446 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:30.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:30.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2021-01-15T19:03:30.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2021-01-15T19:03:30.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:30.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2021-01-15T19:03:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:30.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:30.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:30.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:30.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:30.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2021-01-15T19:03:30.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2021-01-15T19:03:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:30.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:30.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:30.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:30.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:30.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:30.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:30.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:30.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:30.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:30.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:30.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:30.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-4 2021-01-15T19:03:30.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:30.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:30.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:30.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:30.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:30.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-4 2021-01-15T19:03:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:30.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-4 2021-01-15T19:03:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-4 2021-01-15T19:03:30.877 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:31.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:31.267 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:31.268 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:31.268 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:31.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:31.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:31.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:31.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:31.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:31.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:31.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:31.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:31.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:31.272 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:31.273 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:31.273 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:31.273 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:31.273 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:31.274 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:31.274 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:31.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:31.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:31.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:31.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:31.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:31.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:31.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:31.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:31.380 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:31.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:31.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2021-01-15T19:03:31.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2021-01-15T19:03:31.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:31.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:31.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2021-01-15T19:03:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:31.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:31.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:31.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2021-01-15T19:03:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2021-01-15T19:03:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:31.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:31.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:31.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:31.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:31.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:31.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-5 2021-01-15T19:03:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:31.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:31.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:31.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-5 2021-01-15T19:03:31.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:31.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-5 2021-01-15T19:03:31.796 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-5 2021-01-15T19:03:31.796 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:32.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:32.180 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:32.181 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:32.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:32.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:32.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:32.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:32.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:32.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:32.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:32.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:32.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:32.185 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:32.186 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:32.186 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:32.186 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:32.186 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:32.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:32.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:32.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:32.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:32.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:32.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:32.295 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:32.296 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:32.296 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:32.297 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:32.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:32.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2021-01-15T19:03:32.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2021-01-15T19:03:32.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:32.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:32.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2021-01-15T19:03:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:32.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:32.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:32.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2021-01-15T19:03:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:32.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2021-01-15T19:03:32.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:32.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:32.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:32.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:32.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:32.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:32.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:32.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:32.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:32.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:32.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:32.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:32.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:32.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:32.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:32.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:32.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-6 2021-01-15T19:03:32.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:32.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:32.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:32.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-6 2021-01-15T19:03:32.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:32.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-6 2021-01-15T19:03:32.718 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-6 2021-01-15T19:03:32.718 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:33.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:33.107 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:33.107 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:33.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:33.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:33.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:33.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:33.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:33.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:33.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:33.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:33.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:33.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:33.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:33.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:33.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:33.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:33.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:33.112 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:33.113 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:33.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:33.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:33.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:33.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:33.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:33.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:33.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:33.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:33.220 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:33.220 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:33.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:33.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2021-01-15T19:03:33.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2021-01-15T19:03:33.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:33.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:33.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2021-01-15T19:03:33.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:33.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:33.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:33.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:33.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:33.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:33.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2021-01-15T19:03:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2021-01-15T19:03:33.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:33.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:33.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:33.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:33.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:33.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:33.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:33.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:33.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:33.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:33.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:33.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:33.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:33.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:33.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:33.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:33.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-7 2021-01-15T19:03:33.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:33.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:33.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:33.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:33.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:33.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:33.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-7 2021-01-15T19:03:33.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:33.628 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-7 2021-01-15T19:03:33.628 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-7 2021-01-15T19:03:33.628 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:34.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:34.018 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:34.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:34.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:34.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:34.020 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:34.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:34.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:34.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:34.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:34.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:34.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:34.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:34.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:34.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:34.023 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:34.023 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:34.024 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:34.024 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:34.024 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:34.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:34.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:34.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:34.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:34.129 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:34.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:34.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:34.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:34.130 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:34.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:34.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2021-01-15T19:03:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2021-01-15T19:03:34.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:34.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:34.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2021-01-15T19:03:34.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:34.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:34.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:34.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:34.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:34.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:34.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2021-01-15T19:03:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2021-01-15T19:03:34.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:34.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:34.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:34.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:34.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:34.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:34.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:34.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-8 2021-01-15T19:03:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:34.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:34.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:34.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-8 2021-01-15T19:03:34.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:34.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-8 2021-01-15T19:03:34.848 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-8 2021-01-15T19:03:34.848 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:35.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:35.273 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:35.273 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:35.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:35.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:35.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:35.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:35.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:35.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:35.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:35.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:35.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:35.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:35.278 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:35.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:35.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:35.279 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:35.280 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:35.281 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:35.281 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:35.281 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:35.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:35.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:35.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:35.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:35.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:35.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:35.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:35.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:35.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:35.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:35.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:35.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:35.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2021-01-15T19:03:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2021-01-15T19:03:35.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2021-01-15T19:03:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:35.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:35.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:35.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:35.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:35.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:35.621 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2021-01-15T19:03:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2021-01-15T19:03:35.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:35.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:35.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:35.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:35.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:35.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:35.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:35.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-9 2021-01-15T19:03:35.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:35.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:35.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-9 2021-01-15T19:03:35.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:35.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-9 2021-01-15T19:03:35.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-9 2021-01-15T19:03:35.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:36.210 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:36.210 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:36.210 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:36.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:36.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:36.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:36.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:36.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:36.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:36.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:36.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:36.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:36.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:36.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:36.215 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:36.216 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:36.216 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:36.216 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:36.216 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:36.216 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:36.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:36.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:36.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:36.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:36.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:36.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:36.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:36.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:36.322 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:36.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:36.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:36.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2021-01-15T19:03:36.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2021-01-15T19:03:36.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:36.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:36.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2021-01-15T19:03:36.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:36.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:36.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:36.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:36.559 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:36.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:36.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2021-01-15T19:03:36.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:36.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:36.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2021-01-15T19:03:36.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:36.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:36.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:36.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:36.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:36.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:36.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:36.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:36.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:36.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:36.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:36.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:36.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:36.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:36.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:36.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:36.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-10 2021-01-15T19:03:36.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:36.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:36.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:36.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:36.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:36.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:36.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-10 2021-01-15T19:03:36.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:36.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-10 2021-01-15T19:03:36.762 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-10 2021-01-15T19:03:36.762 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:37.150 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:37.151 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:37.151 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:37.151 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:37.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:37.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:37.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:37.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:37.153 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:37.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:37.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:37.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:37.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:37.156 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:37.158 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:37.159 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:37.159 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:37.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:37.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:37.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:37.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:37.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:37.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:37.260 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:37.260 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:37.261 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:37.261 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:37.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:37.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2021-01-15T19:03:37.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2021-01-15T19:03:37.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:37.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:37.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2021-01-15T19:03:37.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:37.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:37.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2021-01-15T19:03:37.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2021-01-15T19:03:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2021-01-15T19:03:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:03:37.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:03:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:03:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:03:37.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:03:37.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:03:37.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:37.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:37.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-11 2021-01-15T19:03:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:37.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:37.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:37.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-11 2021-01-15T19:03:37.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:37.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-11 2021-01-15T19:03:37.697 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-11 2021-01-15T19:03:37.697 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:38.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:38.085 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:38.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:38.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:38.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:38.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:38.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:38.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:38.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:38.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:38.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:38.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:38.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:38.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:38.090 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:38.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:38.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:38.091 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:38.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:38.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:38.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:38.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:38.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:38.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:38.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:38.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:38.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:38.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:38.197 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:38.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:38.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:38.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2021-01-15T19:03:38.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2021-01-15T19:03:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2021-01-15T19:03:38.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:38.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:38.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:38.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:38.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:38.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:38.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2021-01-15T19:03:38.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local poolname=pool-rep 2021-01-15T19:03:38.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: get_pg: local objectname=obj-warn-1 2021-01-15T19:03:38.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: ceph --format json osd map pool-rep obj-warn-1 2021-01-15T19:03:38.437 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: get_pg: jq -r .pgid 2021-01-15T19:03:38.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2021-01-15T19:03:38.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=($(get_osds $poolname ${objbase}-1)) 2021-01-15T19:03:38.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2021-01-15T19:03:38.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:38.841 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-1 2021-01-15T19:03:38.841 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-1 2021-01-15T19:03:38.842 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:39.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:39.231 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2021-01-15T19:03:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2021-01-15T19:03:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2021-01-15T19:03:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2021-01-15T19:03:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:03:39.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:03:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:03:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:03:39.233 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:03:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:03:39.233 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:03:39.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:03:39.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:03:39.234 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:03:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:03:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:03:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:03:39.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:03:39.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:03:39.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:03:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:03:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:03:39.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:03:39.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:39.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:03:39.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803793 2021-01-15T19:03:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803793 2021-01-15T19:03:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803793' 2021-01-15T19:03:39.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:39.885 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:03:40.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574863 2021-01-15T19:03:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574863 2021-01-15T19:03:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803793 1-55834574863' 2021-01-15T19:03:40.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:40.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:03:40.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378637 2021-01-15T19:03:40.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378637 2021-01-15T19:03:40.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803793 1-55834574863 2-81604378637' 2021-01-15T19:03:40.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:40.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:03:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149709 2021-01-15T19:03:40.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149709 2021-01-15T19:03:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803793 1-55834574863 2-81604378637 3-111669149709' 2021-01-15T19:03:40.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:40.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803793 2021-01-15T19:03:40.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:40.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:03:40.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803793 2021-01-15T19:03:40.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:40.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803793 2021-01-15T19:03:40.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2021-01-15T19:03:40.289 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:03:40.289 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803793 2021-01-15T19:03:40.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803793 -lt 25769803793 2021-01-15T19:03:40.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:40.682 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574863 2021-01-15T19:03:40.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:40.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:03:40.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574863 2021-01-15T19:03:40.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:40.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574863 2021-01-15T19:03:40.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574863' 2021-01-15T19:03:40.684 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:03:40.684 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574863 2021-01-15T19:03:41.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574863 -lt 55834574863 2021-01-15T19:03:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:41.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378637 2021-01-15T19:03:41.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:41.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:03:41.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378637 2021-01-15T19:03:41.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:41.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378637 2021-01-15T19:03:41.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2021-01-15T19:03:41.079 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:03:41.079 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378637 2021-01-15T19:03:41.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378637 -lt 81604378637 2021-01-15T19:03:41.468 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:41.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149709 2021-01-15T19:03:41.468 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:41.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:03:41.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149709 2021-01-15T19:03:41.471 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:41.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149709 2021-01-15T19:03:41.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149709' 2021-01-15T19:03:41.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:41.473 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149709 2021-01-15T19:03:41.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149709 -lt 111669149709 2021-01-15T19:03:41.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:03:41.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:41.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:42.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:03:42.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:03:42.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:03:42.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:03:42.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:03:42.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:03:42.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:03:42.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:03:42.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:03:42.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:03:42.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:03:42.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:03:43.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:03:43.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:03:43.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:03:43.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2021-01-15T19:03:43.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2021-01-15T19:03:43.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2021-01-15T19:03:43.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2021-01-15T19:03:43.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2021-01-15T19:03:43.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:03:43.457 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:03:43.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:03:43.847 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:43.847 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:03:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:03:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:03:43.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:43.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:03:43.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803795 2021-01-15T19:03:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803795 2021-01-15T19:03:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795' 2021-01-15T19:03:43.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:43.981 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:03:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574866 2021-01-15T19:03:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574866 2021-01-15T19:03:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574866' 2021-01-15T19:03:44.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:44.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:03:44.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378640 2021-01-15T19:03:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378640 2021-01-15T19:03:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574866 2-81604378640' 2021-01-15T19:03:44.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:03:44.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:03:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149711 2021-01-15T19:03:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149711 2021-01-15T19:03:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-55834574866 2-81604378640 3-111669149711' 2021-01-15T19:03:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:44.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803795 2021-01-15T19:03:44.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:44.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:03:44.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803795 2021-01-15T19:03:44.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:44.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803795 2021-01-15T19:03:44.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2021-01-15T19:03:44.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:03:44.384 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803795 2021-01-15T19:03:44.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803796 -lt 25769803795 2021-01-15T19:03:44.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:44.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574866 2021-01-15T19:03:44.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:44.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:03:44.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574866 2021-01-15T19:03:44.797 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:44.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574866 2021-01-15T19:03:44.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574866' 2021-01-15T19:03:44.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:03:44.799 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574866 2021-01-15T19:03:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574866 -lt 55834574866 2021-01-15T19:03:45.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:45.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378640 2021-01-15T19:03:45.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:45.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:03:45.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378640 2021-01-15T19:03:45.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378640 2021-01-15T19:03:45.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2021-01-15T19:03:45.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:03:45.187 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378640 2021-01-15T19:03:45.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378640 -lt 81604378640 2021-01-15T19:03:45.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:03:45.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149711 2021-01-15T19:03:45.579 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:03:45.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:03:45.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149711 2021-01-15T19:03:45.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:03:45.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149711 2021-01-15T19:03:45.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149711' 2021-01-15T19:03:45.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:45.582 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149711 2021-01-15T19:03:45.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149709 -lt 111669149711 2021-01-15T19:03:45.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:03:46.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:03:46.967 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:03:47.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149712 -lt 111669149711 2021-01-15T19:03:47.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2021-01-15T19:03:47.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-01-15T19:03:47.753 INFO:tasks.workunit.client.0.smithi145.stderr:dumped all 2021-01-15T19:03:47.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2021-01-15T19:03:47.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2021-01-15T19:03:47.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2021-01-15T19:03:47.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2021-01-15T19:03:48.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2021-01-15T19:03:48.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2021-01-15T19:03:48.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2021-01-15T19:03:49.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2021-01-15T19:03:49.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2021-01-15T19:03:49.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2021-01-15T19:03:50.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2021-01-15T19:03:50.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2021-01-15T19:03:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:50.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:50.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:50.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:50.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:50.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:50.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-1 2021-01-15T19:03:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:50.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:50.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:50.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:50.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-1 2021-01-15T19:03:50.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:50.040 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-1 2021-01-15T19:03:50.041 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-1 2021-01-15T19:03:50.041 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:50.433 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:50.433 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:50.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:50.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:50.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:50.435 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:50.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:50.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:50.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:50.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:50.439 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:50.440 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:50.441 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:50.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:50.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:50.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:50.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:50.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:50.546 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:50.546 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:50.547 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:50.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2021-01-15T19:03:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:50.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:50.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-1 2021-01-15T19:03:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:50.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:50.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:50.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-1 2021-01-15T19:03:50.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:50.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-1 2021-01-15T19:03:50.661 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-1 2021-01-15T19:03:50.661 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:51.048 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:51.049 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:51.049 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:51.049 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:51.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:51.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:51.051 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:51.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:51.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:51.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:51.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:51.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:51.054 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:51.054 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:51.055 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:51.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:51.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:51.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:51.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:51.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:51.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:51.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:51.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:51.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:51.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2021-01-15T19:03:51.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:51.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:51.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:51.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:51.272 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:51.273 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:51.273 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:51.273 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:51.273 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:51.274 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:51.274 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:51.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2021-01-15T19:03:51.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:51.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:51.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2021-01-15T19:03:51.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:51.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:51.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:51.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:51.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:51.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:51.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:51.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:51.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:51.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:51.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:51.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:51.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:51.642 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-2 2021-01-15T19:03:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:51.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:51.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:51.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:51.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:51.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:51.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-2 2021-01-15T19:03:51.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:51.647 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-2 2021-01-15T19:03:51.647 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-2 2021-01-15T19:03:51.648 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:52.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:52.029 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:52.029 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:52.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:52.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:52.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:52.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:52.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:52.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:52.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:52.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:52.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:52.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:52.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:52.036 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:52.036 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:52.037 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:52.037 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:52.037 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:52.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:52.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:52.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:52.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:52.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:52.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:52.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:52.140 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:52.141 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:52.141 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:52.141 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:52.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:52.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2021-01-15T19:03:52.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:52.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:52.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:52.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:52.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-2 2021-01-15T19:03:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:52.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:52.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:52.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:52.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:52.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:52.253 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-2 2021-01-15T19:03:52.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:52.254 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-2 2021-01-15T19:03:52.254 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-2 2021-01-15T19:03:52.254 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:52.643 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:52.643 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:52.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:52.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:52.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:52.645 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:52.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:52.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:52.648 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:52.649 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:52.650 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:52.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:52.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:52.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:52.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:52.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:52.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:52.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:52.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:52.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2021-01-15T19:03:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:52.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:52.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:52.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:52.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:52.865 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:52.865 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:52.866 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:52.867 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:52.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2021-01-15T19:03:52.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2021-01-15T19:03:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:52.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:52.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:53.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:53.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:53.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:53.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:53.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:53.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:53.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:53.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:53.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:53.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:53.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-3 2021-01-15T19:03:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:53.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:53.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:53.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-3 2021-01-15T19:03:53.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:53.208 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-3 2021-01-15T19:03:53.208 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-3 2021-01-15T19:03:53.209 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:53.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:53.600 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:53.600 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:53.600 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:53.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:53.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:53.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:53.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:53.604 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:53.604 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:53.605 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:53.606 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:53.606 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:53.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:53.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:53.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:53.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:53.712 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:53.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:53.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:53.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:53.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:53.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2021-01-15T19:03:53.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:53.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-3 2021-01-15T19:03:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:53.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:53.824 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:53.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-3 2021-01-15T19:03:53.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:53.825 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-3 2021-01-15T19:03:53.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-3 2021-01-15T19:03:53.825 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:54.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:54.211 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:54.211 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:54.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:54.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:54.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:54.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:54.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:54.213 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:54.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:54.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:54.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:54.215 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:54.216 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:54.217 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:54.217 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:54.217 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:54.217 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:54.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:54.321 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:54.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:54.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:54.322 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:54.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:54.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:54.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:54.323 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:54.323 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:54.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:54.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2021-01-15T19:03:54.432 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:54.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:54.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:54.435 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:54.436 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:54.436 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:54.436 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:54.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:54.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2021-01-15T19:03:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2021-01-15T19:03:54.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:54.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:54.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:54.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:54.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:54.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:54.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:54.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:54.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:54.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:54.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-4 2021-01-15T19:03:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:54.775 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:54.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:54.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-4 2021-01-15T19:03:54.776 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:54.777 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-4 2021-01-15T19:03:54.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-4 2021-01-15T19:03:54.777 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:55.167 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:55.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:55.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:55.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:55.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:55.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:55.173 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:55.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:55.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:55.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:55.279 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:55.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:55.280 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:55.280 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:55.280 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:55.280 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:55.281 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:55.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:55.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2021-01-15T19:03:55.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:55.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:55.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:55.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:55.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-4 2021-01-15T19:03:55.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:55.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:55.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:55.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:55.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:55.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:55.394 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-4 2021-01-15T19:03:55.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:55.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-4 2021-01-15T19:03:55.395 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-4 2021-01-15T19:03:55.395 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:55.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:55.784 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:55.785 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:55.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:55.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:55.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:55.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:55.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:55.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:55.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:55.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:55.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:55.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:55.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:55.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:55.789 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:55.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:55.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:55.789 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:55.790 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:55.790 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:55.790 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:55.790 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:55.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:55.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:55.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:55.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:55.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:55.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:55.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:55.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:55.898 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:55.899 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:55.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:55.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:55.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2021-01-15T19:03:56.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:56.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:56.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:56.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:56.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:56.008 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:56.008 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:56.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:56.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:56.009 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:56.009 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:56.009 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:56.010 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:56.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2021-01-15T19:03:56.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:56.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:56.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2021-01-15T19:03:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:56.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:56.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:56.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:56.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:56.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:56.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:56.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:56.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:56.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:56.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:56.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-5 2021-01-15T19:03:56.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:56.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:56.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:56.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:56.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:56.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:56.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-5 2021-01-15T19:03:56.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:56.360 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-5 2021-01-15T19:03:56.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-5 2021-01-15T19:03:56.360 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:56.736 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:56.737 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:56.737 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:56.737 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:56.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:56.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:56.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:56.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:56.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:56.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:56.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:56.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:56.743 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:56.743 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:56.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:56.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:56.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:56.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:56.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:56.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:56.850 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:56.850 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:56.850 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:56.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:56.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:56.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2021-01-15T19:03:56.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:56.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:56.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:56.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:56.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-5 2021-01-15T19:03:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:56.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:56.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:56.961 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-5 2021-01-15T19:03:56.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:56.962 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-5 2021-01-15T19:03:56.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-5 2021-01-15T19:03:56.962 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:57.352 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:57.353 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:57.354 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:57.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:57.359 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:57.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:57.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:57.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:57.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:57.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:57.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:57.465 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:57.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:57.466 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:57.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:57.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:57.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2021-01-15T19:03:57.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:57.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:57.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:57.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:57.575 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:57.576 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:57.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:57.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:57.576 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:57.577 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:57.577 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:57.577 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:57.577 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:57.577 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:57.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2021-01-15T19:03:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2021-01-15T19:03:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:57.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:57.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:57.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:57.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:57.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:57.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:57.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:57.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:57.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:57.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-6 2021-01-15T19:03:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:57.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:57.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:57.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:57.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:57.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-6 2021-01-15T19:03:57.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:57.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-6 2021-01-15T19:03:57.916 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-6 2021-01-15T19:03:57.916 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:58.327 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:58.327 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:58.327 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:58.328 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:58.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:58.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:58.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:58.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:58.329 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:58.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:58.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:58.332 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:58.332 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:58.333 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:58.333 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:58.333 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:58.333 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:58.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:58.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:58.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:58.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:03:58.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:58.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:58.441 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:58.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:58.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2021-01-15T19:03:58.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2021-01-15T19:03:58.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:58.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:58.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:58.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-6 2021-01-15T19:03:58.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:03:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:58.553 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:58.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-6 2021-01-15T19:03:58.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:58.554 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-6 2021-01-15T19:03:58.554 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-6 2021-01-15T19:03:58.555 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:58.958 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:58.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:58.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:03:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:58.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:58.960 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:03:58.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:58.962 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:03:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:03:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:58.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:58.963 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:03:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:58.965 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:58.965 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:58.965 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:58.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:03:59.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:59.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:03:59.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:03:59.069 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:03:59.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:03:59.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:03:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:59.070 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:59.071 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:03:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:03:59.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2021-01-15T19:03:59.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:03:59.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:59.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:59.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:03:59.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:03:59.180 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:59.180 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:59.181 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:59.181 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:03:59.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:03:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2021-01-15T19:03:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:03:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2021-01-15T19:03:59.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:03:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:03:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:03:59.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:59.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:59.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:03:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:03:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2021-01-15T19:03:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:03:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:03:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:03:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:03:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:03:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:03:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-7 2021-01-15T19:03:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:03:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:03:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:03:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:03:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:03:59.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:03:59.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-7 2021-01-15T19:03:59.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:03:59.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-7 2021-01-15T19:03:59.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-7 2021-01-15T19:03:59.520 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:03:59.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:03:59.906 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:03:59.906 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:03:59.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:03:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:03:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:03:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:03:59.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:03:59.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:03:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:03:59.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:03:59.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:03:59.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:03:59.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:03:59.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:03:59.911 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:03:59.911 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:03:59.911 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:03:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:03:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:03:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:03:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:03:59.912 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:03:59.913 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:00.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:00.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:00.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:00.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:04:00.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:00.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:00.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:00.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:00.018 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:00.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:00.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:00.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2021-01-15T19:04:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:00.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:00.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-7 2021-01-15T19:04:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:00.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:04:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:00.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-7 2021-01-15T19:04:00.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:00.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-7 2021-01-15T19:04:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-7 2021-01-15T19:04:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:00.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:00.523 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:00.523 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:00.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:00.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:00.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:04:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:00.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:00.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:04:00.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:00.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:04:00.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:00.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:04:00.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:00.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:00.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:00.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:04:00.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:00.530 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:00.531 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:00.532 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:00.532 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:00.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:00.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:00.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:04:00.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:00.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:00.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:00.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:00.631 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:00.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:00.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:00.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2021-01-15T19:04:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:04:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:00.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:04:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:04:00.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:00.743 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:00.744 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:00.744 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:00.745 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:00.745 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:04:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2021-01-15T19:04:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:00.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:04:00.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2021-01-15T19:04:00.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:04:00.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:04:00.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:04:00.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:01.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:01.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:04:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2021-01-15T19:04:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:01.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-8 2021-01-15T19:04:01.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:04:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:01.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-8 2021-01-15T19:04:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:01.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-8 2021-01-15T19:04:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-8 2021-01-15T19:04:01.087 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:01.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:01.507 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:01.507 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:01.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:01.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:01.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:04:01.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:01.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:01.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:04:01.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:01.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:04:01.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:01.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:01.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:01.511 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:01.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:01.512 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:01.512 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:01.513 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:01.513 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:01.514 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:01.515 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:01.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:01.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:01.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:01.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:01.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:04:01.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:01.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:01.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:01.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:01.619 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:01.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:01.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:01.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2021-01-15T19:04:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:01.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:01.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-8 2021-01-15T19:04:01.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:01.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:01.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:01.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:04:01.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:01.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:01.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-8 2021-01-15T19:04:01.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:01.732 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-8 2021-01-15T19:04:01.732 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-8 2021-01-15T19:04:01.733 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:02.115 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:02.115 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:02.116 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:04:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:02.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:02.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:04:02.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:02.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:04:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:04:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:02.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:02.119 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:02.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:04:02.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:02.120 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:02.120 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:02.120 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:02.121 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:02.121 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:02.121 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:02.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:02.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:02.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:02.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:04:02.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:02.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:02.228 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:02.229 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:02.229 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:02.230 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:02.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:02.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2021-01-15T19:04:02.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:04:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:02.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:04:02.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:04:02.339 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:02.340 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:02.341 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:02.342 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:02.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:04:02.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:02.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2021-01-15T19:04:02.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:02.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:04:02.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2021-01-15T19:04:02.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:04:02.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:04:02.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:04:02.447 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:02.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:02.691 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:04:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2021-01-15T19:04:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:02.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:02.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-9 2021-01-15T19:04:02.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:02.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:02.695 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:04:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:02.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:02.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-9 2021-01-15T19:04:02.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:02.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-9 2021-01-15T19:04:02.697 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-9 2021-01-15T19:04:02.697 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:03.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:03.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:04:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:03.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:04:03.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:03.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:04:03.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:03.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:03.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:03.102 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:03.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:03.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:03.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:03.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:03.203 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:04:03.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:03.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:03.204 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:03.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:03.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2021-01-15T19:04:03.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:03.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:03.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-9 2021-01-15T19:04:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:03.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:04:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:03.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:03.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-9 2021-01-15T19:04:03.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:03.317 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-9 2021-01-15T19:04:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-9 2021-01-15T19:04:03.317 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:03.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:03.703 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:03.703 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:03.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:03.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:03.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:04:03.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:03.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:03.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:04:03.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:03.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:04:03.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:03.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:04:03.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:03.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:03.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:03.707 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:04:03.708 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:03.708 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:03.708 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:03.708 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:03.708 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:03.709 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:03.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:03.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:03.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:04:03.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:03.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:03.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:03.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:03.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2021-01-15T19:04:03.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:04:03.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:03.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:03.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:04:03.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:04:03.926 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:03.926 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:03.927 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:03.927 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:03.927 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:03.927 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:03.928 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:03.928 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:03.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:04:04.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:04.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2021-01-15T19:04:04.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:04.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:04:04.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2021-01-15T19:04:04.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:04:04.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:04:04.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:04:04.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:04.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:04.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:04:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2021-01-15T19:04:04.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:04.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:04.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:04.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:04.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-10 2021-01-15T19:04:04.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:04.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:04.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:04.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:04:04.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:04.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:04.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-10 2021-01-15T19:04:04.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:04.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-10 2021-01-15T19:04:04.272 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-10 2021-01-15T19:04:04.272 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:04.685 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:04.686 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:04.686 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:04.686 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:04.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:04:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:04.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:04.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:04:04.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:04.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:04:04.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:04.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:04.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:04.690 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:04.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:04.691 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:04.691 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:04.691 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:04.691 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:04.691 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:04.692 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:04.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:04.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:04.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:04.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:04.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:04.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:04:04.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:04.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:04.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:04.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:04.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:04.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:04.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:04.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2021-01-15T19:04:04.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:04.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:04.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:04.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:04.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:04.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:04.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:04.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-10 2021-01-15T19:04:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:04.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:04:04.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:04.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:04.915 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-10 2021-01-15T19:04:04.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:04.916 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-10 2021-01-15T19:04:04.916 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-10 2021-01-15T19:04:04.917 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:05.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:05.299 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:05.299 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:05.300 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:05.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:05.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:04:05.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:05.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:05.301 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:04:05.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:05.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:04:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:04:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:05.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:05.305 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:05.305 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:04:05.305 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:05.306 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:05.306 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:05.307 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:05.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:05.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:05.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:05.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:05.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:04:05.411 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:05.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:05.412 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:05.412 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:05.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:05.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2021-01-15T19:04:05.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:04:05.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:05.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:05.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:04:05.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:04:05.521 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:05.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:05.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:05.522 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:05.522 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:05.523 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:05.523 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:05.523 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:05.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:04:05.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:05.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2021-01-15T19:04:05.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:05.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:04:05.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2021-01-15T19:04:05.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:04:05.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:04:05.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:04:05.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:05.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:05.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:05.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2021-01-15T19:04:05.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2021-01-15T19:04:05.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:05.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:05.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-11 2021-01-15T19:04:05.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:05.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:05.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:04:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:05.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:05.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-11 2021-01-15T19:04:05.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:05.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-11 2021-01-15T19:04:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-11 2021-01-15T19:04:05.906 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:06.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:06.306 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:06.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=3 2021-01-15T19:04:06.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:06.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:06.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2021-01-15T19:04:06.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:06.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2021-01-15T19:04:06.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:06.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:06.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:06.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:06.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:06.311 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:06.311 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:06.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:06.312 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:06.312 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:06.312 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:06.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:06.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:06.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:06.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:06.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:06.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.3 2021-01-15T19:04:06.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:06.419 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:06.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:06.419 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:06.420 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:06.420 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:06.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:06.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2021-01-15T19:04:06.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:04:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:04:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:04:06.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:04:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=pool-rep 2021-01-15T19:04:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:04:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj-warn-11 2021-01-15T19:04:06.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:04:06.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:06.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:04:06.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:04:06.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:04:06.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:04:06.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds pool-rep obj-warn-11 2021-01-15T19:04:06.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=pool-rep 2021-01-15T19:04:06.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj-warn-11 2021-01-15T19:04:06.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map pool-rep obj-warn-11 2021-01-15T19:04:06.535 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:04:06.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=3 2021-01-15T19:04:06.947 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:06.947 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:04:06.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 3 1 0 2021-01-15T19:04:06.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:04:06.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:04:06.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:04:06.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:04:06.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:04:06.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:04:06.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:04:06.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:06.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:04:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:04:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:04:06.951 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:04:06.951 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:06.952 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:06.952 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:06.952 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:06.952 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:06.952 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:06.953 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:04:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:07.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:04:07.058 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:04:07.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:04:07.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:04:07.058 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:04:07.059 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:07.059 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:07.059 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:07.059 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:04:07.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:04:07.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2021-01-15T19:04:07.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2021-01-15T19:04:07.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:04:07.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=3 2021-01-15T19:04:07.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=osd_debug_feed_pullee 2021-01-15T19:04:07.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=1 2021-01-15T19:04:07.169 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:04:07.169 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.3 2021-01-15T19:04:07.169 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2021-01-15T19:04:07.169 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2021-01-15T19:04:07.170 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:07.171 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.3.asok 2021-01-15T19:04:07.171 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2021-01-15T19:04:07.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:04:07.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2021-01-15T19:04:07.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2021-01-15T19:04:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2021-01-15T19:04:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2021-01-15T19:04:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2021-01-15T19:04:07.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2021-01-15T19:04:07.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:07.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:07.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2021-01-15T19:04:07.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2021-01-15T19:04:07.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:04:07.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:04:07.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:04:07.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:04:07.510 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:04:07.510 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:04:07.511 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:04:07.511 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:04:07.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:04:07.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:04:07.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:04:07.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:04:07.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:04:07.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:04:07.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:04:07.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:04:07.986 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:07.986 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:04:07.987 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:04:07.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:04:07.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:07.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:04:08.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803802 2021-01-15T19:04:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803802 2021-01-15T19:04:08.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803802' 2021-01-15T19:04:08.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:08.145 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:04:08.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574873 2021-01-15T19:04:08.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574873 2021-01-15T19:04:08.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803802 1-55834574873' 2021-01-15T19:04:08.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:08.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:04:08.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378647 2021-01-15T19:04:08.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378647 2021-01-15T19:04:08.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803802 1-55834574873 2-81604378647' 2021-01-15T19:04:08.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:08.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:04:08.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149718 2021-01-15T19:04:08.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149718 2021-01-15T19:04:08.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803802 1-55834574873 2-81604378647 3-111669149718' 2021-01-15T19:04:08.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:08.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803802 2021-01-15T19:04:08.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:08.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:04:08.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803802 2021-01-15T19:04:08.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:08.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803802 2021-01-15T19:04:08.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2021-01-15T19:04:08.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:08.592 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803802 2021-01-15T19:04:08.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803802 -lt 25769803802 2021-01-15T19:04:08.976 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:08.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574873 2021-01-15T19:04:08.976 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:08.977 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:04:08.977 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574873 2021-01-15T19:04:08.978 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:08.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574873 2021-01-15T19:04:08.978 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2021-01-15T19:04:08.979 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:04:08.979 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574873 2021-01-15T19:04:09.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574873 -lt 55834574873 2021-01-15T19:04:09.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:09.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378647 2021-01-15T19:04:09.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:09.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:04:09.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378647 2021-01-15T19:04:09.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378647 2021-01-15T19:04:09.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378647' 2021-01-15T19:04:09.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:04:09.387 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378647 2021-01-15T19:04:09.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378645 -lt 81604378647 2021-01-15T19:04:09.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:04:10.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:04:10.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:04:11.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378647 -lt 81604378647 2021-01-15T19:04:11.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:11.187 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149718 2021-01-15T19:04:11.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:11.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:04:11.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149718 2021-01-15T19:04:11.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:11.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149718 2021-01-15T19:04:11.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149718' 2021-01-15T19:04:11.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:04:11.190 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149718 2021-01-15T19:04:11.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149719 -lt 111669149718 2021-01-15T19:04:11.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:04:11.591 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:11.592 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:12.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:04:12.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:04:12.117 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:04:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:04:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:04:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:04:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:04:12.118 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:04:12.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:04:12.493 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:04:12.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:12.494 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:13.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:04:13.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:04:13.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:04:13.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2021-01-15T19:04:13.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2021-01-15T19:04:13.146 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2021-01-15T19:04:13.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2021-01-15T19:04:13.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2021-01-15T19:04:13.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2021-01-15T19:04:13.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:04:13.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:04:13.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:04:13.533 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:13.533 INFO:tasks.workunit.client.0.smithi145.stderr:2 2021-01-15T19:04:13.533 INFO:tasks.workunit.client.0.smithi145.stderr:3' 2021-01-15T19:04:13.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:04:13.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:13.534 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:04:13.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803805 2021-01-15T19:04:13.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803805 2021-01-15T19:04:13.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805' 2021-01-15T19:04:13.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:13.683 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:04:13.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574876 2021-01-15T19:04:13.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574876 2021-01-15T19:04:13.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-55834574876' 2021-01-15T19:04:13.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:13.816 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:04:13.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378650 2021-01-15T19:04:13.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378650 2021-01-15T19:04:13.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-55834574876 2-81604378650' 2021-01-15T19:04:13.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:13.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-01-15T19:04:14.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=111669149721 2021-01-15T19:04:14.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 111669149721 2021-01-15T19:04:14.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803805 1-55834574876 2-81604378650 3-111669149721' 2021-01-15T19:04:14.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:14.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803805 2021-01-15T19:04:14.083 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:14.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:04:14.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803805 2021-01-15T19:04:14.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:14.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803805 2021-01-15T19:04:14.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803805' 2021-01-15T19:04:14.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:14.086 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803805 2021-01-15T19:04:14.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803806 -lt 25769803805 2021-01-15T19:04:14.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:14.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574876 2021-01-15T19:04:14.472 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:14.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:04:14.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574876 2021-01-15T19:04:14.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:14.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574876 2021-01-15T19:04:14.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2021-01-15T19:04:14.475 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:04:14.475 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574876 2021-01-15T19:04:14.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574876 -lt 55834574876 2021-01-15T19:04:14.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:14.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378650 2021-01-15T19:04:14.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:14.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:04:14.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378650 2021-01-15T19:04:14.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:14.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378650 2021-01-15T19:04:14.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378650' 2021-01-15T19:04:14.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:04:14.898 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378650 2021-01-15T19:04:15.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378650 -lt 81604378650 2021-01-15T19:04:15.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:15.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-111669149721 2021-01-15T19:04:15.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:15.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=3 2021-01-15T19:04:15.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 3-111669149721 2021-01-15T19:04:15.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:15.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=111669149721 2021-01-15T19:04:15.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.3 seq 111669149721' 2021-01-15T19:04:15.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 3 2021-01-15T19:04:15.287 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.3 seq 111669149721 2021-01-15T19:04:15.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 111669149721 -lt 111669149721 2021-01-15T19:04:15.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2021-01-15T19:04:15.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2021-01-15T19:04:16.057 INFO:tasks.workunit.client.0.smithi145.stderr:dumped all 2021-01-15T19:04:16.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2021-01-15T19:04:16.068 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2021-01-15T19:04:16.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2021-01-15T19:04:16.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2021-01-15T19:04:16.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2021-01-15T19:04:16.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2021-01-15T19:04:16.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2021-01-15T19:04:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2021-01-15T19:04:17.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2021-01-15T19:04:17.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2021-01-15T19:04:17.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2021-01-15T19:04:17.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2021-01-15T19:04:17.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2021-01-15T19:04:18.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2021-01-15T19:04:18.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2021-01-15T19:04:18.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2021-01-15T19:04:18.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=pool-rep 2021-01-15T19:04:18.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2021-01-15T19:04:19.124 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'pool-rep' does not exist 2021-01-15T19:04:19.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:04:19.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:04:19.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:04:19.135 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:04:19.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:04:19.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:04:19.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:04:19.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:04:19.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:04:19.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:04:19.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:04:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:04:19.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:04:19.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:04:19.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:04:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:04:19.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:04:19.247 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:04:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:04:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:04:19.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:04:19.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.258 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:04:19.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:04:19.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:04:19.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2021-01-15T19:04:19.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:04:19.262 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:04:19.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:04:19.262 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:04:19.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:04:19.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:04:19.263 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:04:19.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:04:19.264 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:04:19.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:04:19.265 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:04:19.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:04:19.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:04:19.266 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:04:19.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:04:19.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:04:19.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:04:19.267 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:04:19.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:04:19.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:04:19.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:04:19.269 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:04:19.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:04:19.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:04:19.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:04:19.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.937479 2021-01-15T19:04:19.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:04:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T19:04:19.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:04:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2021-01-15T19:04:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2021-01-15T19:04:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:04:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:04:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:04:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2021-01-15T19:04:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2021-01-15T19:04:19.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:04:19.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:04:19.567 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:04:19.567 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:04:19.568 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.568 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.568 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:19.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon_osd_warn_num_repaired=10 2021-01-15T19:04:19.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:04:19.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:04:19.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:04:19.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:04:19.753 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:04:19.754 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:04:19.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:04:19.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:04:19.754 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:04:19.754 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:19.755 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.755 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.755 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:04:19.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:04:19.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get fsid 2021-01-15T19:04:19.853 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:04:19.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:04:19.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:04:19.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:04:19.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:04:19.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:04:19.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:04:19.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:04:19.855 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:04:19.855 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:19.856 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:19.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:04:19.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:04:19.856 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get mon_host 2021-01-15T19:04:19.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2021-01-15T19:04:19.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:04:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:04:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:04:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2021-01-15T19:04:19.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:04:20.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:04:20.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:04:20.453 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:04:20.453 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:04:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:20.454 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:20.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:20.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:04:20.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:04:20.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2021-01-15T19:04:20.796 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:04:20.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2021-01-15T19:04:20.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:20.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2021-01-15T19:04:20.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2021-01-15T19:04:20.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2021-01-15T19:04:20.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2021-01-15T19:04:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:04:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2021-01-15T19:04:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2021-01-15T19:04:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2021-01-15T19:04:20.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2021-01-15T19:04:20.809 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2021-01-15T19:04:20.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2021-01-15T19:04:20.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:04:20.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2021-01-15T19:04:20.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:20.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:04:20.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:04:20.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:04:20.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:04:20.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:04:20.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:04:20.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:04:20.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:04:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:04:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:04:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:04:20.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:04:20.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:04:20.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:04:20.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:04:20.817 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:04:20.817 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:04:20.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:20.818 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:20.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:20.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:20.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:04:20.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:04:20.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:04:20.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:04:20.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:04:20.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:04:20.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:04:20.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:20.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:04:20.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:04:20.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:04:20.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:04:20.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=84a4b6fb-fc11-4288-a7fa-d30a5722d24a 2021-01-15T19:04:20.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 84a4b6fb-fc11-4288-a7fa-d30a5722d24a' 2021-01-15T19:04:20.821 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:04:20.822 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 84a4b6fb-fc11-4288-a7fa-d30a5722d24a 2021-01-15T19:04:20.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA05wFgus89MRAAaju5yrbhSvE+vOLHfrOk6w== 2021-01-15T19:04:20.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA05wFgus89MRAAaju5yrbhSvE+vOLHfrOk6w=="}' 2021-01-15T19:04:20.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 84a4b6fb-fc11-4288-a7fa-d30a5722d24a -i td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:04:21.195 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:04:21.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:04:21.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA05wFgus89MRAAaju5yrbhSvE+vOLHfrOk6w== --osd-uuid 84a4b6fb-fc11-4288-a7fa-d30a5722d24a 2021-01-15T19:04:21.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.231+0000 7f9fe2ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:21.234 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.233+0000 7f9fe2ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:21.235 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.234+0000 7f9fe2ed5080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.235+0000 7f9fe2ed5080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:04:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.235+0000 7f9fe2ed5080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:04:21.236 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.235+0000 7f9fe2ed5080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:04:21.276 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:21.275+0000 7f9fe2ed5080 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2021-01-15T19:04:25.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2021-01-15T19:04:25.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:04:25.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:04:25.961 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:04:25.961 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:04:26.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:04:26.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:26.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:04:26.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:04:26.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:04:26.490 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:04:26.512 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:26.511+0000 7f183cc5f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:26.513 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:26.512+0000 7f183cc5f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:26.515 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:26.514+0000 7f183cc5f080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:26.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:04:26.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:04:26.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:04:26.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:04:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:04:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:04:26.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:26.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:04:26.888 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:04:27.035 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:27.033+0000 7f183cc5f080 -1 Falling back to public interface 2021-01-15T19:04:27.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:04:27.834 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:27.833+0000 7f183cc5f080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:04:28.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:04:28.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:28.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:04:28.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:28.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:04:28.291 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:04:28.696 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/959384,v1:127.0.0.1:6803/959384] [v2:127.0.0.1:6804/959384,v1:127.0.0.1:6805/959384] exists,up 84a4b6fb-fc11-4288-a7fa-d30a5722d24a 2021-01-15T19:04:28.697 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 7f939947-0494-4bf0-b0a4-be136e75852a 2021-01-15T19:04:28.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:04:28.697 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:04:28.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:04:28.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:04:28.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2021-01-15T19:04:28.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:28.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:04:28.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:04:28.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:04:28.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:04:28.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:04:28.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:04:28.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:04:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:04:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:04:28.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:04:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:04:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:04:28.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:04:28.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:04:28.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:04:28.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:04:28.703 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:04:28.704 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:28.704 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:28.704 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:28.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:28.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:04:28.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:04:28.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:04:28.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:04:28.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:04:28.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:04:28.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:04:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:04:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:04:28.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:04:28.708 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:04:28.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7f939947-0494-4bf0-b0a4-be136e75852a 2021-01-15T19:04:28.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 7f939947-0494-4bf0-b0a4-be136e75852a' 2021-01-15T19:04:28.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:04:28.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQA85wFgvmtXKhAAq9LvLHpkCaIG2vcTiRJeVw== 2021-01-15T19:04:28.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQA85wFgvmtXKhAAq9LvLHpkCaIG2vcTiRJeVw=="}' 2021-01-15T19:04:28.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7f939947-0494-4bf0-b0a4-be136e75852a -i td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:04:29.509 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:04:29.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:04:29.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA85wFgvmtXKhAAq9LvLHpkCaIG2vcTiRJeVw== --osd-uuid 7f939947-0494-4bf0-b0a4-be136e75852a 2021-01-15T19:04:29.546 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.545+0000 7fb9b54b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:29.567 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.566+0000 7fb9b54b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:29.568 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.567+0000 7fb9b54b2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:29.569 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.568+0000 7fb9b54b2080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:04:29.569 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.568+0000 7fb9b54b2080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:04:29.569 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.568+0000 7fb9b54b2080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:04:29.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:29.632+0000 7fb9b54b2080 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2021-01-15T19:04:34.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2021-01-15T19:04:34.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:04:34.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:04:34.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:04:34.125 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:04:34.655 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:04:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:04:34.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:04:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:04:34.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:04:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:34.676+0000 7f7288b7a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:34.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:34.697+0000 7f7288b7a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:34.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:34.698+0000 7f7288b7a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:04:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:04:35.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:04:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:04:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:04:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:04:35.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:35.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:04:35.063 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:04:35.216 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:35.214+0000 7f7288b7a080 -1 Falling back to public interface 2021-01-15T19:04:35.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:04:36.008 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:36.007+0000 7f7288b7a080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:04:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:04:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:04:36.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:36.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:04:36.456 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:04:36.862 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/960656,v1:127.0.0.1:6811/960656] [v2:127.0.0.1:6812/960656,v1:127.0.0.1:6813/960656] exists,up 7f939947-0494-4bf0-b0a4-be136e75852a 2021-01-15T19:04:36.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:04:36.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:04:36.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:04:36.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:04:36.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2021-01-15T19:04:36.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:36.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:04:36.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T19:04:36.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:04:36.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:04:36.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:04:36.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:04:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:04:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:04:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T19:04:36.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T19:04:36.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:04:36.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:04:36.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:04:36.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:04:36.867 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:04:36.868 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:04:36.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:04:36.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:04:36.868 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:04:36.869 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:36.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:04:36.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:04:36.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:04:36.869 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:04:36.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:04:36.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:04:36.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:04:36.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:04:36.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:36.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:04:36.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:04:36.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T19:04:36.871 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:04:36.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=efa329e3-e5a5-40bd-92db-1a11f2a39023 2021-01-15T19:04:36.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 efa329e3-e5a5-40bd-92db-1a11f2a39023' 2021-01-15T19:04:36.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:04:36.873 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 efa329e3-e5a5-40bd-92db-1a11f2a39023 2021-01-15T19:04:36.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQBE5wFg98RgNBAAdoc4RXbhrIqHokmya2HEWw== 2021-01-15T19:04:36.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQBE5wFg98RgNBAAdoc4RXbhrIqHokmya2HEWw=="}' 2021-01-15T19:04:36.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new efa329e3-e5a5-40bd-92db-1a11f2a39023 -i td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:04:37.638 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:04:37.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:04:37.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBE5wFg98RgNBAAdoc4RXbhrIqHokmya2HEWw== --osd-uuid efa329e3-e5a5-40bd-92db-1a11f2a39023 2021-01-15T19:04:37.674 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.673+0000 7fe549310080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:37.676 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.675+0000 7fe549310080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:37.678 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.677+0000 7fe549310080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.678+0000 7fe549310080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:04:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.678+0000 7fe549310080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:04:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.678+0000 7fe549310080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:04:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:37.782+0000 7fe549310080 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2021-01-15T19:04:42.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2021-01-15T19:04:42.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:04:42.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T19:04:42.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:04:42.769 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T19:04:43.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T19:04:43.292 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:04:43.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:04:43.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:04:43.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:04:43.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:04:43.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:43.314+0000 7fb37f599080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:43.335 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:43.333+0000 7fb37f599080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:43.336 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:43.335+0000 7fb37f599080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:04:43.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T19:04:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:04:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:04:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:04:43.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:04:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:04:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:43.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:04:43.689 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:04:43.851 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:43.850+0000 7fb37f599080 -1 Falling back to public interface 2021-01-15T19:04:44.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:04:44.669 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:04:44.667+0000 7fb37f599080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T19:04:45.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:04:45.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:04:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:45.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:04:45.096 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:04:45.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:04:46.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:04:46.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:04:46.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T19:04:46.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:04:46.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:04:46.516 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:04:46.918 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/961920,v1:127.0.0.1:6819/961920] [v2:127.0.0.1:6820/961920,v1:127.0.0.1:6821/961920] exists,up efa329e3-e5a5-40bd-92db-1a11f2a39023 2021-01-15T19:04:46.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:04:46.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:04:46.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:04:46.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2021-01-15T19:04:46.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:04:46.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:04:46.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:04:46.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:04:46.921 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:04:46.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:04:46.921 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:04:46.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:04:46.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:04:46.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:04:47.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:04:47.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:04:47.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:04:47.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:04:47.011 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:04:47.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:04:47.409 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:47.410 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:04:47.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:04:47.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:47.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:04:47.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803782 2021-01-15T19:04:47.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803782 2021-01-15T19:04:47.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782' 2021-01-15T19:04:47.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:47.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:04:47.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574853 2021-01-15T19:04:47.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574853 2021-01-15T19:04:47.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782 1-55834574853' 2021-01-15T19:04:47.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:47.688 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:04:47.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345923 2021-01-15T19:04:47.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345923 2021-01-15T19:04:47.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782 1-55834574853 2-85899345923' 2021-01-15T19:04:47.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:47.859 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803782 2021-01-15T19:04:47.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:47.860 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:04:47.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803782 2021-01-15T19:04:47.860 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:47.861 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803782 2021-01-15T19:04:47.862 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2021-01-15T19:04:47.862 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:47.862 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803782 2021-01-15T19:04:48.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803780 -lt 25769803782 2021-01-15T19:04:48.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:04:49.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:04:49.256 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:49.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803782 -lt 25769803782 2021-01-15T19:04:49.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:49.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574853 2021-01-15T19:04:49.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:49.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:04:49.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574853 2021-01-15T19:04:49.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:49.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574853 2021-01-15T19:04:49.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2021-01-15T19:04:49.653 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:04:49.653 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574853 2021-01-15T19:04:50.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574853 -lt 55834574853 2021-01-15T19:04:50.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:50.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-85899345923 2021-01-15T19:04:50.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:50.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:04:50.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-85899345923 2021-01-15T19:04:50.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345923 2021-01-15T19:04:50.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 85899345923' 2021-01-15T19:04:50.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:04:50.103 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 85899345923 2021-01-15T19:04:50.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345923 -lt 85899345923 2021-01-15T19:04:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:04:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:50.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:51.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T19:04:51.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:04:51.038 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:04:51.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:04:51.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:04:51.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:04:51.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:04:51.039 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:04:51.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=6 2021-01-15T19:04:51.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:04:51.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:51.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 6 = 8 2021-01-15T19:04:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 6 '!=' -1 2021-01-15T19:04:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T19:04:51.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=6 2021-01-15T19:04:51.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:04:51.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:04:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:04:52.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:04:52.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:04:52.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:04:52.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:04:52.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:04:52.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:04:52.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:04:52.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:04:52.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:04:52.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:52.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:52.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T19:04:52.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:04:52.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:04:52.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2021-01-15T19:04:52.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2021-01-15T19:04:52.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test-pool 1 1 2021-01-15T19:04:53.471 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test-pool' already exists 2021-01-15T19:04:53.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T19:04:54.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2021-01-15T19:04:54.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:04:54.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:04:54.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:04:54.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:04:54.483 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:04:54.483 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:04:54.484 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:04:54.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:04:54.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:04:54.484 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:04:54.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:04:54.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:04:54.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:04:54.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:04:54.587 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:04:54.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:04:54.986 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:04:54.986 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:04:54.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:04:54.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:54.987 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:04:55.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803786 2021-01-15T19:04:55.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803786 2021-01-15T19:04:55.139 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786' 2021-01-15T19:04:55.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:55.140 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:04:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574856 2021-01-15T19:04:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574856 2021-01-15T19:04:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-55834574856' 2021-01-15T19:04:55.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:04:55.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:04:55.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345926 2021-01-15T19:04:55.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345926 2021-01-15T19:04:55.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803786 1-55834574856 2-85899345926' 2021-01-15T19:04:55.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:55.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803786 2021-01-15T19:04:55.414 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:55.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:04:55.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803786 2021-01-15T19:04:55.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:55.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803786 2021-01-15T19:04:55.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803786' 2021-01-15T19:04:55.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:55.418 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803786 2021-01-15T19:04:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803786 2021-01-15T19:04:55.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:04:56.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:04:56.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:04:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803786 -lt 25769803786 2021-01-15T19:04:57.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:57.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574856 2021-01-15T19:04:57.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:04:57.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574856 2021-01-15T19:04:57.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:57.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574856 2021-01-15T19:04:57.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574856' 2021-01-15T19:04:57.198 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:04:57.198 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574856 2021-01-15T19:04:57.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574856 -lt 55834574856 2021-01-15T19:04:57.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:04:57.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-85899345926 2021-01-15T19:04:57.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:04:57.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:04:57.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-85899345926 2021-01-15T19:04:57.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:04:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345926 2021-01-15T19:04:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 85899345926' 2021-01-15T19:04:57.619 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:04:57.619 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 85899345926 2021-01-15T19:04:58.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345927 -lt 85899345926 2021-01-15T19:04:58.033 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:04:58.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:58.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:04:58.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:04:58.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:04:58.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:04:58.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:04:58.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:04:58.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:04:58.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:04:58.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:04:58.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:04:58.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:04:58.966 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:04:59.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:04:59.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:04:59.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:04:59.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2021-01-15T19:04:59.842 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:04:59.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499812+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.842 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499097+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.843 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.950863+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.843 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.498987+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.843 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.950777+0000 0'0 25:36 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.843 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:54.243657+0000 0'0 25:10 [2,1,0] 2 [2,1,0] 2 0'0 2021-01-15T19:04:53.266959+0000 0'0 2021-01-15T19:04:53.266959+0000 0 2021-01-15T19:04:59.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499920+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.844 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499755+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.845 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.850078+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:04:59.845 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:04:59.846 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:04:59.846 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:04:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2021-01-15T19:04:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:04:59.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2021-01-15T19:04:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2021-01-15T19:04:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:04:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:04:59.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:04:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:04:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:04:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:04:59.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:04:59.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:04:59.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:00.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:05:00.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2021-01-15T19:05:00.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:05:00.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2021-01-15T19:05:00.003 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool myobject 2021-01-15T19:05:00.003 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:05:00.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:05:00.404 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:05:00.404 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:05:00.405 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 0 2021-01-15T19:05:00.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2021-01-15T19:05:00.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2021-01-15T19:05:00.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2021-01-15T19:05:00.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:05:00.406 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:05:00.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:05:00.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:05:00.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:05:00.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:05:00.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2021-01-15T19:05:01.088 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T19:05:01.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2021-01-15T19:05:01.683 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T19:05:01.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2021-01-15T19:05:02.058 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:05:02.059 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499812+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.059 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499097+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.060 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 peering 2021-01-15T19:05:00.693808+0000 0'0 27:16 [2,1] 2 [2,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.060 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.498987+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.061 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2021-01-15T19:04:47.950777+0000 0'0 25:36 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.061 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 1 0 0 0 0 4096 0 0 1 1 peering 2021-01-15T19:05:00.692974+0000 26'1 27:13 [2,1] 2 [2,1] 2 0'0 2021-01-15T19:04:53.266959+0000 0'0 2021-01-15T19:04:53.266959+0000 0 2021-01-15T19:05:02.061 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499920+0000 0'0 25:26 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.062 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:04:47.499755+0000 0'0 25:26 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.062 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 peering 2021-01-15T19:05:00.694240+0000 0'0 27:16 [2,1] 2 [2,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:05:02.063 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:05:02.063 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:05:02.064 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:05:02.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2021-01-15T19:05:02.071 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T19:05:02.071 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T19:05:02.072 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000118132 s, 34.7 MB/s 2021-01-15T19:05:02.072 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2021-01-15T19:05:02.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:02.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:02.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.531 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:03.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:03.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.610 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.611 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.745 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:04.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.013 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.423 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:05.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:05.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.218 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:06.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:06.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.138 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:07.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:07.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:08.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:08.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:09.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:09.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.612 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.613 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:10.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:10.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.398 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:11.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:11.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:12.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:12.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.324 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.457 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:13.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:13.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:14.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:14.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:15.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:15.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.396 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.529 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:16.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:16.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.441 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.442 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.568 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:17.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:17.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.622 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.747 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.748 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:18.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:18.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.668 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:19.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:19.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.070 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:20.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.134 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:21.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:21.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:22.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:23.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:23.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:24.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:24.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:25.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:25.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.749 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:26.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:26.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.669 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:27.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:27.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:28.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:28.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.644 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:29.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:29.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.038 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:30.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:30.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.260 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.771 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:31.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:31.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.147 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.834 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:32.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:32.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.420 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:33.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:33.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.081 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:34.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:34.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:35.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:35.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:36.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:36.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:37.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:37.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.424 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.692 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.693 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.835 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:38.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:38.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:39.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:39.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.057 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.450 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.744 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:40.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:40.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.530 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:41.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:41.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:42.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:42.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:42.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:42.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:05:42.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:05:42.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2021-01-15T19:05:42.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:05:42.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:05:42.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:05:42.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:05:42.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test-pool 2021-01-15T19:05:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:05:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj250 2021-01-15T19:05:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:05:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:05:42.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:05:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:05:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:05:42.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:05:42.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test-pool obj250 2021-01-15T19:05:42.758 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:05:42.759 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj250 2021-01-15T19:05:42.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool obj250 2021-01-15T19:05:42.759 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:05:43.147 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:05:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T19:05:43.148 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 2021-01-15T19:05:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:05:43.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=2 2021-01-15T19:05:43.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:05:43.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:05:43.149 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2021-01-15T19:05:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:05:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2021-01-15T19:05:43.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:05:43.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=2 2021-01-15T19:05:43.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:05:43.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:05:43.152 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:05:43.152 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.2 2021-01-15T19:05:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T19:05:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T19:05:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:05:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:05:43.153 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:05:43.154 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.2.asok 2021-01-15T19:05:43.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:05:43.256 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:05:43.257 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:05:43.258 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:05:43.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.2 2021-01-15T19:05:43.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T19:05:43.259 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T19:05:43.259 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:05:43.260 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:05:43.260 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:05:43.260 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.2.asok 2021-01-15T19:05:43.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:05:43.261 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.2.asok injectdataerr test-pool obj250 2021-01-15T19:05:43.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2021-01-15T19:05:43.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:05:43.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:05:43.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:05:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:05:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test-pool 2021-01-15T19:05:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:05:43.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj250 2021-01-15T19:05:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:05:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:05:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:05:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:05:43.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:05:43.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:05:43.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test-pool obj250 2021-01-15T19:05:43.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:05:43.371 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj250 2021-01-15T19:05:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool obj250 2021-01-15T19:05:43.372 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:05:43.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:05:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T19:05:43.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 2021-01-15T19:05:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:05:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:05:43.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:05:43.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:05:43.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:05:43.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:05:43.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:05:43.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:05:43.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:05:43.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:05:43.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:05:43.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:05:43.770 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:05:43.770 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:05:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:05:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:05:43.771 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:05:43.772 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:05:43.772 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:05:43.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:05:43.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:05:43.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:05:43.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:05:43.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:05:43.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:05:43.872 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:05:43.872 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:05:43.872 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:05:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:05:43.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:05:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:05:43.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr test-pool obj250 2021-01-15T19:05:43.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2021-01-15T19:05:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:05:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:05:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T19:05:43.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:05:43.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:05:43.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:05:43.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:05:43.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:05:43.986 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:05:43.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:05:43.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:05:43.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:05:43.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:05:43.987 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:05:43.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:05:43.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:05:43.988 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:05:43.988 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:05:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:05:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:05:43.989 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:05:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:05:43.989 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:05:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:05:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:05:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:05:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:05:43.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:05:43.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:05:43.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:05:43.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:05:43.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T19:05:43.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:05:43.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2021-01-15T19:05:43.992 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:05:43.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T19:05:43.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:05:43.994 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:05:43.995 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:05:44.007 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:05:44.005+0000 7fdd64c7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:05:44.009 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:05:44.007+0000 7fdd64c7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:05:44.010 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:05:44.008+0000 7fdd64c7b080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:05:44.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T19:05:44.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:05:44.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:05:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:05:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:05:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:05:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:05:44.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:05:44.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:05:44.408 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:05:44.529 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:05:44.527+0000 7fdd64c7b080 -1 Falling back to public interface 2021-01-15T19:05:44.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:05:45.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:05:45.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:05:45.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:05:45.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:05:45.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:05:45.806 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:05:45.885 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:05:45.884+0000 7fdd64c7b080 -1 osd.0 26 log_to_monitors {default=true} 2021-01-15T19:05:46.214 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 30 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/969544,v1:127.0.0.1:6803/969544] [v2:127.0.0.1:6804/969544,v1:127.0.0.1:6805/969544] exists,up 84a4b6fb-fc11-4288-a7fa-d30a5722d24a 2021-01-15T19:05:46.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:05:46.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:05:46.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:05:46.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2021-01-15T19:05:47.371 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T19:05:47.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2021-01-15T19:06:02.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 100 2021-01-15T19:06:02.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:02.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:02.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:02.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:02.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:02.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:02.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:02.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:02.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:02.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:02.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:02.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:02.805 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:03.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:03.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:03.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:03.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:03.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:03.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:04.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:04.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:04.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:04.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:04.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:04.195 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:05.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:05.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:05.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:05.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:05.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:05.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:05.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:05.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:05.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:05.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:05.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:05.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:05.579 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:06.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:06.580 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:06.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:06.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:06.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:06.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:06.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:06.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:06.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:06.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:06.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:06.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:06.967 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:07.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:07.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:07.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:07.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:07.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:07.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:08.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:08.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:08.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:08.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:08.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:08.362 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:09.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:09.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:09.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:09.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:09.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:09.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:09.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:09.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:09.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:09.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:09.761 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:09.762 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:10.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:10.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:10.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:10.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:10.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:10.762 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:11.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:11.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:11.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:11.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:11.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:11.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:11.161 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:12.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:12.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:12.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:12.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:12.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:12.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:12.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:12.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:12.554 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:12.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:12.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:12.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:12.556 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:13.556 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:13.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:13.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:13.557 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:13.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:13.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:13.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:13.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:13.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:13.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:13.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:13.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:13.947 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:14.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:14.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:14.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:14.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:14.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:14.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:15.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:15.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:15.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:15.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:15.340 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:16.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:16.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:16.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:16.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:16.342 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:16.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:16.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:16.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:16.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:16.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:16.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:16.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:16.738 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:17.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:17.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:17.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:17.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:17.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:17.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:18.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:18.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:18.128 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:18.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:18.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:18.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:18.130 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:19.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:19.130 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:19.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:19.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:19.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:19.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:19.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:19.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:19.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:19.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:19.529 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:20.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:20.529 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:20.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:20.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:20.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:20.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:20.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:20.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:20.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:20.926 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:21.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:21.927 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:21.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:21.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:22.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:22.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:22.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:22.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:22.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:22.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:22.318 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:23.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:23.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:23.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:23.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:23.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:23.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:23.714 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:24.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:24.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:24.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:24.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:24.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:24.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:25.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:25.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:25.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:25.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:25.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:25.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:25.109 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:26.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:26.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:26.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:26.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:26.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:26.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:26.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:26.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:26.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:26.506 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:27.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:27.506 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:27.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:27.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:27.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:27.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:27.893 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:28.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:28.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:28.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:28.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:28.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:28.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:29.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:29.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:29.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:29.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:29.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:29.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:29.287 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:30.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:30.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:30.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:30.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:30.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:30.288 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:30.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:30.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:30.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:30.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:30.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:30.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:30.675 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:31.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:31.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:31.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:31.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:31.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:31.679 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:32.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:32.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:32.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:32.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:32.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:32.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:32.076 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:33.072 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:33.073 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:33.074 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:33.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:33.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:33.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:33.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:33.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:33.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:33.464 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:34.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:34.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:34.465 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:34.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:34.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:34.466 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:34.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:34.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:34.855 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:34.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:34.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:34.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:34.856 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:35.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:35.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:35.858 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:36.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:36.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2021-01-15T19:06:36.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2021-01-15T19:06:36.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2021-01-15T19:06:36.241 INFO:tasks.workunit.client.0.smithi145.stdout:active+undersized+degraded+remapped+backfilling 2021-01-15T19:06:37.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 100) 2021-01-15T19:06:37.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2021-01-15T19:06:37.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:06:37.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:06:37.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:06:37.243 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:06:37.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2021-01-15T19:06:37.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2021-01-15T19:06:37.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2021-01-15T19:06:37.625 INFO:tasks.workunit.client.0.smithi145.stdout:active+backfill_unfound+undersized+degraded+remapped 2021-01-15T19:06:37.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2021-01-15T19:06:37.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2021-01-15T19:06:37.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2021-01-15T19:06:37.989 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:06:37.989 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.135826+0000 0'0 37:54 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.990 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.949444+0000 0'0 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.990 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.980796+0000 0'0 37:43 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.990 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.980733+0000 0'0 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.990 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:50.112617+0000 0'0 38:46 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.991 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 301 1 28 0 1 1232896 0 0 101 101 active+backfill_unfound+undersized+degraded+remapped 2021-01-15T19:06:35.673457+0000 29'301 38:889 [2,1,0] 2 [2,1] 2 0'0 2021-01-15T19:04:53.266959+0000 0'0 2021-01-15T19:04:53.266959+0000 0 2021-01-15T19:06:37.991 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.135827+0000 0'0 37:54 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.991 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.949622+0000 0'0 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.991 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:05:48.949737+0000 0'0 37:43 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:04:20.654301+0000 0'0 2021-01-15T19:04:20.654301+0000 0 2021-01-15T19:06:37.992 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:06:37.992 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:06:37.992 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:06:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2021-01-15T19:06:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2021-01-15T19:06:38.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2021-01-15T19:06:43.148 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2021-01-15T19:06:43.367 INFO:tasks.workunit.client.0.smithi145.stderr:pg has no unfound objects 2021-01-15T19:06:43.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2021-01-15T19:06:43.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:06:43.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:06:43.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:06:43.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:06:43.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:06:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:06:43.379 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:06:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:06:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:06:43.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:06:43.469 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:06:43.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:06:43.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:06:43.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:06:43.470 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:06:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:06:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:06:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:06:43.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:06:43.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:06:43.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:06:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018894 2021-01-15T19:06:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018894 2021-01-15T19:06:44.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-128849018894' 2021-01-15T19:06:44.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:06:44.034 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:06:44.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=55834574880 2021-01-15T19:06:44.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 55834574880 2021-01-15T19:06:44.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-128849018894 1-55834574880' 2021-01-15T19:06:44.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:06:44.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:06:44.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345950 2021-01-15T19:06:44.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345950 2021-01-15T19:06:44.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-128849018894 1-55834574880 2-85899345950' 2021-01-15T19:06:44.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:06:44.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-128849018894 2021-01-15T19:06:44.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:06:44.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:06:44.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-128849018894 2021-01-15T19:06:44.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:06:44.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018894 2021-01-15T19:06:44.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 128849018894' 2021-01-15T19:06:44.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:06:44.314 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 128849018894 2021-01-15T19:06:44.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018894 -lt 128849018894 2021-01-15T19:06:44.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:06:44.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-55834574880 2021-01-15T19:06:44.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:06:44.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:06:44.714 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-55834574880 2021-01-15T19:06:44.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:06:44.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=55834574880 2021-01-15T19:06:44.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 55834574880' 2021-01-15T19:06:44.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:06:44.716 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 55834574880 2021-01-15T19:06:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 55834574880 -lt 55834574880 2021-01-15T19:06:45.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:06:45.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-85899345950 2021-01-15T19:06:45.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:06:45.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:06:45.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-85899345950 2021-01-15T19:06:45.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:06:45.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345950 2021-01-15T19:06:45.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 85899345950' 2021-01-15T19:06:45.113 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:06:45.113 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 85899345950 2021-01-15T19:06:45.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345948 -lt 85899345950 2021-01-15T19:06:45.503 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:06:46.504 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:06:46.504 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:06:46.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345951 -lt 85899345950 2021-01-15T19:06:46.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:06:46.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:46.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:47.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:06:47.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:06:47.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:06:47.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:06:47.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:06:47.440 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:06:47.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:06:47.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:06:47.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:06:47.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:06:47.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:47.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:48.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:06:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' -1 2021-01-15T19:06:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T19:06:48.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=8 2021-01-15T19:06:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:06:48.351 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:06:48.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:06:48.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:06:48.451 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:06:48.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:06:48.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:06:48.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:06:48.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:06:48.452 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:06:48.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:06:48.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:06:48.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:48.873 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:49.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:06:49.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:06:49.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:06:49.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:06:49.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:06:49.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:06:49.395 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:06:49.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:06:49.395 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:06:49.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4437 2021-01-15T19:06:49.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4437 '!=' null 2021-01-15T19:06:49.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T19:06:49.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:06:49.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:06:50.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:06:50.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:06:50.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:06:50.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:06:50.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:06:50.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:06:50.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:06:50.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:06:50.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:06:50.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:06:50.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:50.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:50.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:06:50.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:06:50.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:06:50.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:06:50.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:06:50.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:06:50.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:06:50.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:06:50.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:06:51.414 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=6655 2021-01-15T19:06:51.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 6655 '!=' null 2021-01-15T19:06:51.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T19:06:51.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:06:51.415 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:06:51.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:06:51.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:06:51.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:06:51.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:06:51.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:06:51.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:06:51.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:06:51.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:06:51.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:06:51.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:06:51.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:51.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:52.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 9 2021-01-15T19:06:52.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 8 '!=' 8 2021-01-15T19:06:52.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:06:52.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:06:52.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:06:52.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:06:52.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:06:52.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:06:52.485 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:06:52.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=12800 2021-01-15T19:06:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 12800 '!=' null 2021-01-15T19:06:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T19:06:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:06:53.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:06:53.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:06:53.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:06:53.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:06:53.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:06:53.103 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:06:53.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:06:53.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:06:53.104 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:06:53.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:06:53.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:06:53.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:06:53.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:06:53.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:06:53.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:06:53.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:06:53.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2021-01-15T19:06:53.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:53.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2021-01-15T19:06:53.993 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.042 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2021-01-15T19:06:54.043 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2021-01-15T19:06:54.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2021-01-15T19:06:54.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2021-01-15T19:06:54.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2021-01-15T19:06:54.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2021-01-15T19:06:54.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2021-01-15T19:06:54.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2021-01-15T19:06:54.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2021-01-15T19:06:54.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2021-01-15T19:06:54.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2021-01-15T19:06:54.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2021-01-15T19:06:54.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2021-01-15T19:06:54.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2021-01-15T19:06:54.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.425 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2021-01-15T19:06:54.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2021-01-15T19:06:54.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2021-01-15T19:06:54.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2021-01-15T19:06:54.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2021-01-15T19:06:54.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2021-01-15T19:06:54.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2021-01-15T19:06:54.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2021-01-15T19:06:54.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2021-01-15T19:06:54.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2021-01-15T19:06:54.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2021-01-15T19:06:54.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.739 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2021-01-15T19:06:54.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2021-01-15T19:06:54.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2021-01-15T19:06:54.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2021-01-15T19:06:54.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2021-01-15T19:06:54.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2021-01-15T19:06:54.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2021-01-15T19:06:54.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.926 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2021-01-15T19:06:54.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.953 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2021-01-15T19:06:54.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.979 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:54.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:54.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2021-01-15T19:06:54.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2021-01-15T19:06:55.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2021-01-15T19:06:55.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2021-01-15T19:06:55.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2021-01-15T19:06:55.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.113 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2021-01-15T19:06:55.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.140 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2021-01-15T19:06:55.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2021-01-15T19:06:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2021-01-15T19:06:55.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2021-01-15T19:06:55.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.246 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2021-01-15T19:06:55.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2021-01-15T19:06:55.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2021-01-15T19:06:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2021-01-15T19:06:55.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2021-01-15T19:06:55.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2021-01-15T19:06:55.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2021-01-15T19:06:55.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.434 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2021-01-15T19:06:55.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.460 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2021-01-15T19:06:55.461 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.486 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2021-01-15T19:06:55.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.512 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2021-01-15T19:06:55.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2021-01-15T19:06:55.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.566 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2021-01-15T19:06:55.567 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2021-01-15T19:06:55.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2021-01-15T19:06:55.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2021-01-15T19:06:55.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.825 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2021-01-15T19:06:55.826 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2021-01-15T19:06:55.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2021-01-15T19:06:55.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2021-01-15T19:06:55.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2021-01-15T19:06:55.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2021-01-15T19:06:55.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:55.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:55.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2021-01-15T19:06:55.985 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2021-01-15T19:06:56.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.039 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2021-01-15T19:06:56.040 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.065 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2021-01-15T19:06:56.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2021-01-15T19:06:56.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.118 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2021-01-15T19:06:56.119 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2021-01-15T19:06:56.145 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2021-01-15T19:06:56.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2021-01-15T19:06:56.199 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2021-01-15T19:06:56.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2021-01-15T19:06:56.253 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2021-01-15T19:06:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2021-01-15T19:06:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2021-01-15T19:06:56.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2021-01-15T19:06:56.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2021-01-15T19:06:56.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2021-01-15T19:06:56.429 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2021-01-15T19:06:56.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2021-01-15T19:06:56.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2021-01-15T19:06:56.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2021-01-15T19:06:56.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.579 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2021-01-15T19:06:56.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2021-01-15T19:06:56.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2021-01-15T19:06:56.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2021-01-15T19:06:56.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2021-01-15T19:06:56.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2021-01-15T19:06:56.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2021-01-15T19:06:56.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.766 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2021-01-15T19:06:56.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2021-01-15T19:06:56.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2021-01-15T19:06:56.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2021-01-15T19:06:56.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2021-01-15T19:06:56.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2021-01-15T19:06:56.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2021-01-15T19:06:56.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2021-01-15T19:06:56.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:56.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:56.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2021-01-15T19:06:56.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2021-01-15T19:06:57.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2021-01-15T19:06:57.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2021-01-15T19:06:57.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2021-01-15T19:06:57.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.114 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2021-01-15T19:06:57.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2021-01-15T19:06:57.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2021-01-15T19:06:57.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2021-01-15T19:06:57.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2021-01-15T19:06:57.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2021-01-15T19:06:57.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2021-01-15T19:06:57.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2021-01-15T19:06:57.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2021-01-15T19:06:57.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2021-01-15T19:06:57.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2021-01-15T19:06:57.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2021-01-15T19:06:57.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2021-01-15T19:06:57.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2021-01-15T19:06:57.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2021-01-15T19:06:57.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2021-01-15T19:06:57.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2021-01-15T19:06:57.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2021-01-15T19:06:57.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2021-01-15T19:06:57.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2021-01-15T19:06:57.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2021-01-15T19:06:57.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.676 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2021-01-15T19:06:57.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2021-01-15T19:06:57.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2021-01-15T19:06:57.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2021-01-15T19:06:57.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2021-01-15T19:06:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2021-01-15T19:06:57.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.836 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2021-01-15T19:06:57.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.863 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2021-01-15T19:06:57.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2021-01-15T19:06:57.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2021-01-15T19:06:57.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2021-01-15T19:06:57.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2021-01-15T19:06:57.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:57.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:57.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2021-01-15T19:06:57.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2021-01-15T19:06:58.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2021-01-15T19:06:58.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2021-01-15T19:06:58.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2021-01-15T19:06:58.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2021-01-15T19:06:58.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2021-01-15T19:06:58.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2021-01-15T19:06:58.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2021-01-15T19:06:58.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2021-01-15T19:06:58.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2021-01-15T19:06:58.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2021-01-15T19:06:58.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2021-01-15T19:06:58.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2021-01-15T19:06:58.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2021-01-15T19:06:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2021-01-15T19:06:58.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2021-01-15T19:06:58.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2021-01-15T19:06:58.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2021-01-15T19:06:58.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2021-01-15T19:06:58.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2021-01-15T19:06:58.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.606 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2021-01-15T19:06:58.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2021-01-15T19:06:58.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2021-01-15T19:06:58.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2021-01-15T19:06:58.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2021-01-15T19:06:58.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2021-01-15T19:06:58.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2021-01-15T19:06:58.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2021-01-15T19:06:58.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2021-01-15T19:06:58.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2021-01-15T19:06:58.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2021-01-15T19:06:58.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2021-01-15T19:06:58.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2021-01-15T19:06:58.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.954 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2021-01-15T19:06:58.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:58.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2021-01-15T19:06:58.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2021-01-15T19:06:59.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2021-01-15T19:06:59.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2021-01-15T19:06:59.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2021-01-15T19:06:59.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2021-01-15T19:06:59.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2021-01-15T19:06:59.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2021-01-15T19:06:59.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2021-01-15T19:06:59.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2021-01-15T19:06:59.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2021-01-15T19:06:59.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2021-01-15T19:06:59.276 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2021-01-15T19:06:59.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2021-01-15T19:06:59.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2021-01-15T19:06:59.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2021-01-15T19:06:59.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2021-01-15T19:06:59.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2021-01-15T19:06:59.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.462 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2021-01-15T19:06:59.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.489 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2021-01-15T19:06:59.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2021-01-15T19:06:59.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2021-01-15T19:06:59.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2021-01-15T19:06:59.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2021-01-15T19:06:59.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2021-01-15T19:06:59.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2021-01-15T19:06:59.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2021-01-15T19:06:59.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2021-01-15T19:06:59.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2021-01-15T19:06:59.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2021-01-15T19:06:59.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2021-01-15T19:06:59.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2021-01-15T19:06:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2021-01-15T19:06:59.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2021-01-15T19:06:59.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2021-01-15T19:06:59.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2021-01-15T19:06:59.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2021-01-15T19:06:59.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2021-01-15T19:06:59.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.997 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:06:59.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:06:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2021-01-15T19:06:59.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2021-01-15T19:07:00.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2021-01-15T19:07:00.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.077 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2021-01-15T19:07:00.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2021-01-15T19:07:00.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2021-01-15T19:07:00.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2021-01-15T19:07:00.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2021-01-15T19:07:00.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2021-01-15T19:07:00.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2021-01-15T19:07:00.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2021-01-15T19:07:00.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2021-01-15T19:07:00.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2021-01-15T19:07:00.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2021-01-15T19:07:00.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2021-01-15T19:07:00.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.426 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.427 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2021-01-15T19:07:00.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2021-01-15T19:07:00.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.484 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2021-01-15T19:07:00.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.507 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2021-01-15T19:07:00.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2021-01-15T19:07:00.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2021-01-15T19:07:00.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2021-01-15T19:07:00.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2021-01-15T19:07:00.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2021-01-15T19:07:00.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2021-01-15T19:07:00.689 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2021-01-15T19:07:00.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2021-01-15T19:07:00.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2021-01-15T19:07:00.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2021-01-15T19:07:00.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2021-01-15T19:07:00.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2021-01-15T19:07:00.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2021-01-15T19:07:00.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2021-01-15T19:07:00.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2021-01-15T19:07:00.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.983 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:00.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:00.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2021-01-15T19:07:00.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2021-01-15T19:07:01.011 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.035 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test-pool/obj250: (2) No such file or directory 2021-01-15T19:07:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2021-01-15T19:07:01.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2021-01-15T19:07:01.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2021-01-15T19:07:01.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2021-01-15T19:07:01.117 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2021-01-15T19:07:01.144 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2021-01-15T19:07:01.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2021-01-15T19:07:01.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2021-01-15T19:07:01.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2021-01-15T19:07:01.252 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.277 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2021-01-15T19:07:01.278 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.302 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2021-01-15T19:07:01.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.329 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2021-01-15T19:07:01.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2021-01-15T19:07:01.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2021-01-15T19:07:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2021-01-15T19:07:01.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2021-01-15T19:07:01.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2021-01-15T19:07:01.465 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.490 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2021-01-15T19:07:01.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2021-01-15T19:07:01.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2021-01-15T19:07:01.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2021-01-15T19:07:01.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2021-01-15T19:07:01.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.623 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2021-01-15T19:07:01.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2021-01-15T19:07:01.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2021-01-15T19:07:01.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2021-01-15T19:07:01.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2021-01-15T19:07:01.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2021-01-15T19:07:01.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2021-01-15T19:07:01.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2021-01-15T19:07:01.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.840 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.841 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2021-01-15T19:07:01.842 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2021-01-15T19:07:01.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2021-01-15T19:07:01.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.918 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2021-01-15T19:07:01.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2021-01-15T19:07:01.946 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2021-01-15T19:07:01.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.998 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:01.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:01.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2021-01-15T19:07:01.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2021-01-15T19:07:02.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2021-01-15T19:07:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2021-01-15T19:07:02.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2021-01-15T19:07:02.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2021-01-15T19:07:02.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2021-01-15T19:07:02.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2021-01-15T19:07:02.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2021-01-15T19:07:02.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.238 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2021-01-15T19:07:02.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2021-01-15T19:07:02.267 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2021-01-15T19:07:02.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2021-01-15T19:07:02.350 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:07:02.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2021-01-15T19:07:02.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.401 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:07:02.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2021-01-15T19:07:02.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test-pool 2021-01-15T19:07:02.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2021-01-15T19:07:02.945 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test-pool' does not exist 2021-01-15T19:07:02.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:07:02.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:02.955 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:07:02.956 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:07:02.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:07:02.957 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:07:02.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:07:02.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:07:02.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:07:03.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:07:03.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:07:03.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:07:03.064 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:07:03.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:07:03.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:07:03.065 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:07:03.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:07:03.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:07:03.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:07:03.067 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:07:03.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:07:03.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:07:03.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:07:03.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.092 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:07:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:07:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:07:03.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2021-01-15T19:07:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:07:03.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:07:03.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:07:03.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:07:03.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:07:03.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:07:03.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:07:03.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:07:03.099 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:07:03.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:07:03.100 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:07:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:07:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:07:03.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:07:03.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:07:03.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:07:03.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:07:03.102 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:07:03.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:07:03.104 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:07:03.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:07:03.105 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:07:03.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:07:03.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:07:03.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:07:03.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.107 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.937479 2021-01-15T19:07:03.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:07:03.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T19:07:03.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:07:03.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2021-01-15T19:07:03.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2021-01-15T19:07:03.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:07:03.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:07:03.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:07:03.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2021-01-15T19:07:03.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2021-01-15T19:07:03.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:07:03.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:03.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:03.418 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:03.418 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.418 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.418 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:03.419 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon_osd_warn_num_repaired=10 2021-01-15T19:07:03.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:07:03.612 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:07:03.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:07:03.613 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:07:03.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:07:03.614 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:07:03.615 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:07:03.615 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:07:03.616 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:07:03.616 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:07:03.617 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.617 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.617 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:07:03.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:07:03.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get fsid 2021-01-15T19:07:03.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:07:03.712 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:07:03.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:07:03.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:07:03.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:07:03.713 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:07:03.714 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:07:03.714 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:07:03.714 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:07:03.714 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:03.714 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:03.715 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:07:03.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:07:03.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get mon_host 2021-01-15T19:07:03.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2021-01-15T19:07:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:07:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:07:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:07:03.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2021-01-15T19:07:03.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:07:04.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:07:04.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:04.713 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:04.714 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:04.715 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:04.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:04.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:07:04.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:07:04.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2021-01-15T19:07:05.072 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:07:05.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_read_unfound td/osd-rep-recov-eio.sh 2021-01-15T19:07:05.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:420: TEST_rep_read_unfound: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:05.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:421: TEST_rep_read_unfound: local objname=myobject 2021-01-15T19:07:05.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:423: TEST_rep_read_unfound: setup_osds 3 _filestore 2021-01-15T19:07:05.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2021-01-15T19:07:05.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2021-01-15T19:07:05.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type=_filestore 2021-01-15T19:07:05.086 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2021-01-15T19:07:05.087 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2021-01-15T19:07:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:07:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 0 2021-01-15T19:07:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:05.087 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T19:07:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=0 2021-01-15T19:07:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T19:07:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:07:05.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:07:05.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:07:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T19:07:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T19:07:05.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:05.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T19:07:05.090 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:05.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:05.091 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:05.091 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T19:07:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T19:07:05.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:05.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:05.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:05.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:05.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:05.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T19:07:05.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T19:07:05.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:07:05.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T19:07:05.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=636b893f-a801-4538-b60b-678010371b40 2021-01-15T19:07:05.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd2 636b893f-a801-4538-b60b-678010371b40' 2021-01-15T19:07:05.095 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T19:07:05.096 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 636b893f-a801-4538-b60b-678010371b40 2021-01-15T19:07:05.102 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQDZ5wFgytYTBhAAutfCaGpXnKaBE3p69705nw== 2021-01-15T19:07:05.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQDZ5wFgytYTBhAAutfCaGpXnKaBE3p69705nw=="}' 2021-01-15T19:07:05.103 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 636b893f-a801-4538-b60b-678010371b40 -i td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:07:05.452 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:07:05.463 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:07:05.464 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDZ5wFgytYTBhAAutfCaGpXnKaBE3p69705nw== --osd-uuid 636b893f-a801-4538-b60b-678010371b40 --osd-objectstore=filestore 2021-01-15T19:07:05.490 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:05.489+0000 7ff9b0396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:05.492 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:05.491+0000 7ff9b0396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:05.493 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:05.492+0000 7ff9b0396080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:05.853 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:05.852+0000 7ff9b0396080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:06.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.230+0000 7ff9b0396080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:06.232 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.230+0000 7ff9b0396080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:06.233 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.230+0000 7ff9b0396080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:06.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2021-01-15T19:07:06.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T19:07:06.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd0 key to auth repository 2021-01-15T19:07:06.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:07:06.662 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:07:06.850 INFO:tasks.workunit.client.0.smithi145.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2021-01-15T19:07:06.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.0 2021-01-15T19:07:06.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:06.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T19:07:06.852 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:07:06.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T19:07:06.853 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T19:07:06.875 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.874+0000 7f89cfae7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:06.878 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.876+0000 7f89cfae7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:06.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.878+0000 7f89cfae7080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:06.880 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:06.879+0000 7f89cfae7080 -1 Falling back to public interface 2021-01-15T19:07:07.177 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:07.176+0000 7f89cfae7080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:07.184+0000 7f89cfae7080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:07.185 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:07.184+0000 7f89cfae7080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:07.192 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:07.190+0000 7f89cfae7080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:07:07.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 0 2021-01-15T19:07:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:07:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:07.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:07.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:07:07.211 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:07:07.492 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:07:08.493 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:07:08.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:08.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:07:08.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:08.494 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:07:08.495 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:07:08.796 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/978283,v1:127.0.0.1:6801/978283] [v2:127.0.0.1:6802/978283,v1:127.0.0.1:6803/978283] exists,up 636b893f-a801-4538-b60b-678010371b40 2021-01-15T19:07:08.796 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 51f30012-bdd0-405b-bd96-b0d9b62963a9 2021-01-15T19:07:08.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:08.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:08.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:08.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:07:08.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 1 2021-01-15T19:07:08.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:08.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T19:07:08.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=1 2021-01-15T19:07:08.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T19:07:08.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:08.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:08.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:07:08.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:07:08.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T19:07:08.800 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T19:07:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:08.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T19:07:08.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:08.801 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:08.801 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:08.802 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:08.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:08.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:08.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T19:07:08.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T19:07:08.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:08.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:08.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:08.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:08.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:08.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T19:07:08.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T19:07:08.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:08.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T19:07:08.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=51f30012-bdd0-405b-bd96-b0d9b62963a9 2021-01-15T19:07:08.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd2 51f30012-bdd0-405b-bd96-b0d9b62963a9' 2021-01-15T19:07:08.806 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T19:07:08.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQDc5wFg4hT+LxAAt847a7PgMqavHQ8z6Bb8Eg== 2021-01-15T19:07:08.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQDc5wFg4hT+LxAAt847a7PgMqavHQ8z6Bb8Eg=="}' 2021-01-15T19:07:08.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 51f30012-bdd0-405b-bd96-b0d9b62963a9 -i td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:07:09.368 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:07:09.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:07:09.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDc5wFg4hT+LxAAt847a7PgMqavHQ8z6Bb8Eg== --osd-uuid 51f30012-bdd0-405b-bd96-b0d9b62963a9 --osd-objectstore=filestore 2021-01-15T19:07:09.405 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:09.404+0000 7f8b5bffa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:09.407 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:09.406+0000 7f8b5bffa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:09.408 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:09.407+0000 7f8b5bffa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:09.994 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:09.992+0000 7f8b5bffa080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:10.429 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:10.428+0000 7f8b5bffa080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:10.430 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:10.428+0000 7f8b5bffa080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:10.430 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:10.428+0000 7f8b5bffa080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:11.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2021-01-15T19:07:11.046 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T19:07:11.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd1 key to auth repository 2021-01-15T19:07:11.047 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:07:11.047 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:07:11.388 INFO:tasks.workunit.client.0.smithi145.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2021-01-15T19:07:11.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.1 2021-01-15T19:07:11.390 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:07:11.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:11.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T19:07:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T19:07:11.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T19:07:11.413 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.412+0000 7fca61da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:11.415 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.414+0000 7fca61da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:11.417 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.416+0000 7fca61da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:11.418 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.417+0000 7fca61da0080 -1 Falling back to public interface 2021-01-15T19:07:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.732+0000 7fca61da0080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.733+0000 7fca61da0080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:11.734 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.733+0000 7fca61da0080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:11.741 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:11.738+0000 7fca61da0080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:07:11.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 1 2021-01-15T19:07:11.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:11.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:07:11.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:11.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:11.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:11.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:11.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:11.960 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:07:11.961 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:07:12.485 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:07:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:07:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:07:13.487 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:13.488 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:07:13.488 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:07:13.892 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 14 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/978639,v1:127.0.0.1:6811/978639] [v2:127.0.0.1:6812/978639,v1:127.0.0.1:6813/978639] exists,up 51f30012-bdd0-405b-bd96-b0d9b62963a9 2021-01-15T19:07:13.893 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 8c11dd84-5b41-42d6-ae59-9d75c922be65 2021-01-15T19:07:13.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:13.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:07:13.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 2 2021-01-15T19:07:13.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:687: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:13.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:688: run_osd_filestore: shift 2021-01-15T19:07:13.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local id=2 2021-01-15T19:07:13.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2021-01-15T19:07:13.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:07:13.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:13.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:694: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:13.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:13.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T19:07:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T19:07:13.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --chdir=' 2021-01-15T19:07:13.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+= 2021-01-15T19:07:13.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:13.900 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: get_asok_path 2021-01-15T19:07:13.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:13.901 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:13.902 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:13.902 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:13.903 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:13.903 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:13.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:13.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:13.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --debug-ms=1' 2021-01-15T19:07:13.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-monc=20' 2021-01-15T19:07:13.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:13.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:13.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:13.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:13.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:13.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' ' 2021-01-15T19:07:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+= 2021-01-15T19:07:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T19:07:13.906 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: uuidgen 2021-01-15T19:07:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: local uuid=8c11dd84-5b41-42d6-ae59-9d75c922be65 2021-01-15T19:07:13.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:716: run_osd_filestore: echo 'add osd2 8c11dd84-5b41-42d6-ae59-9d75c922be65' 2021-01-15T19:07:13.907 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: ceph-authtool --gen-print-key 2021-01-15T19:07:13.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: OSD_SECRET=AQDh5wFg644ONhAA+ssOeVMCGA3YAf27aalRVg== 2021-01-15T19:07:13.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo '{"cephx_secret": "AQDh5wFg644ONhAA+ssOeVMCGA3YAf27aalRVg=="}' 2021-01-15T19:07:13.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph osd new 8c11dd84-5b41-42d6-ae59-9d75c922be65 -i td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:07:14.441 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:07:14.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: rm td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:07:14.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDh5wFg644ONhAA+ssOeVMCGA3YAf27aalRVg== --osd-uuid 8c11dd84-5b41-42d6-ae59-9d75c922be65 --osd-objectstore=filestore 2021-01-15T19:07:14.479 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:14.478+0000 7f6dc2579080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:14.481 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:14.480+0000 7f6dc2579080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:14.482 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:14.481+0000 7f6dc2579080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:15.110 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:15.108+0000 7f6dc2579080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:15.513 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:15.512+0000 7f6dc2579080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:15.514 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:15.512+0000 7f6dc2579080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:15.514 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:15.512+0000 7f6dc2579080 -1 journal do_read_entry(4096): bad header magic 2021-01-15T19:07:15.980 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2021-01-15T19:07:15.981 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:724: run_osd_filestore: cat 2021-01-15T19:07:15.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:728: run_osd_filestore: echo adding osd2 key to auth repository 2021-01-15T19:07:15.982 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:729: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:07:15.983 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T19:07:16.508 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo start osd.2 2021-01-15T19:07:16.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:16.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: ceph osd dump --format=json 2021-01-15T19:07:16.509 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: jq '.flags_set[]' 2021-01-15T19:07:16.510 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:734: run_osd_filestore: grep -q '"noup"' 2021-01-15T19:07:16.510 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:07:16.530 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:16.529+0000 7fb6a6f88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:16.533 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:16.531+0000 7fb6a6f88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:16.534 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:16.533+0000 7fb6a6f88080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:16.536 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:16.535+0000 7fb6a6f88080 -1 Falling back to public interface 2021-01-15T19:07:16.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:737: run_osd_filestore: wait_for_osd up 2 2021-01-15T19:07:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:07:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:16.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:16.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:16.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:16.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:16.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:07:16.936 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:07:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:17.059+0000 7fb6a6f88080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:17.061 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:17.059+0000 7fb6a6f88080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:17.062 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:17.059+0000 7fb6a6f88080 -1 journal do_read_entry(8192): bad header magic 2021-01-15T19:07:17.067 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:17.065+0000 7fb6a6f88080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T19:07:17.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:07:18.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:07:18.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:18.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:07:18.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:18.328 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:07:18.328 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:07:18.836 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 20 up_thru 20 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/978934,v1:127.0.0.1:6819/978934] [v2:127.0.0.1:6820/978934,v1:127.0.0.1:6821/978934] exists,up 8c11dd84-5b41-42d6-ae59-9d75c922be65 2021-01-15T19:07:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2021-01-15T19:07:18.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:07:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:07:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:07:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:07:18.838 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:07:18.838 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:07:18.839 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:07:18.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:07:18.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:07:18.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:07:18.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:07:18.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:07:18.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:07:18.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:18.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:19.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:19.322 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:19.323 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:19.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:19.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:19.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:19.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803780 2021-01-15T19:07:19.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803780 2021-01-15T19:07:19.458 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803780' 2021-01-15T19:07:19.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:19.459 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542148 2021-01-15T19:07:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542148 2021-01-15T19:07:19.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803780 1-60129542148' 2021-01-15T19:07:19.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:19.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:19.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345923 2021-01-15T19:07:19.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345923 2021-01-15T19:07:19.729 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803780 1-60129542148 2-85899345923' 2021-01-15T19:07:19.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:19.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803780 2021-01-15T19:07:19.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:19.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:19.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803780 2021-01-15T19:07:19.731 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:19.732 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803780 2021-01-15T19:07:19.735 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803780' 2021-01-15T19:07:19.735 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:19.735 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803780 2021-01-15T19:07:20.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803778 -lt 25769803780 2021-01-15T19:07:20.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:07:21.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:07:21.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:21.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803781 -lt 25769803780 2021-01-15T19:07:21.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:21.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-60129542148 2021-01-15T19:07:21.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:21.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:21.524 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-60129542148 2021-01-15T19:07:21.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:21.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542148 2021-01-15T19:07:21.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 60129542148' 2021-01-15T19:07:21.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:07:21.526 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 60129542148 2021-01-15T19:07:21.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542148 -lt 60129542148 2021-01-15T19:07:21.945 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:21.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-85899345923 2021-01-15T19:07:21.946 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:21.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:07:21.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-85899345923 2021-01-15T19:07:21.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:21.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345923 2021-01-15T19:07:21.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 85899345923' 2021-01-15T19:07:21.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:07:21.949 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 85899345923 2021-01-15T19:07:22.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345923 -lt 85899345923 2021-01-15T19:07:22.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:07:22.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:22.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:23.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T19:07:23.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:07:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:07:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:07:23.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:07:23.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:07:23.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:07:23.017 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:07:23.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:07:23.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:07:23.401 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:23.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:23.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T19:07:23.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:07:23.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:07:23.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:425: TEST_rep_read_unfound: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2021-01-15T19:07:24.433 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' does not exist 2021-01-15T19:07:24.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:426: TEST_rep_read_unfound: local poolname=test-pool 2021-01-15T19:07:24.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:427: TEST_rep_read_unfound: create_pool test-pool 1 1 2021-01-15T19:07:24.443 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test-pool 1 1 2021-01-15T19:07:25.146 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test-pool' already exists 2021-01-15T19:07:25.155 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T19:07:26.156 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:428: TEST_rep_read_unfound: ceph osd pool set test-pool size 2 2021-01-15T19:07:26.919 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 2 size to 2 2021-01-15T19:07:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:429: TEST_rep_read_unfound: wait_for_clean 2021-01-15T19:07:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:07:26.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:07:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:07:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:07:26.930 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:07:26.930 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:07:26.931 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:07:26.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:07:26.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:07:26.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:07:27.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:07:27.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:07:27.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:07:27.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:27.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:27.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:27.526 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:27.526 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:27.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:27.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:27.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803784 2021-01-15T19:07:27.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803784 2021-01-15T19:07:27.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784' 2021-01-15T19:07:27.661 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:27.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:27.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542151 2021-01-15T19:07:27.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542151 2021-01-15T19:07:27.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-60129542151' 2021-01-15T19:07:27.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:27.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:27.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=85899345926 2021-01-15T19:07:27.929 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 85899345926 2021-01-15T19:07:27.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803784 1-60129542151 2-85899345926' 2021-01-15T19:07:27.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:27.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803784 2021-01-15T19:07:27.931 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:27.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:27.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803784 2021-01-15T19:07:27.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:27.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803784 2021-01-15T19:07:27.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803784' 2021-01-15T19:07:27.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:27.934 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803784 2021-01-15T19:07:28.322 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803782 -lt 25769803784 2021-01-15T19:07:28.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:07:29.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:07:29.324 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:29.720 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803784 -lt 25769803784 2021-01-15T19:07:29.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:29.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-60129542151 2021-01-15T19:07:29.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:29.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:29.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-60129542151 2021-01-15T19:07:29.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542151 2021-01-15T19:07:29.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 60129542151' 2021-01-15T19:07:29.730 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:07:29.730 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 60129542151 2021-01-15T19:07:30.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542152 -lt 60129542151 2021-01-15T19:07:30.123 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:30.123 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-85899345926 2021-01-15T19:07:30.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:30.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:07:30.124 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-85899345926 2021-01-15T19:07:30.125 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:30.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=85899345926 2021-01-15T19:07:30.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 85899345926' 2021-01-15T19:07:30.126 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:07:30.126 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 85899345926 2021-01-15T19:07:30.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 85899345926 -lt 85899345926 2021-01-15T19:07:30.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:07:30.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:30.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:31.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T19:07:31.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:07:31.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:07:31.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:07:31.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:07:31.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:07:31.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:07:31.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:07:31.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T19:07:31.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:07:31.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:31.434 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T19:07:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:07:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:07:31.952 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:431: TEST_rep_read_unfound: ceph pg dump pgs 2021-01-15T19:07:32.319 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:07:32.319 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:07:27.921923+0000 0'0 28:20 [2,1] 2 [2,1] 2 0'0 2021-01-15T19:07:24.786082+0000 0'0 2021-01-15T19:07:24.786082+0000 0 2021-01-15T19:07:32.320 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:07:32.320 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:07:32.321 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:07:32.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:433: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:32.331 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T19:07:32.331 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T19:07:32.331 INFO:tasks.workunit.client.0.smithi145.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 7.8226e-05 s, 105 MB/s 2021-01-15T19:07:32.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:434: TEST_rep_read_unfound: rados -p test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:32.416 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: get_primary test-pool myobject 2021-01-15T19:07:32.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test-pool 2021-01-15T19:07:32.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=myobject 2021-01-15T19:07:32.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test-pool myobject 2021-01-15T19:07:32.417 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T19:07:32.854 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: local primary=2 2021-01-15T19:07:32.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: get_not_primary test-pool myobject 2021-01-15T19:07:32.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local poolname=test-pool 2021-01-15T19:07:32.855 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: get_not_primary: local objectname=myobject 2021-01-15T19:07:32.855 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: get_primary test-pool myobject 2021-01-15T19:07:32.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local poolname=test-pool 2021-01-15T19:07:32.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: get_primary: local objectname=myobject 2021-01-15T19:07:32.856 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: ceph --format json osd map test-pool myobject 2021-01-15T19:07:32.857 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1138: get_primary: jq .acting_primary 2021-01-15T19:07:33.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: local primary=2 2021-01-15T19:07:33.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: ceph --format json osd map test-pool myobject 2021-01-15T19:07:33.251 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2021-01-15T19:07:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: local other=1 2021-01-15T19:07:33.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:439: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:33.645 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records in 2021-01-15T19:07:33.646 INFO:tasks.workunit.client.0.smithi145.stderr:1+0 records out 2021-01-15T19:07:33.646 INFO:tasks.workunit.client.0.smithi145.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 8.193e-05 s, 100 MB/s 2021-01-15T19:07:33.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:440: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:33.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:33.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2021-01-15T19:07:33.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=2 2021-01-15T19:07:33.648 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2021-01-15T19:07:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:33.649 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2021-01-15T19:07:33.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=2 2021-01-15T19:07:33.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2021-01-15T19:07:33.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.2 2021-01-15T19:07:33.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:07:33.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:07:33.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:07:33.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:07:33.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:07:33.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:07:33.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T19:07:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=2 2021-01-15T19:07:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T19:07:33.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:07:33.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:34.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 2 2021-01-15T19:07:34.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:34.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:07:34.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T19:07:34.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:07:34.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:07:34.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:34.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:34.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:34.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:34.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T19:07:34.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T19:07:34.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:07:34.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:07:34.675 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:34.675 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:07:34.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:34.676 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:34.676 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:34.676 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:34.677 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:34.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:34.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:07:34.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:07:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T19:07:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T19:07:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:start osd.2 2021-01-15T19:07:34.680 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:34.681 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-rep-recov-eio.sh/2/whoami 2021-01-15T19:07:34.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T19:07:34.681 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:07:34.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:07:34.682 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:07:34.694 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:34.693+0000 7f86cf6fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:34.698 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:34.696+0000 7f86cf6fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:34.699 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:34.698+0000 7f86cf6fa080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:34.702 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:34.700+0000 7f86cf6fa080 -1 Falling back to public interface 2021-01-15T19:07:35.076 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:35.074+0000 7f86cf6fa080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:35.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:35.082+0000 7f86cf6fa080 -1 journal do_read_entry(450560): bad header magic 2021-01-15T19:07:35.083 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:35.082+0000 7f86cf6fa080 -1 journal do_read_entry(450560): bad header magic 2021-01-15T19:07:35.089 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:35.088+0000 7f86cf6fa080 -1 osd.2 28 log_to_monitors {default=true} 2021-01-15T19:07:35.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T19:07:35.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:35.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:07:35.106 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:35.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:35.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:35.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T19:07:35.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:35.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:07:35.526 INFO:tasks.workunit.client.0.smithi145.stderr:osd.2 up in weight 1 up_from 31 up_thru 31 down_at 29 last_clean_interval [20,28) [v2:127.0.0.1:6818/980156,v1:127.0.0.1:6819/980156] [v2:127.0.0.1:6820/980156,v1:127.0.0.1:6821/980156] exists,up 8c11dd84-5b41-42d6-ae59-9d75c922be65 2021-01-15T19:07:35.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:35.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:35.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:35.527 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2021-01-15T19:07:35.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:07:35.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:07:35.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:07:35.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:07:35.528 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:07:35.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:07:35.529 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:07:35.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:07:35.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:07:35.530 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:07:35.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:07:35.617 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:07:35.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:07:35.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:35.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:36.012 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:36.472 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:36.473 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:36.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:36.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:36.473 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:36.475 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803788 2021-01-15T19:07:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803788 2021-01-15T19:07:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788' 2021-01-15T19:07:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:36.476 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:36.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=60129542155 2021-01-15T19:07:36.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 60129542155 2021-01-15T19:07:36.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-60129542155' 2021-01-15T19:07:36.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:36.477 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:36.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986179 2021-01-15T19:07:36.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986179 2021-01-15T19:07:36.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803788 1-60129542155 2-133143986179' 2021-01-15T19:07:36.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:36.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803788 2021-01-15T19:07:36.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:36.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803788 2021-01-15T19:07:36.479 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803788 2021-01-15T19:07:36.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803788' 2021-01-15T19:07:36.480 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.0 seq 25769803788 2021-01-15T19:07:36.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:36.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803788 -lt 25769803788 2021-01-15T19:07:36.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:36.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-60129542155 2021-01-15T19:07:36.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:36.876 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:36.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-60129542155 2021-01-15T19:07:36.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:36.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=60129542155 2021-01-15T19:07:36.877 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 60129542155' 2021-01-15T19:07:36.877 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.1 seq 60129542155 2021-01-15T19:07:36.878 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:07:37.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 60129542155 -lt 60129542155 2021-01-15T19:07:37.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:37.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-133143986179 2021-01-15T19:07:37.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:37.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:07:37.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986179 2021-01-15T19:07:37.273 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:37.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986179 2021-01-15T19:07:37.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2021-01-15T19:07:37.274 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.2 seq 133143986179 2021-01-15T19:07:37.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:07:37.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986179 -lt 133143986179 2021-01-15T19:07:37.667 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:07:37.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:37.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:38.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T19:07:38.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:07:38.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:07:38.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:07:38.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:07:38.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:07:38.227 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:07:38.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:07:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T19:07:38.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:07:38.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:38.657 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:39.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T19:07:39.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:07:39.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:07:39.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:441: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:39.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:39.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2021-01-15T19:07:39.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2021-01-15T19:07:39.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2021-01-15T19:07:39.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:39.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:39.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2021-01-15T19:07:39.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2021-01-15T19:07:39.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2021-01-15T19:07:39.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2021-01-15T19:07:39.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:07:39.168 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:07:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:07:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:07:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:07:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:07:39.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T19:07:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T19:07:39.271 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T19:07:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:39.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2021-01-15T19:07:40.228 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2021-01-15T19:07:40.229 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:40.230 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:07:40.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T19:07:40.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:07:40.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:40.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:40.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:40.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:40.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:40.236 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:07:40.237 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:07:40.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:07:40.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:07:40.244 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:40.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:07:40.244 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:40.245 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:40.245 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:40.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:40.246 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:40.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:40.247 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:40.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:40.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:40.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:07:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:07:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:40.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T19:07:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:start osd.1 2021-01-15T19:07:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:40.250 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2021-01-15T19:07:40.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T19:07:40.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:07:40.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:07:40.251 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:07:40.255 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.252+0000 7f1e46ea1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:40.257 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.254+0000 7f1e46ea1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:40.258 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.255+0000 7f1e46ea1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:40.264 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.258+0000 7f1e46ea1080 -1 Falling back to public interface 2021-01-15T19:07:40.524 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.522+0000 7f1e46ea1080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:40.526 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.522+0000 7f1e46ea1080 -1 journal do_read_entry(675840): bad header magic 2021-01-15T19:07:40.527 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.522+0000 7f1e46ea1080 -1 journal do_read_entry(675840): bad header magic 2021-01-15T19:07:40.530 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:40.529+0000 7f1e46ea1080 -1 osd.1 32 log_to_monitors {default=true} 2021-01-15T19:07:40.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T19:07:40.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:40.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:07:40.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:40.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:40.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:40.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:40.637 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T19:07:40.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:40.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:07:41.074 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 up in weight 1 up_from 35 up_thru 29 down_at 33 last_clean_interval [14,32) [v2:127.0.0.1:6810/980727,v1:127.0.0.1:6811/980727] [v2:127.0.0.1:6812/980727,v1:127.0.0.1:6813/980727] exists,up 51f30012-bdd0-405b-bd96-b0d9b62963a9 2021-01-15T19:07:41.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:41.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:41.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:41.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2021-01-15T19:07:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:07:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:07:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:07:41.076 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:07:41.077 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:07:41.077 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:07:41.077 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:07:41.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:07:41.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:07:41.078 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:07:41.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:07:41.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:07:41.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:07:41.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:41.165 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:41.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:41.562 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:41.562 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:41.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:41.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:41.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:41.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803791 2021-01-15T19:07:41.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803791 2021-01-15T19:07:41.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791' 2021-01-15T19:07:41.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:41.700 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:41.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855363 2021-01-15T19:07:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855363 2021-01-15T19:07:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-150323855363' 2021-01-15T19:07:41.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:41.832 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:41.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986182 2021-01-15T19:07:41.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986182 2021-01-15T19:07:41.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803791 1-150323855363 2-133143986182' 2021-01-15T19:07:41.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:41.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803791 2021-01-15T19:07:41.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:41.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:41.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803791 2021-01-15T19:07:41.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:41.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803791 2021-01-15T19:07:41.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803791' 2021-01-15T19:07:41.970 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.0 seq 25769803791 2021-01-15T19:07:41.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:42.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803789 -lt 25769803791 2021-01-15T19:07:42.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:07:43.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:07:43.365 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:43.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803791 -lt 25769803791 2021-01-15T19:07:43.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:43.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-150323855363 2021-01-15T19:07:43.763 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:43.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:43.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-150323855363 2021-01-15T19:07:43.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855363 2021-01-15T19:07:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 150323855363' 2021-01-15T19:07:43.765 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.1 seq 150323855363 2021-01-15T19:07:43.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:07:44.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855363 -lt 150323855363 2021-01-15T19:07:44.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:44.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-133143986182 2021-01-15T19:07:44.160 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:44.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:07:44.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986182 2021-01-15T19:07:44.161 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:44.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986182 2021-01-15T19:07:44.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 133143986182' 2021-01-15T19:07:44.162 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.2 seq 133143986182 2021-01-15T19:07:44.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:07:44.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986182 -lt 133143986182 2021-01-15T19:07:44.551 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:07:44.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:44.552 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:45.083 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T19:07:45.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:07:45.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:07:45.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:07:45.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:07:45.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:07:45.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:07:45.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:07:45.459 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T19:07:45.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:07:45.460 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:07:45.462 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:07:45.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T19:07:45.967 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:07:45.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:07:45.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:445: TEST_rep_read_unfound: sleep 5 2021-01-15T19:07:45.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:443: TEST_rep_read_unfound: timeout 30 rados -p test-pool get myobject td/osd-rep-recov-eio.sh/tmp 2021-01-15T19:07:50.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:447: TEST_rep_read_unfound: flush_pg_stats 2021-01-15T19:07:50.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:50.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:51.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:51.376 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:51.376 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:51.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:51.377 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:51.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803795 2021-01-15T19:07:51.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803795 2021-01-15T19:07:51.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795' 2021-01-15T19:07:51.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:51.526 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=150323855367 2021-01-15T19:07:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 150323855367 2021-01-15T19:07:51.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-150323855367' 2021-01-15T19:07:51.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:51.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986186 2021-01-15T19:07:51.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986186 2021-01-15T19:07:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803795 1-150323855367 2-133143986186' 2021-01-15T19:07:51.793 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:51.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803795 2021-01-15T19:07:51.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:51.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:51.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803795 2021-01-15T19:07:51.795 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:51.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803795 2021-01-15T19:07:51.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803795' 2021-01-15T19:07:51.796 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:51.796 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803795 2021-01-15T19:07:52.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803793 -lt 25769803795 2021-01-15T19:07:52.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:07:53.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:07:53.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:53.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803795 -lt 25769803795 2021-01-15T19:07:53.580 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-150323855367 2021-01-15T19:07:53.581 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:53.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:53.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-150323855367 2021-01-15T19:07:53.582 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:53.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=150323855367 2021-01-15T19:07:53.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 150323855367' 2021-01-15T19:07:53.583 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:07:53.584 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 150323855367 2021-01-15T19:07:53.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 150323855367 -lt 150323855367 2021-01-15T19:07:53.996 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:53.998 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-133143986186 2021-01-15T19:07:53.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:53.999 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:07:53.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986186 2021-01-15T19:07:53.999 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986186 2021-01-15T19:07:54.000 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 133143986186' 2021-01-15T19:07:54.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:07:54.001 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 133143986186 2021-01-15T19:07:54.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986186 -lt 133143986186 2021-01-15T19:07:54.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2021-01-15T19:07:54.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: jq . 2021-01-15T19:07:54.776 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:07:54.786 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T19:07:54.787 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_ready": true, 2021-01-15T19:07:54.787 INFO:tasks.workunit.client.0.smithi145.stdout: "pg_stats": [ 2021-01-15T19:07:54.788 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:07:54.788 INFO:tasks.workunit.client.0.smithi145.stdout: "pgid": "2.0", 2021-01-15T19:07:54.789 INFO:tasks.workunit.client.0.smithi145.stdout: "version": "28'1", 2021-01-15T19:07:54.789 INFO:tasks.workunit.client.0.smithi145.stdout: "reported_seq": "52", 2021-01-15T19:07:54.790 INFO:tasks.workunit.client.0.smithi145.stdout: "reported_epoch": "36", 2021-01-15T19:07:54.790 INFO:tasks.workunit.client.0.smithi145.stdout: "state": "active+recovery_unfound+degraded", 2021-01-15T19:07:54.791 INFO:tasks.workunit.client.0.smithi145.stdout: "last_fresh": "2021-01-15T19:07:45.993136+0000", 2021-01-15T19:07:54.791 INFO:tasks.workunit.client.0.smithi145.stdout: "last_change": "2021-01-15T19:07:46.194176+0000", 2021-01-15T19:07:54.791 INFO:tasks.workunit.client.0.smithi145.stdout: "last_active": "2021-01-15T19:07:45.993136+0000", 2021-01-15T19:07:54.792 INFO:tasks.workunit.client.0.smithi145.stdout: "last_peered": "2021-01-15T19:07:45.993136+0000", 2021-01-15T19:07:54.792 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean": "2021-01-15T19:07:41.535412+0000", 2021-01-15T19:07:54.793 INFO:tasks.workunit.client.0.smithi145.stdout: "last_became_active": "2021-01-15T19:07:41.443665+0000", 2021-01-15T19:07:54.793 INFO:tasks.workunit.client.0.smithi145.stdout: "last_became_peered": "2021-01-15T19:07:41.443665+0000", 2021-01-15T19:07:54.793 INFO:tasks.workunit.client.0.smithi145.stdout: "last_unstale": "2021-01-15T19:07:45.993136+0000", 2021-01-15T19:07:54.794 INFO:tasks.workunit.client.0.smithi145.stdout: "last_undegraded": "2021-01-15T19:07:45.992773+0000", 2021-01-15T19:07:54.794 INFO:tasks.workunit.client.0.smithi145.stdout: "last_fullsized": "2021-01-15T19:07:45.993136+0000", 2021-01-15T19:07:54.795 INFO:tasks.workunit.client.0.smithi145.stdout: "mapping_epoch": 35, 2021-01-15T19:07:54.795 INFO:tasks.workunit.client.0.smithi145.stdout: "log_start": "0'0", 2021-01-15T19:07:54.796 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_start": "0'0", 2021-01-15T19:07:54.796 INFO:tasks.workunit.client.0.smithi145.stdout: "created": 24, 2021-01-15T19:07:54.797 INFO:tasks.workunit.client.0.smithi145.stdout: "last_epoch_clean": 36, 2021-01-15T19:07:54.797 INFO:tasks.workunit.client.0.smithi145.stdout: "parent": "0.0", 2021-01-15T19:07:54.798 INFO:tasks.workunit.client.0.smithi145.stdout: "parent_split_bits": 0, 2021-01-15T19:07:54.798 INFO:tasks.workunit.client.0.smithi145.stdout: "last_scrub": "0'0", 2021-01-15T19:07:54.798 INFO:tasks.workunit.client.0.smithi145.stdout: "last_scrub_stamp": "2021-01-15T19:07:24.786082+0000", 2021-01-15T19:07:54.799 INFO:tasks.workunit.client.0.smithi145.stdout: "last_deep_scrub": "0'0", 2021-01-15T19:07:54.799 INFO:tasks.workunit.client.0.smithi145.stdout: "last_deep_scrub_stamp": "2021-01-15T19:07:24.786082+0000", 2021-01-15T19:07:54.800 INFO:tasks.workunit.client.0.smithi145.stdout: "last_clean_scrub_stamp": "2021-01-15T19:07:24.786082+0000", 2021-01-15T19:07:54.800 INFO:tasks.workunit.client.0.smithi145.stdout: "log_size": 1, 2021-01-15T19:07:54.801 INFO:tasks.workunit.client.0.smithi145.stdout: "ondisk_log_size": 1, 2021-01-15T19:07:54.801 INFO:tasks.workunit.client.0.smithi145.stdout: "stats_invalid": false, 2021-01-15T19:07:54.801 INFO:tasks.workunit.client.0.smithi145.stdout: "dirty_stats_invalid": false, 2021-01-15T19:07:54.802 INFO:tasks.workunit.client.0.smithi145.stdout: "omap_stats_invalid": false, 2021-01-15T19:07:54.802 INFO:tasks.workunit.client.0.smithi145.stdout: "hitset_stats_invalid": false, 2021-01-15T19:07:54.803 INFO:tasks.workunit.client.0.smithi145.stdout: "hitset_bytes_stats_invalid": false, 2021-01-15T19:07:54.803 INFO:tasks.workunit.client.0.smithi145.stdout: "pin_stats_invalid": false, 2021-01-15T19:07:54.803 INFO:tasks.workunit.client.0.smithi145.stdout: "manifest_stats_invalid": false, 2021-01-15T19:07:54.804 INFO:tasks.workunit.client.0.smithi145.stdout: "snaptrimq_len": 0, 2021-01-15T19:07:54.804 INFO:tasks.workunit.client.0.smithi145.stdout: "stat_sum": { 2021-01-15T19:07:54.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes": 8192, 2021-01-15T19:07:54.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects": 1, 2021-01-15T19:07:54.805 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_clones": 0, 2021-01-15T19:07:54.806 INFO:tasks.workunit.client.0.smithi145.stdout: "num_object_copies": 2, 2021-01-15T19:07:54.806 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing_on_primary": 1, 2021-01-15T19:07:54.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_missing": 0, 2021-01-15T19:07:54.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_degraded": 2, 2021-01-15T19:07:54.807 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_misplaced": 0, 2021-01-15T19:07:54.808 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_unfound": 1, 2021-01-15T19:07:54.808 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_dirty": 1, 2021-01-15T19:07:54.808 INFO:tasks.workunit.client.0.smithi145.stdout: "num_whiteouts": 0, 2021-01-15T19:07:54.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read": 0, 2021-01-15T19:07:54.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_read_kb": 0, 2021-01-15T19:07:54.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write": 1, 2021-01-15T19:07:54.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_write_kb": 8, 2021-01-15T19:07:54.809 INFO:tasks.workunit.client.0.smithi145.stdout: "num_scrub_errors": 0, 2021-01-15T19:07:54.810 INFO:tasks.workunit.client.0.smithi145.stdout: "num_shallow_scrub_errors": 0, 2021-01-15T19:07:54.810 INFO:tasks.workunit.client.0.smithi145.stdout: "num_deep_scrub_errors": 0, 2021-01-15T19:07:54.810 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_recovered": 0, 2021-01-15T19:07:54.810 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_recovered": 0, 2021-01-15T19:07:54.810 INFO:tasks.workunit.client.0.smithi145.stdout: "num_keys_recovered": 0, 2021-01-15T19:07:54.811 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_omap": 0, 2021-01-15T19:07:54.811 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_hit_set_archive": 0, 2021-01-15T19:07:54.811 INFO:tasks.workunit.client.0.smithi145.stdout: "num_bytes_hit_set_archive": 0, 2021-01-15T19:07:54.811 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush": 0, 2021-01-15T19:07:54.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_kb": 0, 2021-01-15T19:07:54.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict": 0, 2021-01-15T19:07:54.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_kb": 0, 2021-01-15T19:07:54.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_promote": 0, 2021-01-15T19:07:54.812 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_high": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_flush_mode_low": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_some": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_evict_mode_full": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_pinned": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_legacy_snapsets": 0, 2021-01-15T19:07:54.813 INFO:tasks.workunit.client.0.smithi145.stdout: "num_large_omap_objects": 0, 2021-01-15T19:07:54.814 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_manifest": 0, 2021-01-15T19:07:54.814 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_bytes": 0, 2021-01-15T19:07:54.814 INFO:tasks.workunit.client.0.smithi145.stdout: "num_omap_keys": 0, 2021-01-15T19:07:54.814 INFO:tasks.workunit.client.0.smithi145.stdout: "num_objects_repaired": 0 2021-01-15T19:07:54.815 INFO:tasks.workunit.client.0.smithi145.stdout: }, 2021-01-15T19:07:54.815 INFO:tasks.workunit.client.0.smithi145.stdout: "up": [ 2021-01-15T19:07:54.815 INFO:tasks.workunit.client.0.smithi145.stdout: 2, 2021-01-15T19:07:54.815 INFO:tasks.workunit.client.0.smithi145.stdout: 1 2021-01-15T19:07:54.815 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:07:54.816 INFO:tasks.workunit.client.0.smithi145.stdout: "acting": [ 2021-01-15T19:07:54.816 INFO:tasks.workunit.client.0.smithi145.stdout: 2, 2021-01-15T19:07:54.816 INFO:tasks.workunit.client.0.smithi145.stdout: 1 2021-01-15T19:07:54.816 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:07:54.816 INFO:tasks.workunit.client.0.smithi145.stdout: "avail_no_missing": [], 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: "object_location_counts": [ 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: { 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: "shards": "", 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: "objects": 1 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:07:54.817 INFO:tasks.workunit.client.0.smithi145.stdout: ], 2021-01-15T19:07:54.818 INFO:tasks.workunit.client.0.smithi145.stdout: "blocked_by": [], 2021-01-15T19:07:54.818 INFO:tasks.workunit.client.0.smithi145.stdout: "up_primary": 2, 2021-01-15T19:07:54.818 INFO:tasks.workunit.client.0.smithi145.stdout: "acting_primary": 2, 2021-01-15T19:07:54.818 INFO:tasks.workunit.client.0.smithi145.stdout: "purged_snaps": [] 2021-01-15T19:07:54.818 INFO:tasks.workunit.client.0.smithi145.stdout: } 2021-01-15T19:07:54.819 INFO:tasks.workunit.client.0.smithi145.stdout: ] 2021-01-15T19:07:54.819 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T19:07:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2021-01-15T19:07:54.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: jq '.pg_stats | .[0].state' 2021-01-15T19:07:54.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: grep -q recovery_unfound 2021-01-15T19:07:55.152 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:07:55.165 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:456: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:55.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:55.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: objectstore_tool: shift 2021-01-15T19:07:55.166 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: objectstore_tool: local id=1 2021-01-15T19:07:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: objectstore_tool: shift 2021-01-15T19:07:55.167 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: _objectstore_tool_nowait: shift 2021-01-15T19:07:55.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: _objectstore_tool_nowait: local id=1 2021-01-15T19:07:55.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: _objectstore_tool_nowait: shift 2021-01-15T19:07:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2021-01-15T19:07:55.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:07:55.172 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:07:55.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:07:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:07:55.173 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:07:55.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:07:55.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:55.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:55.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: shift 2021-01-15T19:07:55.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: local id=1 2021-01-15T19:07:55.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: shift 2021-01-15T19:07:55.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:55.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:07:56.279 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1215: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2021-01-15T19:07:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:07:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:07:56.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=1 2021-01-15T19:07:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:07:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:07:56.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:07:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:07:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:07:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:07:56.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:07:56.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:07:56.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:07:56.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:07:56.284 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:07:56.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:07:56.285 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:07:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:07:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:07:56.285 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:07:56.286 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:07:56.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:07:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:07:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:07:56.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:07:56.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:07:56.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:56.288 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:07:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:07:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:07:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.1 2021-01-15T19:07:56.289 INFO:tasks.workunit.client.0.smithi145.stderr:start osd.1 2021-01-15T19:07:56.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:07:56.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2021-01-15T19:07:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 1 = 1 ']' 2021-01-15T19:07:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:07:56.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:07:56.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:07:56.305 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.303+0000 7f59d587e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.306+0000 7f59d587e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:56.307 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.307+0000 7f59d587e080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:07:56.309 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.308+0000 7f59d587e080 -1 Falling back to public interface 2021-01-15T19:07:56.570 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.569+0000 7f59d587e080 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2021-01-15T19:07:56.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.569+0000 7f59d587e080 -1 journal do_read_entry(733184): bad header magic 2021-01-15T19:07:56.571 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.569+0000 7f59d587e080 -1 journal do_read_entry(733184): bad header magic 2021-01-15T19:07:56.576 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:07:56.575+0000 7f59d587e080 -1 osd.1 36 log_to_monitors {default=true} 2021-01-15T19:07:56.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 1 2021-01-15T19:07:56.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:07:56.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:07:56.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:07:56.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:07:56.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:07:56.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:07:56.702 INFO:tasks.workunit.client.0.smithi145.stderr:0 2021-01-15T19:07:56.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:07:56.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:07:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 up in weight 1 up_from 39 up_thru 29 down_at 37 last_clean_interval [35,36) [v2:127.0.0.1:6810/981607,v1:127.0.0.1:6811/981607] [v2:127.0.0.1:6812/981607,v1:127.0.0.1:6813/981607] exists,up 51f30012-bdd0-405b-bd96-b0d9b62963a9 2021-01-15T19:07:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:07:57.094 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:07:57.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:07:57.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: objectstore_tool: wait_for_clean 2021-01-15T19:07:57.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:07:57.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:07:57.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:07:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:07:57.096 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:07:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:07:57.096 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:07:57.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:07:57.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:07:57.097 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:07:57.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:07:57.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:07:57.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:07:57.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:07:57.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:07:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:07:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:07:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:07:57.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:07:57.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:57.620 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:07:57.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803798 2021-01-15T19:07:57.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803798 2021-01-15T19:07:57.754 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803798' 2021-01-15T19:07:57.755 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:57.755 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:07:57.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=167503724547 2021-01-15T19:07:57.887 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 167503724547 2021-01-15T19:07:57.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803798 1-167503724547' 2021-01-15T19:07:57.888 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:07:57.888 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:07:58.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=133143986189 2021-01-15T19:07:58.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 133143986189 2021-01-15T19:07:58.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803798 1-167503724547 2-133143986189' 2021-01-15T19:07:58.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:58.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803798 2021-01-15T19:07:58.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:58.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:07:58.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803798 2021-01-15T19:07:58.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:58.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803798 2021-01-15T19:07:58.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2021-01-15T19:07:58.024 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.0 seq 25769803798 2021-01-15T19:07:58.024 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:58.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803796 -lt 25769803798 2021-01-15T19:07:58.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:07:59.412 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:07:59.413 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:07:59.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803798 -lt 25769803798 2021-01-15T19:07:59.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:07:59.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-167503724547 2021-01-15T19:07:59.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:07:59.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:07:59.812 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-167503724547 2021-01-15T19:07:59.813 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:07:59.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=167503724547 2021-01-15T19:07:59.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 167503724547' 2021-01-15T19:07:59.814 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.1 seq 167503724547 2021-01-15T19:07:59.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:08:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 167503724547 -lt 167503724547 2021-01-15T19:08:00.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:00.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-133143986189 2021-01-15T19:08:00.211 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:00.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:08:00.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-133143986189 2021-01-15T19:08:00.212 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:00.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=133143986189 2021-01-15T19:08:00.213 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 133143986189' 2021-01-15T19:08:00.214 INFO:tasks.workunit.client.0.smithi145.stderr:waiting osd.2 seq 133143986189 2021-01-15T19:08:00.214 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:08:00.639 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 133143986189 -lt 133143986189 2021-01-15T19:08:00.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:08:00.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:00.640 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:01.153 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T19:08:01.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:08:01.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:08:01.154 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:08:01.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:08:01.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:08:01.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:08:01.155 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:08:01.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T19:08:01.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:08:01.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:01.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:02.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T19:08:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:08:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:08:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:458: TEST_rep_read_unfound: wait 2021-01-15T19:08:02.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:460: TEST_rep_read_unfound: cmp td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/tmp 2021-01-15T19:08:02.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:08:02.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:08:02.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:08:02.056 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:08:02.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:08:02.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:08:02.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:08:02.161 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:08:02.162 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:08:02.162 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:08:02.163 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:08:02.163 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:08:02.164 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:08:02.164 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:08:02.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:08:02.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:08:02.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:08:02.170 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:08:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:08:02.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:08:02.172 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.177 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:08:02.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:08:02.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:08:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:08:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2021-01-15T19:08:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:08:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:08:02.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:08:02.182 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:08:02.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:08:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:08:02.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:08:02.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:08:02.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:08:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:08:02.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:08:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:08:02.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:08:02.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:08:02.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:08:02.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:08:02.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:08:02.186 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:08:02.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:08:02.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:08:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:08:02.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:08:02.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:08:02.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:08:02.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:08:02.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.937479 2021-01-15T19:08:02.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:08:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T19:08:02.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:08:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2021-01-15T19:08:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2021-01-15T19:08:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:08:02.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:08:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:08:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2021-01-15T19:08:02.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2021-01-15T19:08:02.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:08:02.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:02.512 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:02.513 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:02.513 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.513 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.513 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:02.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon_osd_warn_num_repaired=10 2021-01-15T19:08:02.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:08:02.690 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:08:02.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:08:02.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:08:02.691 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:08:02.692 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:08:02.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:08:02.692 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:08:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:08:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:02.693 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.694 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:08:02.694 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:08:02.695 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get fsid 2021-01-15T19:08:02.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:08:02.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:08:02.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:08:02.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:08:02.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:08:02.794 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:08:02.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:08:02.795 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:08:02.796 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:02.796 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:02.798 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:02.799 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-mon.a.asok 2021-01-15T19:08:02.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:08:02.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.937479/ceph-mon.a.asok config get mon_host 2021-01-15T19:08:02.889 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2021-01-15T19:08:02.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:02.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:08:02.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:08:02.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:08:02.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2021-01-15T19:08:02.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:08:03.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:08:03.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:03.398 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:03.398 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:03.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:03.399 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:03.399 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:03.400 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:08:03.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:08:03.421 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2021-01-15T19:08:03.740 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:08:03.750 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2021-01-15T19:08:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2021-01-15T19:08:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2021-01-15T19:08:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2021-01-15T19:08:03.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2021-01-15T19:08:03.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2021-01-15T19:08:03.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2021-01-15T19:08:03.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2021-01-15T19:08:03.752 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2021-01-15T19:08:03.756 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2021-01-15T19:08:03.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:08:03.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2021-01-15T19:08:03.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:03.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:08:03.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:08:03.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:08:03.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:08:03.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:08:03.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:08:03.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:08:03.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:08:03.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:08:03.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:08:03.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:08:03.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:08:03.760 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:08:03.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:08:03.760 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:03.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:03.761 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:03.761 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:03.761 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:03.761 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:08:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:08:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:08:03.762 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:08:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:08:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:08:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:08:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:03.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:08:03.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:08:03.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:08:03.764 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:08:03.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=0caae53a-3b23-417b-94f1-b7e9e8452426 2021-01-15T19:08:03.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 0caae53a-3b23-417b-94f1-b7e9e8452426' 2021-01-15T19:08:03.765 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:08:03.765 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 0caae53a-3b23-417b-94f1-b7e9e8452426 2021-01-15T19:08:03.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAT6AFgtqLzLRAAH2+ojpC3tAPy2Szv58fJCw== 2021-01-15T19:08:03.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAT6AFgtqLzLRAAH2+ojpC3tAPy2Szv58fJCw=="}' 2021-01-15T19:08:03.772 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 0caae53a-3b23-417b-94f1-b7e9e8452426 -i td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:08:04.115 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:08:04.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2021-01-15T19:08:04.127 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAT6AFgtqLzLRAAH2+ojpC3tAPy2Szv58fJCw== --osd-uuid 0caae53a-3b23-417b-94f1-b7e9e8452426 2021-01-15T19:08:04.151 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.150+0000 7f2b7d5e1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:04.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.152+0000 7f2b7d5e1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:04.156 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.154+0000 7f2b7d5e1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:04.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.155+0000 7f2b7d5e1080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:08:04.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.155+0000 7f2b7d5e1080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:08:04.157 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.155+0000 7f2b7d5e1080 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2021-01-15T19:08:04.187 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:04.186+0000 7f2b7d5e1080 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2021-01-15T19:08:08.890 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2021-01-15T19:08:08.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:08:08.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:08:08.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:08:08.893 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:08:09.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:08:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:08:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:08:09.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:08:09.419 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:08:09.441 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:09.439+0000 7fdafbdb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:09.461 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:09.459+0000 7fdafbdb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:09.463 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:09.461+0000 7fdafbdb8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:09.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:08:09.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:08:09.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:08:09.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:08:09.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:08:09.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:09.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:08:09.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:09.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:08:09.811 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:08:09.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:09.979+0000 7fdafbdb8080 -1 Falling back to public interface 2021-01-15T19:08:10.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:08:10.762 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:10.761+0000 7fdafbdb8080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:08:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:08:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:11.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:08:11.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:11.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:08:11.211 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:08:11.655 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/983227,v1:127.0.0.1:6803/983227] [v2:127.0.0.1:6804/983227,v1:127.0.0.1:6805/983227] exists,up 0caae53a-3b23-417b-94f1-b7e9e8452426 2021-01-15T19:08:11.656 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 e7ba8f10-b32a-47c8-9206-c8cdd5555e54 2021-01-15T19:08:11.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:08:11.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:08:11.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:08:11.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:08:11.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2021-01-15T19:08:11.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:11.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:08:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:08:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:08:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2021-01-15T19:08:11.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:08:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:08:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:08:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:08:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2021-01-15T19:08:11.659 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2021-01-15T19:08:11.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:08:11.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:08:11.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:08:11.660 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:08:11.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:11.661 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:11.661 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:11.661 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:11.662 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:11.662 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:11.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:11.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:08:11.662 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:08:11.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:08:11.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:08:11.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:08:11.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:08:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:08:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:08:11.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:08:11.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2021-01-15T19:08:11.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:08:11.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=e7ba8f10-b32a-47c8-9206-c8cdd5555e54 2021-01-15T19:08:11.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 e7ba8f10-b32a-47c8-9206-c8cdd5555e54' 2021-01-15T19:08:11.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:08:11.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAb6AFgv8KUJxAAVjgeK9qIow04RdXThBOEpA== 2021-01-15T19:08:11.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAb6AFgv8KUJxAAVjgeK9qIow04RdXThBOEpA=="}' 2021-01-15T19:08:11.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new e7ba8f10-b32a-47c8-9206-c8cdd5555e54 -i td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:08:12.275 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:08:12.287 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2021-01-15T19:08:12.289 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAb6AFgv8KUJxAAVjgeK9qIow04RdXThBOEpA== --osd-uuid e7ba8f10-b32a-47c8-9206-c8cdd5555e54 2021-01-15T19:08:12.314 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.312+0000 7f9c973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:12.315 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.314+0000 7f9c973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:12.317 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.316+0000 7f9c973a8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:12.317 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.316+0000 7f9c973a8080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:08:12.318 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.316+0000 7f9c973a8080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:08:12.319 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.316+0000 7f9c973a8080 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2021-01-15T19:08:12.361 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:12.360+0000 7f9c973a8080 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2021-01-15T19:08:17.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2021-01-15T19:08:17.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:08:17.079 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:08:17.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:08:17.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:08:17.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:08:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:08:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:08:17.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:08:17.605 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:08:17.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:17.626+0000 7f64e8a99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:17.633 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:17.627+0000 7f64e8a99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:17.634 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:17.628+0000 7f64e8a99080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:18.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:08:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:08:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:08:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:08:18.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:08:18.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:18.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:08:18.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:18.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:08:18.018 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:08:18.153 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:18.152+0000 7f64e8a99080 -1 Falling back to public interface 2021-01-15T19:08:18.405 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:08:19.002 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:19.001+0000 7f64e8a99080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:08:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:08:19.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:19.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:08:19.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:19.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:08:19.407 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:08:19.813 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/984491,v1:127.0.0.1:6811/984491] [v2:127.0.0.1:6812/984491,v1:127.0.0.1:6813/984491] exists,up e7ba8f10-b32a-47c8-9206-c8cdd5555e54 2021-01-15T19:08:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:08:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:08:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:08:19.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2021-01-15T19:08:19.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2021-01-15T19:08:19.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:19.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:08:19.815 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T19:08:19.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:08:19.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2021-01-15T19:08:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:08:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:08:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:08:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:08:19.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2021-01-15T19:08:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2021-01-15T19:08:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:08:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:08:19.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:08:19.818 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:08:19.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:19.819 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:19.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:19.819 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:19.820 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:08:19.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:08:19.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:08:19.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:08:19.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:08:19.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:08:19.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:08:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:08:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:08:19.822 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2021-01-15T19:08:19.822 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:08:19.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=f9ce3978-fc58-4e85-a874-6bcfd6e9ead5 2021-01-15T19:08:19.823 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 f9ce3978-fc58-4e85-a874-6bcfd6e9ead5' 2021-01-15T19:08:19.824 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:08:19.824 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 f9ce3978-fc58-4e85-a874-6bcfd6e9ead5 2021-01-15T19:08:19.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQAj6AFgh89eMRAAcHmbAoWVXCnSN81UOFZ+/g== 2021-01-15T19:08:19.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQAj6AFgh89eMRAAcHmbAoWVXCnSN81UOFZ+/g=="}' 2021-01-15T19:08:19.829 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new f9ce3978-fc58-4e85-a874-6bcfd6e9ead5 -i td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:08:20.689 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:08:20.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2021-01-15T19:08:20.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAj6AFgh89eMRAAcHmbAoWVXCnSN81UOFZ+/g== --osd-uuid f9ce3978-fc58-4e85-a874-6bcfd6e9ead5 2021-01-15T19:08:20.727 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.725+0000 7f1d0a265080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:20.728 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.727+0000 7f1d0a265080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:20.729 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.728+0000 7f1d0a265080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:20.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.728+0000 7f1d0a265080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:08:20.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.728+0000 7f1d0a265080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:08:20.730 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.728+0000 7f1d0a265080 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2021-01-15T19:08:20.770 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:20.769+0000 7f1d0a265080 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2021-01-15T19:08:25.561 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2021-01-15T19:08:25.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:08:25.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T19:08:25.563 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:08:25.563 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T19:08:26.085 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:08:26.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T19:08:26.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:26.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:08:26.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:08:26.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:08:26.107 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:26.106+0000 7f188e6b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:26.109 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:26.108+0000 7f188e6b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:26.111 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:26.110+0000 7f188e6b9080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:26.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T19:08:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:08:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:08:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:08:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:08:26.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:26.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:08:26.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:26.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:08:26.482 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:08:26.627 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:26.626+0000 7f188e6b9080 -1 Falling back to public interface 2021-01-15T19:08:26.902 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:08:27.470 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:27.469+0000 7f188e6b9080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T19:08:27.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:08:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:08:27.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:27.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:08:27.905 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:08:28.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:08:29.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:08:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 2 2021-01-15T19:08:29.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:29.341 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:08:29.341 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:08:29.837 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/985755,v1:127.0.0.1:6819/985755] [v2:127.0.0.1:6820/985755,v1:127.0.0.1:6821/985755] exists,up f9ce3978-fc58-4e85-a874-6bcfd6e9ead5 2021-01-15T19:08:29.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:08:29.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:08:29.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:08:29.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2021-01-15T19:08:29.838 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:08:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:08:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:08:29.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:08:29.839 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:08:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:08:29.840 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:08:29.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:08:29.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:08:29.840 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:08:29.927 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:08:29.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:08:29.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:08:29.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:08:29.928 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:08:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:08:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:08:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:08:30.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:08:30.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:30.318 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:08:30.477 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803782 2021-01-15T19:08:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803782 2021-01-15T19:08:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782' 2021-01-15T19:08:30.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:30.478 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:08:30.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607557 2021-01-15T19:08:30.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607557 2021-01-15T19:08:30.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782 1-51539607557' 2021-01-15T19:08:30.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:30.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:08:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378628 2021-01-15T19:08:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378628 2021-01-15T19:08:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803782 1-51539607557 2-81604378628' 2021-01-15T19:08:30.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803782 2021-01-15T19:08:30.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:30.767 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:08:30.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803782 2021-01-15T19:08:30.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:30.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803782 2021-01-15T19:08:30.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803782' 2021-01-15T19:08:30.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:08:30.768 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803782 2021-01-15T19:08:31.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803780 -lt 25769803782 2021-01-15T19:08:31.154 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:08:32.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:08:32.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:08:32.557 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803782 -lt 25769803782 2021-01-15T19:08:32.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:32.558 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607557 2021-01-15T19:08:32.559 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:32.560 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:08:32.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607557 2021-01-15T19:08:32.561 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:32.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607557 2021-01-15T19:08:32.562 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607557' 2021-01-15T19:08:32.563 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:08:32.563 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607557 2021-01-15T19:08:32.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607557 -lt 51539607557 2021-01-15T19:08:32.966 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:32.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378628 2021-01-15T19:08:32.968 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:32.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:08:32.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378628 2021-01-15T19:08:32.969 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:32.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378628 2021-01-15T19:08:32.970 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378628' 2021-01-15T19:08:32.970 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:08:32.970 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378628 2021-01-15T19:08:33.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378628 -lt 81604378628 2021-01-15T19:08:33.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:08:33.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:33.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:33.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 8 == 0 2021-01-15T19:08:33.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:08:33.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:08:33.908 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:08:33.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:08:33.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:08:33.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:08:33.909 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:08:34.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=8 2021-01-15T19:08:34.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:08:34.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:34.287 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:34.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 8 = 8 2021-01-15T19:08:34.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:08:34.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:08:34.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2021-01-15T19:08:34.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2021-01-15T19:08:34.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create test-pool 1 1 2021-01-15T19:08:35.350 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test-pool' already exists 2021-01-15T19:08:35.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T19:08:36.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2021-01-15T19:08:36.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:08:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:08:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:08:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:08:36.363 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:08:36.363 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:08:36.364 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:08:36.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:08:36.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:08:36.364 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:08:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:08:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:08:36.455 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:08:36.456 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:08:36.456 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:08:36.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:08:36.851 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:08:36.852 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:08:36.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:08:36.852 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:36.852 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:08:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=25769803785 2021-01-15T19:08:37.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 25769803785 2021-01-15T19:08:37.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785' 2021-01-15T19:08:37.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:37.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:08:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607560 2021-01-15T19:08:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607560 2021-01-15T19:08:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607560' 2021-01-15T19:08:37.157 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:08:37.158 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:08:37.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378631 2021-01-15T19:08:37.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378631 2021-01-15T19:08:37.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-25769803785 1-51539607560 2-81604378631' 2021-01-15T19:08:37.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:37.306 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-25769803785 2021-01-15T19:08:37.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:37.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:08:37.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-25769803785 2021-01-15T19:08:37.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:37.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=25769803785 2021-01-15T19:08:37.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2021-01-15T19:08:37.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:08:37.309 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 25769803785 2021-01-15T19:08:37.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 25769803786 -lt 25769803785 2021-01-15T19:08:37.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:37.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607560 2021-01-15T19:08:37.702 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:37.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:08:37.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607560 2021-01-15T19:08:37.703 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:37.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607560 2021-01-15T19:08:37.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607560' 2021-01-15T19:08:37.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:08:37.705 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607560 2021-01-15T19:08:38.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607560 -lt 51539607560 2021-01-15T19:08:38.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:08:38.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378631 2021-01-15T19:08:38.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:08:38.110 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:08:38.110 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378631 2021-01-15T19:08:38.111 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:08:38.111 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378631 2021-01-15T19:08:38.112 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378631' 2021-01-15T19:08:38.112 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:08:38.112 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378631 2021-01-15T19:08:38.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378629 -lt 81604378631 2021-01-15T19:08:38.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:08:39.501 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:08:39.502 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:08:39.903 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378632 -lt 81604378631 2021-01-15T19:08:39.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:08:39.904 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:39.905 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:40.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:08:40.437 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:08:40.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:08:40.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:08:40.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:08:40.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:08:40.438 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:08:40.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:08:40.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:08:40.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:08:40.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:08:40.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:08:41.325 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:08:41.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:08:41.326 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:08:41.327 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2021-01-15T19:08:41.693 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:08:41.694 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475661+0000 0'0 24:27 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.694 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475518+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.694 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.631128+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.695 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475689+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.695 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.662729+0000 0'0 23:35 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.695 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:36.102542+0000 0'0 24:10 [2,1,0] 2 [2,1,0] 2 0'0 2021-01-15T19:08:35.136607+0000 0'0 2021-01-15T19:08:35.136607+0000 0 2021-01-15T19:08:41.696 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475953+0000 0'0 24:27 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.696 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475216+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.696 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.662688+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:41.697 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:08:41.697 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:08:41.698 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:08:41.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2021-01-15T19:08:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2021-01-15T19:08:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2021-01-15T19:08:41.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:08:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2021-01-15T19:08:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:08:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2021-01-15T19:08:41.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:08:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2021-01-15T19:08:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2021-01-15T19:08:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2021-01-15T19:08:41.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:41.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:08:41.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2021-01-15T19:08:41.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:08:41.845 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=myobject 2021-01-15T19:08:41.845 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool myobject 2021-01-15T19:08:41.845 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:08:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:08:42.240 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:08:42.240 INFO:tasks.workunit.client.0.smithi145.stderr:0' 2021-01-15T19:08:42.240 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 0 2021-01-15T19:08:42.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2021-01-15T19:08:42.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2021-01-15T19:08:42.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2021-01-15T19:08:42.241 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:08:42.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:08:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:08:42.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:08:42.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:08:42.345 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:08:42.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2021-01-15T19:08:42.895 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already down. 2021-01-15T19:08:42.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2021-01-15T19:08:43.534 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. 2021-01-15T19:08:43.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2021-01-15T19:08:43.946 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:08:43.947 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475661+0000 0'0 24:27 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.947 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475518+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.947 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.631128+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.948 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475689+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.948 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2021-01-15T19:08:29.662729+0000 0'0 24:36 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.948 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:36.102542+0000 0'0 24:10 [2,1,0] 2 [2,1,0] 2 0'0 2021-01-15T19:08:35.136607+0000 0'0 2021-01-15T19:08:35.136607+0000 0 2021-01-15T19:08:43.949 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475953+0000 0'0 24:27 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.949 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.475216+0000 0'0 24:27 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.949 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:08:29.662688+0000 0'0 24:15 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:08:43.949 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:08:43.949 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:08:43.950 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:08:43.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2021-01-15T19:08:43.957 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records in 2021-01-15T19:08:43.958 INFO:tasks.workunit.client.0.smithi145.stderr:4+0 records out 2021-01-15T19:08:43.958 INFO:tasks.workunit.client.0.smithi145.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.4654e-05 s, 43.3 MB/s 2021-01-15T19:08:43.959 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2021-01-15T19:08:43.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:43.959 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.410 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.411 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.555 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.558 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:44.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:44.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.349 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.473 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.474 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.759 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:45.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:45.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.017 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.160 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.670 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:46.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:46.930 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.055 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.332 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.467 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.601 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:47.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:47.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.136 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.137 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:48.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:48.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.064 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.483 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.620 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.733 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.734 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:49.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:49.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.268 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.653 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:50.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:50.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.323 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.851 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:51.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:51.984 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.135 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.402 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.528 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.654 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.781 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.782 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:52.957 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:52.958 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.259 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:53.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:53.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.312 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.564 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.565 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:54.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:54.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.243 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.660 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.794 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.795 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:55.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:55.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.170 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.171 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.413 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:56.941 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:56.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:57.066 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:57.067 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:57.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:08:57.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2021-01-15T19:08:57.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2021-01-15T19:08:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:08:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:08:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:08:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:08:57.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test-pool 2021-01-15T19:08:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:08:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj75 2021-01-15T19:08:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:08:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:57.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:08:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=0 2021-01-15T19:08:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:08:57.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:08:57.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test-pool obj75 2021-01-15T19:08:57.355 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:08:57.356 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj75 2021-01-15T19:08:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool obj75 2021-01-15T19:08:57.356 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:08:57.784 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:08:57.784 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T19:08:57.785 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 2021-01-15T19:08:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:08:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=2 2021-01-15T19:08:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:08:57.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:08:57.786 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2021-01-15T19:08:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:08:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2021-01-15T19:08:57.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:08:57.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=2 2021-01-15T19:08:57.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:08:57.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:08:57.790 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:08:57.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.2 2021-01-15T19:08:57.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T19:08:57.791 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T19:08:57.791 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:57.791 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:57.792 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:57.792 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.2.asok 2021-01-15T19:08:57.792 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:08:57.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:08:57.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:08:57.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:08:57.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.2 2021-01-15T19:08:57.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2021-01-15T19:08:57.896 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2021-01-15T19:08:57.897 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:57.897 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:57.897 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:57.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.2.asok 2021-01-15T19:08:57.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:08:57.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.2.asok injectdataerr test-pool obj75 2021-01-15T19:08:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2021-01-15T19:08:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: local pooltype=rep 2021-01-15T19:08:58.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: shift 2021-01-15T19:08:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: local which=data 2021-01-15T19:08:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: shift 2021-01-15T19:08:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: local poolname=test-pool 2021-01-15T19:08:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: shift 2021-01-15T19:08:58.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: local objname=obj75 2021-01-15T19:08:58.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: shift 2021-01-15T19:08:58.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:58.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: shift 2021-01-15T19:08:58.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: local shard_id=1 2021-01-15T19:08:58.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shift 2021-01-15T19:08:58.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2021-01-15T19:08:58.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: get_osds test-pool obj75 2021-01-15T19:08:58.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local poolname=test-pool 2021-01-15T19:08:58.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: get_osds: local objectname=obj75 2021-01-15T19:08:58.009 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: ceph --format json osd map test-pool obj75 2021-01-15T19:08:58.010 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: jq '.acting | .[]' 2021-01-15T19:08:58.402 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: get_osds: local 'osds=2 2021-01-15T19:08:58.402 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T19:08:58.403 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: get_osds: echo 2 1 2021-01-15T19:08:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local -a initial_osds 2021-01-15T19:08:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: local osd_id=1 2021-01-15T19:08:58.403 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: '[' rep '!=' ec ']' 2021-01-15T19:08:58.404 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: shard_id= 2021-01-15T19:08:58.404 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2021-01-15T19:08:58.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: type=bluestore 2021-01-15T19:08:58.406 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2021-01-15T19:08:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local daemon=osd 2021-01-15T19:08:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local id=1 2021-01-15T19:08:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local config=bluestore_debug_inject_read_err 2021-01-15T19:08:58.407 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: set_config: local value=true 2021-01-15T19:08:58.407 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: jq 'has("success")' 2021-01-15T19:08:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: get_asok_path osd.1 2021-01-15T19:08:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:08:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:08:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:58.408 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:58.409 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:58.409 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:08:58.409 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.937479/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2021-01-15T19:08:58.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: set_config: test true == true 2021-01-15T19:08:58.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: local loop=0 2021-01-15T19:08:58.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: inject_eio: grep -q Invalid 2021-01-15T19:08:58.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: get_asok_path osd.1 2021-01-15T19:08:58.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-01-15T19:08:58.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-01-15T19:08:58.524 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:08:58.524 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:58.524 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:58.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.937479/ceph-osd.1.asok 2021-01-15T19:08:58.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: CEPH_ARGS= 2021-01-15T19:08:58.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: ceph --admin-daemon /tmp/ceph-asok.937479/ceph-osd.1.asok injectdataerr test-pool obj75 2021-01-15T19:08:58.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2021-01-15T19:08:58.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:08:58.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:08:58.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=0 2021-01-15T19:08:58.626 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:08:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2021-01-15T19:08:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2021-01-15T19:08:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:08:58.627 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:08:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:08:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2021-01-15T19:08:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2021-01-15T19:08:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:08:58.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:08:58.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2021-01-15T19:08:58.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:08:58.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:08:58.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:08:58.629 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:08:58.630 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:08:58.630 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:08:58.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:58.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' 2021-01-15T19:08:58.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:08:58.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2021-01-15T19:08:58.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2021-01-15T19:08:58.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:08:58.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:08:58.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:08:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:08:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2021-01-15T19:08:58.634 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.0 2021-01-15T19:08:58.635 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:08:58.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 0 --fsid=8eaa1c99-38be-4c5f-89cb-fe64e89d2e5e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.937479/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:08:58.636 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2021-01-15T19:08:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 0 = 0 ']' 2021-01-15T19:08:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:08:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:08:58.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:08:58.649 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:58.647+0000 7f06127c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:58.650 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:58.649+0000 7f06127c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:58.651 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:58.650+0000 7f06127c2080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:08:59.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 0 2021-01-15T19:08:59.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:08:59.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:08:59.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:08:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:08:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:08:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:08:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:08:59.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:08:59.023 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:08:59.169 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:08:59.168+0000 7f06127c2080 -1 Falling back to public interface 2021-01-15T19:08:59.433 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:09:00.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:09:00.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:09:00.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:09:00.435 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:09:00.436 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:09:00.436 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:09:00.467 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:00.465+0000 7f06127c2080 -1 osd.0 25 log_to_monitors {default=true} 2021-01-15T19:09:00.907 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up out weight 0 up_from 29 up_thru 19 down_at 26 last_clean_interval [6,25) [v2:127.0.0.1:6802/989325,v1:127.0.0.1:6803/989325] [v2:127.0.0.1:6804/989325,v1:127.0.0.1:6805/989325] exists,up 0caae53a-3b23-417b-94f1-b7e9e8452426 2021-01-15T19:09:00.907 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:09:00.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:09:00.908 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:09:00.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2021-01-15T19:09:02.096 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. 2021-01-15T19:09:02.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2021-01-15T19:09:17.106 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2021-01-15T19:09:17.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:17.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:17.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:17.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:17.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:17.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:17.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:17.505 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:17.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:17.507 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:18.506 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:18.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:18.507 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:18.508 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:18.509 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:18.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:18.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:18.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:18.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:18.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:18.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:18.893 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:19.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:19.894 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:19.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:19.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:19.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:19.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:20.280 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:20.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:20.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:20.282 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:21.282 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:21.283 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:21.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:21.665 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:21.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:21.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:21.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:21.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:21.667 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:22.667 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:22.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:22.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:22.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:22.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:22.669 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:23.059 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:23.060 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:23.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:23.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:23.062 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:24.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:24.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:24.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:24.062 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:24.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:24.063 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:24.451 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovering+degraded 2021-01-15T19:09:24.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovering+degraded 2021-01-15T19:09:24.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:24.452 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 1 = 0 ']' 2021-01-15T19:09:24.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:387: TEST_rep_recovery_unfound: echo 'active+recovering+degraded ' 2021-01-15T19:09:24.454 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:388: TEST_rep_recovery_unfound: sleep 1 2021-01-15T19:09:24.454 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovering+degraded 2021-01-15T19:09:25.453 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2021-01-15T19:09:25.454 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2021-01-15T19:09:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2021-01-15T19:09:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2021-01-15T19:09:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2021-01-15T19:09:25.455 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2021-01-15T19:09:25.848 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2021-01-15T19:09:25.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2021-01-15T19:09:25.849 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2021-01-15T19:09:25.850 INFO:tasks.workunit.client.0.smithi145.stdout:active+recovery_unfound+degraded 2021-01-15T19:09:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2021-01-15T19:09:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2021-01-15T19:09:25.850 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2021-01-15T19:09:26.216 INFO:tasks.workunit.client.0.smithi145.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-01-15T19:09:26.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.7 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.056359+0000 0'0 36:55 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.216 INFO:tasks.workunit.client.0.smithi145.stdout:1.6 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.805536+0000 0'0 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.1 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.773716+0000 0'0 37:44 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.805575+0000 0'0 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.217 INFO:tasks.workunit.client.0.smithi145.stdout:1.2 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.899474+0000 0'0 37:45 [0,1,2] 0 [0,1,2] 0 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.217 INFO:tasks.workunit.client.0.smithi145.stdout:2.0 101 1 3 0 1 413696 0 0 101 101 active+recovery_unfound+degraded 2021-01-15T19:09:20.061135+0000 28'101 37:339 [2,1,0] 2 [2,1,0] 2 0'0 2021-01-15T19:08:35.136607+0000 0'0 2021-01-15T19:08:35.136607+0000 0 2021-01-15T19:09:26.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.3 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:02.994672+0000 0'0 36:55 [1,2,0] 1 [1,2,0] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.4 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.805248+0000 0'0 37:56 [1,0,2] 1 [1,0,2] 1 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.218 INFO:tasks.workunit.client.0.smithi145.stdout:1.5 0 0 0 0 0 0 0 0 0 0 active+clean 2021-01-15T19:09:03.774035+0000 0'0 37:44 [2,0,1] 2 [2,0,1] 2 0'0 2021-01-15T19:08:03.604739+0000 0'0 2021-01-15T19:08:03.604739+0000 0 2021-01-15T19:09:26.219 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:09:26.219 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:09:26.220 INFO:tasks.workunit.client.0.smithi145.stderr:dumped pgs 2021-01-15T19:09:26.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2021-01-15T19:09:26.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2021-01-15T19:09:26.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:31.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2021-01-15T19:09:31.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2021-01-15T19:09:31.605 INFO:tasks.workunit.client.0.smithi145.stderr:pg has no unfound objects 2021-01-15T19:09:31.615 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2021-01-15T19:09:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:09:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:09:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:09:31.616 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:09:31.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:09:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:09:31.617 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:09:31.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:09:31.617 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:09:31.618 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:09:31.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:09:31.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:09:31.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:09:31.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:09:31.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:09:32.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:09:32.099 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:09:32.100 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:09:32.100 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:09:32.101 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:09:32.101 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:09:32.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=124554051593 2021-01-15T19:09:32.241 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 124554051593 2021-01-15T19:09:32.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-124554051593' 2021-01-15T19:09:32.242 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:09:32.242 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:09:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=51539607573 2021-01-15T19:09:32.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 51539607573 2021-01-15T19:09:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-124554051593 1-51539607573' 2021-01-15T19:09:32.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:09:32.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:09:32.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=81604378644 2021-01-15T19:09:32.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 81604378644 2021-01-15T19:09:32.514 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-124554051593 1-51539607573 2-81604378644' 2021-01-15T19:09:32.515 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:09:32.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-124554051593 2021-01-15T19:09:32.515 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:09:32.516 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:09:32.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-124554051593 2021-01-15T19:09:32.516 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:09:32.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=124554051593 2021-01-15T19:09:32.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 124554051593' 2021-01-15T19:09:32.518 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:09:32.518 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 124554051593 2021-01-15T19:09:32.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051591 -lt 124554051593 2021-01-15T19:09:32.909 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:09:33.910 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:09:33.910 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:09:34.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 124554051593 -lt 124554051593 2021-01-15T19:09:34.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:09:34.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-51539607573 2021-01-15T19:09:34.314 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:09:34.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:09:34.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-51539607573 2021-01-15T19:09:34.315 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:09:34.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=51539607573 2021-01-15T19:09:34.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2021-01-15T19:09:34.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:09:34.317 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 51539607573 2021-01-15T19:09:34.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 51539607573 -lt 51539607573 2021-01-15T19:09:34.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:09:34.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-81604378644 2021-01-15T19:09:34.739 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:09:34.740 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:09:34.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-81604378644 2021-01-15T19:09:34.740 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:09:34.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=81604378644 2021-01-15T19:09:34.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2021-01-15T19:09:34.741 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:09:34.742 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 81604378644 2021-01-15T19:09:35.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 81604378644 -lt 81604378644 2021-01-15T19:09:35.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:09:35.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:09:35.131 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:09:35.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 9 == 0 2021-01-15T19:09:35.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:09:35.650 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:09:35.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:09:35.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:09:35.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:09:35.651 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:09:35.652 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:09:36.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=9 2021-01-15T19:09:36.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:09:36.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:09:36.031 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:09:36.542 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 9 = 9 2021-01-15T19:09:36.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:09:36.543 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:09:36.543 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2021-01-15T19:09:36.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2021-01-15T19:09:36.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2021-01-15T19:09:36.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.618 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2021-01-15T19:09:36.619 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2021-01-15T19:09:36.646 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2021-01-15T19:09:36.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2021-01-15T19:09:36.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2021-01-15T19:09:36.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.751 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.752 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2021-01-15T19:09:36.753 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2021-01-15T19:09:36.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.804 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2021-01-15T19:09:36.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.831 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2021-01-15T19:09:36.832 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.858 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2021-01-15T19:09:36.859 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2021-01-15T19:09:36.886 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.911 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.912 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2021-01-15T19:09:36.913 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.938 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2021-01-15T19:09:36.939 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2021-01-15T19:09:36.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:36.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:36.991 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2021-01-15T19:09:36.992 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.018 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2021-01-15T19:09:37.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.044 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2021-01-15T19:09:37.045 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.071 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.073 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2021-01-15T19:09:37.074 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2021-01-15T19:09:37.099 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.124 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.125 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2021-01-15T19:09:37.126 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.151 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2021-01-15T19:09:37.152 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2021-01-15T19:09:37.179 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2021-01-15T19:09:37.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.232 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2021-01-15T19:09:37.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.283 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2021-01-15T19:09:37.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2021-01-15T19:09:37.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2021-01-15T19:09:37.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2021-01-15T19:09:37.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2021-01-15T19:09:37.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2021-01-15T19:09:37.418 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2021-01-15T19:09:37.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2021-01-15T19:09:37.472 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2021-01-15T19:09:37.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.544 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.545 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2021-01-15T19:09:37.546 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2021-01-15T19:09:37.572 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.598 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2021-01-15T19:09:37.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.624 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2021-01-15T19:09:37.625 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.650 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.651 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2021-01-15T19:09:37.652 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2021-01-15T19:09:37.679 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2021-01-15T19:09:37.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.730 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2021-01-15T19:09:37.731 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.756 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.757 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2021-01-15T19:09:37.758 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.783 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.784 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2021-01-15T19:09:37.785 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2021-01-15T19:09:37.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.837 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2021-01-15T19:09:37.839 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.864 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.865 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2021-01-15T19:09:37.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.891 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.892 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2021-01-15T19:09:37.893 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2021-01-15T19:09:37.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2021-01-15T19:09:37.949 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.971 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:37.972 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:37.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2021-01-15T19:09:37.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2021-01-15T19:09:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2021-01-15T19:09:38.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2021-01-15T19:09:38.053 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.078 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.079 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2021-01-15T19:09:38.080 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.105 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2021-01-15T19:09:38.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.131 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.132 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2021-01-15T19:09:38.133 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.158 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2021-01-15T19:09:38.159 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2021-01-15T19:09:38.186 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.211 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2021-01-15T19:09:38.212 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.239 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2021-01-15T19:09:38.240 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.264 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.265 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2021-01-15T19:09:38.266 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2021-01-15T19:09:38.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.336 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.337 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2021-01-15T19:09:38.338 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.363 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2021-01-15T19:09:38.365 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2021-01-15T19:09:38.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.416 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2021-01-15T19:09:38.417 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.444 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2021-01-15T19:09:38.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.470 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2021-01-15T19:09:38.471 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.496 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2021-01-15T19:09:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.524 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2021-01-15T19:09:38.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.550 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2021-01-15T19:09:38.552 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2021-01-15T19:09:38.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.603 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.604 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2021-01-15T19:09:38.605 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.629 INFO:tasks.workunit.client.0.smithi145.stderr:error getting test-pool/obj75: (2) No such file or directory 2021-01-15T19:09:38.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2021-01-15T19:09:38.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.656 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.657 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2021-01-15T19:09:38.658 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.684 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2021-01-15T19:09:38.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2021-01-15T19:09:38.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.736 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.737 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2021-01-15T19:09:38.738 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.763 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.764 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2021-01-15T19:09:38.765 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2021-01-15T19:09:38.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.820 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2021-01-15T19:09:38.821 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2021-01-15T19:09:38.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.870 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.871 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2021-01-15T19:09:38.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2021-01-15T19:09:38.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:38.924 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:38.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2021-01-15T19:09:38.925 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2021-01-15T19:09:39.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.036 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2021-01-15T19:09:39.037 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.061 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.062 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2021-01-15T19:09:39.063 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.088 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2021-01-15T19:09:39.089 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.115 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2021-01-15T19:09:39.116 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.141 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.142 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2021-01-15T19:09:39.143 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.168 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2021-01-15T19:09:39.169 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.197 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2021-01-15T19:09:39.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2021-01-15T19:09:39.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.248 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.249 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2021-01-15T19:09:39.250 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.284 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.285 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2021-01-15T19:09:39.286 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2021-01-15T19:09:39.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.339 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2021-01-15T19:09:39.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2021-01-15T19:09:39.340 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.364 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2021-01-15T19:09:39.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2021-01-15T19:09:39.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: local poolname=test-pool 2021-01-15T19:09:39.367 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2021-01-15T19:09:39.887 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'test-pool' does not exist 2021-01-15T19:09:39.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2021-01-15T19:09:39.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:09:39.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:09:39.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:09:39.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:09:39.898 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:09:39.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:09:39.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:09:39.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:09:40.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:09:40.005 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:09:40.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:09:40.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:09:40.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:09:40.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:09:40.007 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:09:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.008 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:09:40.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:09:40.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:09:40.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:09:40.010 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:09:40.021 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:09:40.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.022 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:09:40.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T19:09:40.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-rep-recov-eio.sh 0 2021-01-15T19:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2021-01-15T19:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T19:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2021-01-15T19:09:40.025 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:09:40.026 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:09:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:09:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:09:40.026 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:09:40.027 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:09:40.027 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:09:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:09:40.028 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:09:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:09:40.028 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:09:40.029 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:09:40.029 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:09:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:09:40.030 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.030 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:09:40.031 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T19:09:40.032 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2021-01-15T19:09:40.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:09:40.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.032 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.937479 2021-01-15T19:09:40.033 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.937479 2021-01-15T19:09:40.034 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:09:40.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:09:40.035 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T19:09:40.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T19:09:40.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T19:09:40.107 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2021-01-15T19:09:40.108 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2021-01-15T19:09:40.168 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T19:09:40.174 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T19:09:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T19:09:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T19:09:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T19:09:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T19:09:40.175 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T19:09:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T19:09:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T19:09:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T19:09:40.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/osd-reuse-id 2021-01-15T19:09:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2021-01-15T19:09:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2021-01-15T19:09:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2021-01-15T19:09:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2021-01-15T19:09:40.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2021-01-15T19:09:40.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2021-01-15T19:09:40.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none ' 2021-01-15T19:09:40.178 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2021-01-15T19:09:40.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2021-01-15T19:09:40.178 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T19:09:40.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2021-01-15T19:09:40.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2021-01-15T19:09:40.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: TEST_reuse_id td/osd-reuse-id 2021-01-15T19:09:40.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:35: TEST_reuse_id: local dir=td/osd-reuse-id 2021-01-15T19:09:40.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: setup td/osd-reuse-id 2021-01-15T19:09:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2021-01-15T19:09:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2021-01-15T19:09:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2021-01-15T19:09:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:09:40.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2021-01-15T19:09:40.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:09:40.183 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:09:40.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:09:40.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:09:40.183 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:09:40.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:09:40.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:09:40.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:09:40.184 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:09:40.184 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:09:40.185 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:09:40.185 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:09:40.187 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:09:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:09:40.188 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:09:40.188 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:09:40.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:09:40.189 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2021-01-15T19:09:40.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:09:40.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.189 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.992733 2021-01-15T19:09:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:09:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:09:40.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2021-01-15T19:09:40.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:09:40.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.992733 2021-01-15T19:09:40.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:09:40.191 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T19:09:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T19:09:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:09:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2021-01-15T19:09:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:38: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:09:40.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2021-01-15T19:09:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:09:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:09:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:09:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2021-01-15T19:09:40.193 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:09:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:09:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:09:40.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:09:40.481 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:09:40.481 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.481 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:40.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:40.481 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:09:40.664 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:09:40.665 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:09:40.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:09:40.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:09:40.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:09:40.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:09:40.666 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:09:40.667 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:09:40.667 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:09:40.667 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:09:40.667 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.667 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:40.668 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.992733/ceph-mon.a.asok 2021-01-15T19:09:40.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:09:40.668 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.992733/ceph-mon.a.asok config get fsid 2021-01-15T19:09:40.766 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:09:40.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:09:40.767 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:09:40.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:09:40.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:09:40.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:09:40.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:09:40.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:09:40.770 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:09:40.770 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:40.771 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:40.772 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.992733/ceph-mon.a.asok 2021-01-15T19:09:40.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:09:40.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.992733/ceph-mon.a.asok config get mon_host 2021-01-15T19:09:40.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mgr td/osd-reuse-id x 2021-01-15T19:09:40.866 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/osd-reuse-id 2021-01-15T19:09:40.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:09:40.867 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:09:40.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:09:40.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/osd-reuse-id/x 2021-01-15T19:09:40.868 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:09:41.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:09:41.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:09:41.768 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:09:41.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:09:41.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:41.769 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:41.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:41.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:09:41.770 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:09:41.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_osd td/osd-reuse-id 0 2021-01-15T19:09:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-reuse-id 2021-01-15T19:09:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:09:41.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:09:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:09:41.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-reuse-id/0 2021-01-15T19:09:41.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 ' 2021-01-15T19:09:41.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:09:41.799 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:09:41.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:09:41.801 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2021-01-15T19:09:41.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2021-01-15T19:09:41.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:09:41.802 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:09:41.803 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2021-01-15T19:09:41.803 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:09:41.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:09:41.804 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:09:41.804 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:09:41.805 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:41.805 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:41.805 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:41.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:41.806 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:09:41.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:09:41.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:09:41.807 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2021-01-15T19:09:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2021-01-15T19:09:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:09:41.808 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:09:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:09:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:09:41.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:09:41.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-reuse-id/0 2021-01-15T19:09:41.810 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:09:41.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=7648ee14-80d9-4c18-8fbe-709934f8ae13 2021-01-15T19:09:41.811 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 7648ee14-80d9-4c18-8fbe-709934f8ae13' 2021-01-15T19:09:41.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:09:41.812 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 7648ee14-80d9-4c18-8fbe-709934f8ae13 2021-01-15T19:09:41.812 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB16AFgSbL7LxAAcoeWwuUCihyWVKBN7vefGg== 2021-01-15T19:09:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB16AFgSbL7LxAAcoeWwuUCihyWVKBN7vefGg=="}' 2021-01-15T19:09:41.813 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 7648ee14-80d9-4c18-8fbe-709934f8ae13 -i td/osd-reuse-id/0/new.json 2021-01-15T19:09:42.166 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:09:42.176 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-reuse-id/0/new.json 2021-01-15T19:09:42.177 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB16AFgSbL7LxAAcoeWwuUCihyWVKBN7vefGg== --osd-uuid 7648ee14-80d9-4c18-8fbe-709934f8ae13 2021-01-15T19:09:42.197 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.196+0000 7fa16f5d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:42.198 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.198+0000 7fa16f5d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:42.200 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.199+0000 7fa16f5d3080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:42.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.200+0000 7fa16f5d3080 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2021-01-15T19:09:42.201 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.200+0000 7fa16f5d3080 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2021-01-15T19:09:42.202 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.200+0000 7fa16f5d3080 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2021-01-15T19:09:42.238 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:42.237+0000 7fa16f5d3080 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2021-01-15T19:09:46.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2021-01-15T19:09:46.922 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:09:46.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:09:46.923 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:09:46.924 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:09:47.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:09:47.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:09:47.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:09:47.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:09:47.446 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:09:47.446 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:09:47.464 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:47.463+0000 7fbba6e62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:47.465 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:47.464+0000 7fbba6e62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:47.466 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:47.465+0000 7fbba6e62080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:47.844 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:09:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:09:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:09:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:09:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:09:47.845 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:09:47.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:09:47.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:09:47.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:09:47.847 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:09:47.982 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:47.981+0000 7fbba6e62080 -1 Falling back to public interface 2021-01-15T19:09:48.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:09:48.771 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:48.770+0000 7fbba6e62080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:09:49.234 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:09:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:09:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:09:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:09:49.235 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:09:49.235 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:09:49.628 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/993772,v1:127.0.0.1:6803/993772] [v2:127.0.0.1:6804/993772,v1:127.0.0.1:6805/993772] exists,up 7648ee14-80d9-4c18-8fbe-709934f8ae13 2021-01-15T19:09:49.628 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 4544a806-2669-4268-878a-c97e9c751581 2021-01-15T19:09:49.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:09:49.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:09:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:09:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 1 2021-01-15T19:09:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-reuse-id 2021-01-15T19:09:49.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:09:49.630 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:09:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:09:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-reuse-id/1 2021-01-15T19:09:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 ' 2021-01-15T19:09:49.631 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:09:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:09:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:09:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2021-01-15T19:09:49.632 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2021-01-15T19:09:49.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:09:49.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:09:49.633 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2021-01-15T19:09:49.633 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:09:49.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:09:49.634 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:09:49.634 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:09:49.634 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:09:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:09:49.635 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:09:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:09:49.635 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:09:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:09:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2021-01-15T19:09:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2021-01-15T19:09:49.636 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:09:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:09:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:09:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:09:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:09:49.637 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-reuse-id/1 2021-01-15T19:09:49.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:09:49.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4544a806-2669-4268-878a-c97e9c751581 2021-01-15T19:09:49.638 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 4544a806-2669-4268-878a-c97e9c751581' 2021-01-15T19:09:49.638 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:09:49.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQB96AFg7h0iJhAA3+KWZrRnknWhWvzNJPEmJg== 2021-01-15T19:09:49.640 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQB96AFg7h0iJhAA3+KWZrRnknWhWvzNJPEmJg=="}' 2021-01-15T19:09:49.641 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4544a806-2669-4268-878a-c97e9c751581 -i td/osd-reuse-id/1/new.json 2021-01-15T19:09:50.137 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:09:50.149 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-reuse-id/1/new.json 2021-01-15T19:09:50.150 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB96AFg7h0iJhAA3+KWZrRnknWhWvzNJPEmJg== --osd-uuid 4544a806-2669-4268-878a-c97e9c751581 2021-01-15T19:09:50.171 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.170+0000 7f9e764bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:50.172 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.171+0000 7f9e764bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.172+0000 7f9e764bc080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.173+0000 7f9e764bc080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:09:50.175 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.173+0000 7f9e764bc080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:09:50.176 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.173+0000 7f9e764bc080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:09:50.269 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:50.268+0000 7f9e764bc080 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2021-01-15T19:09:55.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2021-01-15T19:09:55.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:09:55.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:09:55.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:09:55.052 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:09:55.573 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:09:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:09:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:09:55.573 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:09:55.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:09:55.574 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:09:55.592 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:55.591+0000 7f95f1da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:55.594 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:55.593+0000 7f95f1da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:55.595 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:55.594+0000 7f95f1da0080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:09:55.973 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:09:55.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:09:55.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:09:55.974 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:09:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:09:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:09:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:09:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:09:55.975 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:09:55.976 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:09:56.116 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:56.115+0000 7f95f1da0080 -1 Falling back to public interface 2021-01-15T19:09:56.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:09:56.927 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:09:56.926+0000 7f95f1da0080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:09:57.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:09:57.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:09:57.368 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:09:57.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:09:57.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:09:57.369 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:09:57.786 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/995036,v1:127.0.0.1:6811/995036] [v2:127.0.0.1:6812/995036,v1:127.0.0.1:6813/995036] exists,up 4544a806-2669-4268-878a-c97e9c751581 2021-01-15T19:09:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:09:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:09:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:09:57.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: create_rbd_pool 2021-01-15T19:09:57.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:534: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2021-01-15T19:09:58.213 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' does not exist 2021-01-15T19:09:58.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: create_pool rbd 4 2021-01-15T19:09:58.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: ceph osd pool create rbd 4 2021-01-15T19:09:58.806 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'rbd' already exists 2021-01-15T19:09:58.816 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: sleep 1 2021-01-15T19:09:59.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: rbd pool init rbd 2021-01-15T19:10:00.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: wait_for_clean 2021-01-15T19:10:00.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:10:00.920 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:10:00.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:10:00.921 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:10:00.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:10:00.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:10:00.922 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:10:00.922 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:10:00.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:10:00.923 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:10:01.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:10:01.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:10:01.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:10:01.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:10:01.023 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:10:01.408 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:10:01.409 INFO:tasks.workunit.client.0.smithi145.stderr:1' 2021-01-15T19:10:01.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:10:01.409 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:01.410 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:10:01.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836485 2021-01-15T19:10:01.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836485 2021-01-15T19:10:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485' 2021-01-15T19:10:01.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:01.585 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:10:01.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672963 2021-01-15T19:10:01.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672963 2021-01-15T19:10:01.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836485 1-42949672963' 2021-01-15T19:10:01.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:01.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836485 2021-01-15T19:10:01.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:01.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:10:01.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836485 2021-01-15T19:10:01.721 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:01.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836485 2021-01-15T19:10:01.723 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2021-01-15T19:10:01.723 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836485 2021-01-15T19:10:01.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:02.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836483 -lt 21474836485 2021-01-15T19:10:02.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:10:03.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:10:03.108 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:03.497 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836483 -lt 21474836485 2021-01-15T19:10:03.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:10:04.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 299 -eq 0 ']' 2021-01-15T19:10:04.499 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:04.894 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836485 2021-01-15T19:10:04.895 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:04.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672963 2021-01-15T19:10:04.895 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:04.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:10:04.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672963 2021-01-15T19:10:04.897 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:04.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672963 2021-01-15T19:10:04.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2021-01-15T19:10:04.898 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672963 2021-01-15T19:10:04.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:10:05.281 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672964 -lt 42949672963 2021-01-15T19:10:05.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:10:05.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:05.282 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:05.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 4 == 0 2021-01-15T19:10:05.798 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:10:05.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:10:05.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:10:05.799 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:10:05.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:10:05.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:10:05.800 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:10:06.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=4 2021-01-15T19:10:06.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:10:06.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:06.196 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:06.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 4 = 4 2021-01-15T19:10:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:10:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:10:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2021-01-15T19:10:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: local dir=td/osd-reuse-id 2021-01-15T19:10:06.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: local id=1 2021-01-15T19:10:06.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: ceph osd out osd.1 2021-01-15T19:10:07.283 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already out. 2021-01-15T19:10:07.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:786: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2021-01-15T19:10:07.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:10:07.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:10:07.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:10:07.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:10:07.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:10:07.399 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:10:07.400 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd down osd.1 2021-01-15T19:10:07.818 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 is already down. 2021-01-15T19:10:07.827 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2021-01-15T19:10:08.280 INFO:tasks.workunit.client.0.smithi145.stderr:osd.1 does not exist 2021-01-15T19:10:08.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: teardown td/osd-reuse-id/1 2021-01-15T19:10:08.290 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2021-01-15T19:10:08.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:10:08.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2021-01-15T19:10:08.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:10:08.291 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:10:08.291 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:10:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:10:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:10:08.292 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:10:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:10:08.293 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:10:08.293 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:10:08.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:10:08.294 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:10:08.294 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:10:08.295 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:08.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:10:08.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:10:08.297 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:08.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:10:08.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:10:08.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2021-01-15T19:10:08.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:10:08.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:08.299 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:10:08.299 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.992733 2021-01-15T19:10:08.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:10:08.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:10:08.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: rm -fr td/osd-reuse-id/1 2021-01-15T19:10:08.303 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: run_osd td/osd-reuse-id 1 2021-01-15T19:10:08.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/osd-reuse-id 2021-01-15T19:10:08.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:10:08.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:10:08.304 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:10:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/osd-reuse-id/1 2021-01-15T19:10:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 ' 2021-01-15T19:10:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:10:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:10:08.305 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:10:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2021-01-15T19:10:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2021-01-15T19:10:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:10:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:10:08.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2021-01-15T19:10:08.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:10:08.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:08.307 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:08.307 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:10:08.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:10:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' 2021-01-15T19:10:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:10:08.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:10:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:10:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2021-01-15T19:10:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2021-01-15T19:10:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:10:08.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:10:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:10:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:10:08.310 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/osd-reuse-id/1 2021-01-15T19:10:08.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:10:08.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=dc47b4e9-5488-4815-9afb-8489a23705d5 2021-01-15T19:10:08.311 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 dc47b4e9-5488-4815-9afb-8489a23705d5' 2021-01-15T19:10:08.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:10:08.311 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 dc47b4e9-5488-4815-9afb-8489a23705d5 2021-01-15T19:10:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCQ6AFgxnbNEhAAAWmpI3heUMAa/9ODrB3l7Q== 2021-01-15T19:10:08.316 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCQ6AFgxnbNEhAAAWmpI3heUMAa/9ODrB3l7Q=="}' 2021-01-15T19:10:08.317 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new dc47b4e9-5488-4815-9afb-8489a23705d5 -i td/osd-reuse-id/1/new.json 2021-01-15T19:10:08.806 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:08.817 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/osd-reuse-id/1/new.json 2021-01-15T19:10:08.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCQ6AFgxnbNEhAAAWmpI3heUMAa/9ODrB3l7Q== --osd-uuid dc47b4e9-5488-4815-9afb-8489a23705d5 2021-01-15T19:10:08.839 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.838+0000 7f126df30080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:08.840 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.839+0000 7f126df30080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:08.842 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.841+0000 7f126df30080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.841+0000 7f126df30080 -1 asok(0x55dadae96000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.992733/ceph-osd.1.asok': (2) No such file or directory 2021-01-15T19:10:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.841+0000 7f126df30080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:10:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.841+0000 7f126df30080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:10:08.843 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.841+0000 7f126df30080 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2021-01-15T19:10:08.891 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:08.891+0000 7f126df30080 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2021-01-15T19:10:13.360 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2021-01-15T19:10:13.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:10:13.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:10:13.362 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:10:13.362 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:10:13.883 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:10:13.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=18047272-5074-4f9e-809f-a13d69a137fd --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.992733/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:13.884 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:10:13.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:10:13.885 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:10:13.886 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:10:13.902 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:13.901+0000 7f13f6d72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:13.916 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:13.915+0000 7f13f6d72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:13.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:13.917+0000 7f13f6d72080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:13.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:13.917+0000 7f13f6d72080 -1 asok(0x557e69a04000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.992733/ceph-osd.1.asok': (2) No such file or directory 2021-01-15T19:10:14.273 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:10:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:10:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:10:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:10:14.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:10:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:10:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:14.275 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:10:14.276 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:10:14.434 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:14.432+0000 7f13f6d72080 -1 Falling back to public interface 2021-01-15T19:10:14.683 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:10:15.260 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:15.259+0000 7f13f6d72080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:10:15.685 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:10:15.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:15.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:10:15.687 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:15.688 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:10:15.688 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:16.089 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/996829,v1:127.0.0.1:6811/996829] [v2:127.0.0.1:6812/996829,v1:127.0.0.1:6813/996829] exists,up dc47b4e9-5488-4815-9afb-8489a23705d5 2021-01-15T19:10:16.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:10:16.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:10:16.090 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:10:16.091 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: code=0 2021-01-15T19:10:16.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: teardown td/osd-reuse-id 0 2021-01-15T19:10:16.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2021-01-15T19:10:16.092 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-01-15T19:10:16.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2021-01-15T19:10:16.093 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:10:16.094 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:10:16.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:10:16.095 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:10:16.096 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:10:16.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:10:16.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:10:16.195 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:10:16.195 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:10:16.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:10:16.196 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:10:16.197 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:10:16.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:16.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:10:16.198 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:10:16.200 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:16.200 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:10:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-01-15T19:10:16.201 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2021-01-15T19:10:16.204 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:10:16.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.992733 2021-01-15T19:10:16.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.992733 2021-01-15T19:10:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:10:16.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:10:16.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: main: return 0 2021-01-15T19:10:16.208 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-01-15T19:10:16.209 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-01-15T19:10:16.280 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2021-01-15T19:10:16.280 DEBUG:teuthology.orchestra.run.smithi145:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=892c51dd3c5f7108e766bea30cd5e0d801b0abd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2021-01-15T19:10:16.344 INFO:tasks.workunit.client.0.smithi145.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-01-15T19:10:16.351 INFO:tasks.workunit.client.0.smithi145.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-01-15T19:10:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T19:10:16.352 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-01-15T19:10:16.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export PYTHONWARNINGS=ignore 2021-01-15T19:10:16.353 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: PYTHONWARNINGS=ignore 2021-01-15T19:10:16.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: export CEPH_CONF=/dev/null 2021-01-15T19:10:16.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: CEPH_CONF=/dev/null 2021-01-15T19:10:16.354 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: main: unset CEPH_ARGS 2021-01-15T19:10:16.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: local code 2021-01-15T19:10:16.355 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: run td/pg-split-merge 2021-01-15T19:10:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2021-01-15T19:10:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2021-01-15T19:10:16.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2021-01-15T19:10:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2021-01-15T19:10:16.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2021-01-15T19:10:16.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2021-01-15T19:10:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none ' 2021-01-15T19:10:16.369 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:10:16.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2021-01-15T19:10:16.370 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-01-15T19:10:16.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2021-01-15T19:10:16.371 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_import_after_merge_and_gap 2021-01-15T19:10:16.371 INFO:tasks.workunit.client.0.smithi145.stderr:TEST_import_after_split' 2021-01-15T19:10:16.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2021-01-15T19:10:16.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: TEST_a_merge_empty td/pg-split-merge 2021-01-15T19:10:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:20: TEST_a_merge_empty: local dir=td/pg-split-merge 2021-01-15T19:10:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: setup td/pg-split-merge 2021-01-15T19:10:16.373 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2021-01-15T19:10:16.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2021-01-15T19:10:16.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2021-01-15T19:10:16.374 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:10:16.375 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2021-01-15T19:10:16.375 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:10:16.376 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:10:16.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:10:16.376 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:10:16.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:10:16.377 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:10:16.378 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:10:16.378 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:10:16.379 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:10:16.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:10:16.379 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:10:16.380 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:10:16.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:16.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:10:16.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:10:16.381 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:10:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:10:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:10:16.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2021-01-15T19:10:16.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:10:16.383 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:16.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.997368 2021-01-15T19:10:16.385 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:10:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:10:16.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2021-01-15T19:10:16.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:10:16.387 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.388 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:16.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.997368 2021-01-15T19:10:16.389 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:10:16.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-01-15T19:10:16.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-01-15T19:10:16.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:10:16.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2021-01-15T19:10:16.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:23: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2021-01-15T19:10:16.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2021-01-15T19:10:16.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:10:16.392 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:10:16.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:10:16.393 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2021-01-15T19:10:16.394 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2021-01-15T19:10:16.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:10:16.663 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:16.664 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:16.664 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:16.665 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.665 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:16.666 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:16.666 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=3 2021-01-15T19:10:16.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:10:16.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:10:16.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:10:16.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:10:16.850 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:10:16.851 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:10:16.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:10:16.851 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:10:16.852 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:10:16.852 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:10:16.853 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.853 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:16.853 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.997368/ceph-mon.a.asok 2021-01-15T19:10:16.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:10:16.854 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.997368/ceph-mon.a.asok config get fsid 2021-01-15T19:10:16.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:10:16.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:10:16.947 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:10:16.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:10:16.948 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:10:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:10:16.948 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:10:16.949 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:10:16.949 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:10:16.949 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:16.949 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:16.950 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.997368/ceph-mon.a.asok 2021-01-15T19:10:16.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:10:16.950 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.997368/ceph-mon.a.asok config get mon_host 2021-01-15T19:10:17.048 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2021-01-15T19:10:17.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/pg-split-merge 2021-01-15T19:10:17.049 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:10:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:10:17.050 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:10:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/pg-split-merge/x 2021-01-15T19:10:17.051 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:10:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:10:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:17.549 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:17.549 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:17.550 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:17.550 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:17.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:17.550 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:10:17.551 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:10:17.581 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2021-01-15T19:10:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/pg-split-merge 2021-01-15T19:10:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:10:17.582 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:10:17.583 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:10:17.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/pg-split-merge/0 2021-01-15T19:10:17.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:10:17.584 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:10:17.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:10:17.585 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:10:17.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2021-01-15T19:10:17.586 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2021-01-15T19:10:17.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:10:17.587 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:10:17.588 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2021-01-15T19:10:17.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:10:17.588 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:17.589 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:17.589 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:17.590 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:17.590 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:17.590 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:17.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:17.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:10:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:10:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:10:17.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2021-01-15T19:10:17.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2021-01-15T19:10:17.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:10:17.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:10:17.594 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:17.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:10:17.595 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:10:17.596 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/pg-split-merge/0 2021-01-15T19:10:17.596 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:10:17.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=54a32646-3564-4d5a-aedb-849f0b109154 2021-01-15T19:10:17.597 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 54a32646-3564-4d5a-aedb-849f0b109154' 2021-01-15T19:10:17.597 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:10:17.598 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 54a32646-3564-4d5a-aedb-849f0b109154 2021-01-15T19:10:17.599 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCZ6AFgOI4iIxAA7l+OvrUe175iIVbdUUCerw== 2021-01-15T19:10:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCZ6AFgOI4iIxAA7l+OvrUe175iIVbdUUCerw=="}' 2021-01-15T19:10:17.600 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 54a32646-3564-4d5a-aedb-849f0b109154 -i td/pg-split-merge/0/new.json 2021-01-15T19:10:17.939 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:10:17.950 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/pg-split-merge/0/new.json 2021-01-15T19:10:17.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCZ6AFgOI4iIxAA7l+OvrUe175iIVbdUUCerw== --osd-uuid 54a32646-3564-4d5a-aedb-849f0b109154 2021-01-15T19:10:17.975 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.974+0000 7f7ce9d9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:17.977 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.976+0000 7f7ce9d9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:17.978 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.977+0000 7f7ce9d9d080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:17.979 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.977+0000 7f7ce9d9d080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:10:17.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.977+0000 7f7ce9d9d080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:10:17.980 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:17.977+0000 7f7ce9d9d080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:10:18.021 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:18.019+0000 7f7ce9d9d080 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2021-01-15T19:10:22.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/pg-split-merge/0/keyring 2021-01-15T19:10:22.796 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:10:22.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:10:22.797 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:10:22.797 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:10:23.319 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:10:23.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:10:23.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:10:23.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:10:23.320 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:10:23.341 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:23.340+0000 7f5f42339080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:23.361 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:23.360+0000 7f5f42339080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:23.363 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:23.362+0000 7f5f42339080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:23.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:10:23.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:10:23.724 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:10:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:10:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:10:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:10:23.725 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:23.726 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:10:23.726 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:10:23.879 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:23.878+0000 7f5f42339080 -1 Falling back to public interface 2021-01-15T19:10:24.120 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:10:24.671 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:24.670+0000 7f5f42339080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:10:25.121 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:10:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:10:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:25.122 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:10:25.123 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:25.532 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/998407,v1:127.0.0.1:6803/998407] [v2:127.0.0.1:6804/998407,v1:127.0.0.1:6805/998407] exists,up 54a32646-3564-4d5a-aedb-849f0b109154 2021-01-15T19:10:25.532 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:10:25.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:10:25.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:10:25.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2021-01-15T19:10:25.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/pg-split-merge 2021-01-15T19:10:25.533 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:10:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=1 2021-01-15T19:10:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:10:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/pg-split-merge/1 2021-01-15T19:10:25.534 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:10:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:10:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:10:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:10:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2021-01-15T19:10:25.535 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2021-01-15T19:10:25.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:10:25.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:10:25.536 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2021-01-15T19:10:25.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:10:25.536 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:25.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:25.537 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:25.537 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:25.537 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:25.537 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:25.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:25.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:10:25.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:10:25.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:10:25.538 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2021-01-15T19:10:25.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2021-01-15T19:10:25.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:10:25.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:10:25.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:25.539 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:10:25.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:10:25.540 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/pg-split-merge/1 2021-01-15T19:10:25.540 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:10:25.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59 2021-01-15T19:10:25.541 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd1 8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59' 2021-01-15T19:10:25.541 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:10:25.541 INFO:tasks.workunit.client.0.smithi145.stdout:add osd1 8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59 2021-01-15T19:10:25.547 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCh6AFgVhqZIBAAudSGpi0kmJA7gX7bpmAJ6w== 2021-01-15T19:10:25.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCh6AFgVhqZIBAAudSGpi0kmJA7gX7bpmAJ6w=="}' 2021-01-15T19:10:25.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59 -i td/pg-split-merge/1/new.json 2021-01-15T19:10:26.118 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:26.129 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/pg-split-merge/1/new.json 2021-01-15T19:10:26.130 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 1 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCh6AFgVhqZIBAAudSGpi0kmJA7gX7bpmAJ6w== --osd-uuid 8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59 2021-01-15T19:10:26.154 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.153+0000 7f40b2290080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:26.156 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.155+0000 7f40b2290080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:26.158 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.157+0000 7f40b2290080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:26.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.158+0000 7f40b2290080 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2021-01-15T19:10:26.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.158+0000 7f40b2290080 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2021-01-15T19:10:26.159 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.158+0000 7f40b2290080 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2021-01-15T19:10:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:26.209+0000 7f40b2290080 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2021-01-15T19:10:30.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/pg-split-merge/1/keyring 2021-01-15T19:10:30.968 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:10:30.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd1 key to auth repository 2021-01-15T19:10:30.969 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:10:30.969 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd1 key to auth repository 2021-01-15T19:10:31.498 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.1 2021-01-15T19:10:31.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 1 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:31.499 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:10:31.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:10:31.500 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:10:31.501 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.1 2021-01-15T19:10:31.520 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:31.519+0000 7f30f5cf8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:31.522 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:31.521+0000 7f30f5cf8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:31.524 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:31.523+0000 7f30f5cf8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:31.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 1 2021-01-15T19:10:31.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:10:31.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=1 2021-01-15T19:10:31.904 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:10:31.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:10:31.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:31.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:10:31.905 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:31.906 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:10:31.906 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:10:32.039 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:32.038+0000 7f30f5cf8080 -1 Falling back to public interface 2021-01-15T19:10:32.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:10:32.826 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:32.823+0000 7f30f5cf8080 -1 osd.1 0 log_to_monitors {default=true} 2021-01-15T19:10:33.296 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:10:33.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:33.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:10:33.297 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:33.298 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.1 up' 2021-01-15T19:10:33.298 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:33.705 INFO:tasks.workunit.client.0.smithi145.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/999671,v1:127.0.0.1:6811/999671] [v2:127.0.0.1:6812/999671,v1:127.0.0.1:6813/999671] exists,up 8e09e6d2-74b9-4b85-8fbc-4650ca7d2d59 2021-01-15T19:10:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:10:33.705 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:10:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:10:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2021-01-15T19:10:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/pg-split-merge 2021-01-15T19:10:33.706 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:10:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=2 2021-01-15T19:10:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:10:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/pg-split-merge/2 2021-01-15T19:10:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:10:33.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:10:33.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:10:33.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:10:33.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2021-01-15T19:10:33.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2021-01-15T19:10:33.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:10:33.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:10:33.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2021-01-15T19:10:33.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:10:33.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:10:33.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:10:33.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:10:33.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:10:33.710 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:10:33.710 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:33.710 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:10:33.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:10:33.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:10:33.711 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:10:33.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2021-01-15T19:10:33.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2021-01-15T19:10:33.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:10:33.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:10:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:10:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:10:33.714 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/pg-split-merge/2 2021-01-15T19:10:33.715 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:10:33.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=4f78930a-7a21-47df-952b-118abb81cae5 2021-01-15T19:10:33.716 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd2 4f78930a-7a21-47df-952b-118abb81cae5' 2021-01-15T19:10:33.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:10:33.716 INFO:tasks.workunit.client.0.smithi145.stdout:add osd2 4f78930a-7a21-47df-952b-118abb81cae5 2021-01-15T19:10:33.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQCp6AFgj1r/KhAAjHvGYs01PX3w07P/3i+ouQ== 2021-01-15T19:10:33.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQCp6AFgj1r/KhAAjHvGYs01PX3w07P/3i+ouQ=="}' 2021-01-15T19:10:33.722 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new 4f78930a-7a21-47df-952b-118abb81cae5 -i td/pg-split-merge/2/new.json 2021-01-15T19:10:34.242 INFO:tasks.workunit.client.0.smithi145.stdout:2 2021-01-15T19:10:34.254 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/pg-split-merge/2/new.json 2021-01-15T19:10:34.255 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 2 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCp6AFgj1r/KhAAjHvGYs01PX3w07P/3i+ouQ== --osd-uuid 4f78930a-7a21-47df-952b-118abb81cae5 2021-01-15T19:10:34.278 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.277+0000 7fd784034080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:34.280 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.279+0000 7fd784034080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:34.282 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.281+0000 7fd784034080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:34.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.281+0000 7fd784034080 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2021-01-15T19:10:34.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.281+0000 7fd784034080 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2021-01-15T19:10:34.283 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.281+0000 7fd784034080 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2021-01-15T19:10:34.329 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:34.328+0000 7fd784034080 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2021-01-15T19:10:39.022 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/pg-split-merge/2/keyring 2021-01-15T19:10:39.023 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:10:39.024 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd2 key to auth repository 2021-01-15T19:10:39.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd2 key to auth repository 2021-01-15T19:10:39.024 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:10:39.548 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:10:39.548 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.2 2021-01-15T19:10:39.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 2 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:10:39.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:10:39.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:10:39.549 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:10:39.570 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:39.569+0000 7fb5a2fb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:39.572 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:39.571+0000 7fb5a2fb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:39.574 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:39.573+0000 7fb5a2fb1080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:10:39.942 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 2 2021-01-15T19:10:39.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:10:39.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:10:39.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:10:39.943 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:10:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:10:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:39.944 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:10:39.945 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:10:40.091 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:40.090+0000 7fb5a2fb1080 -1 Falling back to public interface 2021-01-15T19:10:40.346 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:10:40.940 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:10:40.939+0000 7fb5a2fb1080 -1 osd.2 0 log_to_monitors {default=true} 2021-01-15T19:10:41.347 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:10:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:10:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:10:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:10:41.348 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:10:41.349 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:10:41.818 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1000943,v1:127.0.0.1:6819/1000943] [v2:127.0.0.1:6820/1000943,v1:127.0.0.1:6821/1000943] exists,up 4f78930a-7a21-47df-952b-118abb81cae5 2021-01-15T19:10:41.818 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:10:41.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:10:41.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:10:41.819 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:29: TEST_a_merge_empty: ceph osd pool create foo 2 2021-01-15T19:10:42.467 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:10:42.476 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2021-01-15T19:10:43.598 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 pgp_num to 1 2021-01-15T19:10:43.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:32: TEST_a_merge_empty: wait_for_clean 2021-01-15T19:10:43.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:10:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:10:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:10:43.609 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:10:43.610 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:10:43.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:10:43.610 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:10:43.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:10:43.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:10:43.611 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:10:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:10:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:10:43.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:10:43.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:10:43.701 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:10:44.107 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:10:44.108 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:10:44.108 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:10:44.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:10:44.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:44.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:10:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836487 2021-01-15T19:10:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836487 2021-01-15T19:10:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487' 2021-01-15T19:10:44.245 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:44.246 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:10:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672965 2021-01-15T19:10:44.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672965 2021-01-15T19:10:44.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2021-01-15T19:10:44.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:44.382 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:10:44.517 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509443 2021-01-15T19:10:44.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509443 2021-01-15T19:10:44.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836487 1-42949672965 2-64424509443' 2021-01-15T19:10:44.518 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:44.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2021-01-15T19:10:44.519 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:44.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:10:44.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836487 2021-01-15T19:10:44.520 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:44.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836487 2021-01-15T19:10:44.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2021-01-15T19:10:44.522 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836487 2021-01-15T19:10:44.522 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:44.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836485 -lt 21474836487 2021-01-15T19:10:44.914 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:10:45.915 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:10:45.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:46.356 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836487 -lt 21474836487 2021-01-15T19:10:46.357 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:46.357 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2021-01-15T19:10:46.358 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:46.358 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:10:46.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672965 2021-01-15T19:10:46.359 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:46.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672965 2021-01-15T19:10:46.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-01-15T19:10:46.361 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:10:46.361 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672965 2021-01-15T19:10:46.773 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672965 -lt 42949672965 2021-01-15T19:10:46.774 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:46.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509443 2021-01-15T19:10:46.775 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:46.776 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:10:46.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509443 2021-01-15T19:10:46.778 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:46.778 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509443 2021-01-15T19:10:46.779 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2021-01-15T19:10:46.779 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:10:46.780 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509443 2021-01-15T19:10:47.190 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509443 -lt 64424509443 2021-01-15T19:10:47.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:10:47.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:47.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:47.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T19:10:47.704 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:10:47.705 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:10:47.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:10:47.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:10:47.706 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:10:47.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:10:47.707 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:10:48.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T19:10:48.084 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:10:48.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:48.085 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:48.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T19:10:48.607 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:10:48.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:10:48.608 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:36: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2021-01-15T19:10:48.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2021-01-15T19:10:48.951 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2021-01-15T19:10:49.093 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2021-01-15T19:10:49.227 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:41: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2021-01-15T19:10:49.361 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2021-01-15T19:10:49.513 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2021-01-15T19:10:49.663 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2021-01-15T19:10:49.805 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2021-01-15T19:10:49.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2021-01-15T19:10:50.082 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2021-01-15T19:10:50.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:49: TEST_a_merge_empty: wait_for_clean 2021-01-15T19:10:50.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:10:50.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:10:50.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:10:50.226 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:10:50.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:10:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:10:50.227 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:10:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:10:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:10:50.228 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:10:50.314 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:10:50.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:10:50.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:10:50.315 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:10:50.316 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:10:50.809 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:10:50.810 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:10:50.810 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:10:50.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:10:50.810 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:50.811 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:10:50.947 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836490 2021-01-15T19:10:50.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836490 2021-01-15T19:10:50.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490' 2021-01-15T19:10:50.948 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:50.949 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:10:51.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672968 2021-01-15T19:10:51.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672968 2021-01-15T19:10:51.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968' 2021-01-15T19:10:51.085 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:51.086 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:10:51.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509446 2021-01-15T19:10:51.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509446 2021-01-15T19:10:51.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836490 1-42949672968 2-64424509446' 2021-01-15T19:10:51.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:51.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2021-01-15T19:10:51.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:51.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:10:51.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836490 2021-01-15T19:10:51.224 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836490 2021-01-15T19:10:51.225 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2021-01-15T19:10:51.225 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836490 2021-01-15T19:10:51.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836490 2021-01-15T19:10:51.643 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:51.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2021-01-15T19:10:51.644 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:51.645 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:10:51.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672968 2021-01-15T19:10:51.646 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:51.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672968 2021-01-15T19:10:51.647 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2021-01-15T19:10:51.647 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672968 2021-01-15T19:10:51.648 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:10:52.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672968 -lt 42949672968 2021-01-15T19:10:52.052 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:52.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2021-01-15T19:10:52.053 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:52.054 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:10:52.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509446 2021-01-15T19:10:52.055 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:52.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509446 2021-01-15T19:10:52.056 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2021-01-15T19:10:52.056 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509446 2021-01-15T19:10:52.057 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:10:52.445 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509446 -lt 64424509446 2021-01-15T19:10:52.446 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:10:52.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:52.447 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:52.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T19:10:52.990 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:10:52.991 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:10:52.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:10:52.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:10:52.992 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:10:52.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:10:52.993 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:10:53.366 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T19:10:53.367 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:10:53.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:53.368 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:53.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T19:10:53.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:10:53.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:10:53.878 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:51: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2021-01-15T19:10:54.005 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T19:10:54.006 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "" 2021-01-15T19:10:54.006 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T19:10:54.015 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph osd pool set foo size 2 2021-01-15T19:10:54.781 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 2 2021-01-15T19:10:54.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: wait_for_clean 2021-01-15T19:10:54.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:10:54.791 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:10:54.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:10:54.792 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:10:54.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:10:54.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:10:54.793 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:10:54.793 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:10:54.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:10:54.794 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:10:54.879 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:10:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:10:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:10:54.880 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:10:54.881 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:10:55.300 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:10:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:10:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:10:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:10:55.301 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:55.302 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:10:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836493 2021-01-15T19:10:55.438 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836493 2021-01-15T19:10:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493' 2021-01-15T19:10:55.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:55.439 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:10:55.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672971 2021-01-15T19:10:55.577 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672971 2021-01-15T19:10:55.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672971' 2021-01-15T19:10:55.578 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:10:55.578 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:10:55.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=64424509449 2021-01-15T19:10:55.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 64424509449 2021-01-15T19:10:55.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836493 1-42949672971 2-64424509449' 2021-01-15T19:10:55.715 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:55.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2021-01-15T19:10:55.716 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:55.717 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:10:55.717 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836493 2021-01-15T19:10:55.718 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:55.718 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836493 2021-01-15T19:10:55.719 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2021-01-15T19:10:55.719 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836493 2021-01-15T19:10:55.720 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:56.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836491 -lt 21474836493 2021-01-15T19:10:56.109 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:10:57.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:10:57.109 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:10:57.521 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836494 -lt 21474836493 2021-01-15T19:10:57.522 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:57.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672971 2021-01-15T19:10:57.523 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:57.523 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:10:57.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672971 2021-01-15T19:10:57.525 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:57.525 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672971 2021-01-15T19:10:57.526 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672971' 2021-01-15T19:10:57.526 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672971 2021-01-15T19:10:57.527 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:10:57.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672971 -lt 42949672971 2021-01-15T19:10:57.916 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:10:57.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2021-01-15T19:10:57.917 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:10:57.917 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:10:57.918 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-64424509449 2021-01-15T19:10:57.919 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:10:57.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=64424509449 2021-01-15T19:10:57.919 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2021-01-15T19:10:57.920 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 64424509449 2021-01-15T19:10:57.920 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:10:58.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 64424509449 -lt 64424509449 2021-01-15T19:10:58.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:10:58.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:58.308 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:58.846 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T19:10:58.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:10:58.847 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:10:58.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:10:58.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:10:58.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:10:58.848 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:10:58.849 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:10:59.224 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T19:10:59.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:10:59.225 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:10:59.226 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:10:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T19:10:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:10:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:10:59.741 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:55: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2021-01-15T19:10:59.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:10:59.742 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:10:59.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:10:59.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:10:59.742 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:10:59.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:10:59.847 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2021-01-15T19:11:00.862 INFO:tasks.workunit.client.0.smithi145.stdout: marking collection for removal 2021-01-15T19:11:00.863 INFO:tasks.workunit.client.0.smithi145.stdout:setting '_remove' omap key 2021-01-15T19:11:00.863 INFO:tasks.workunit.client.0.smithi145.stdout:finish_remove_pgs 1.1_head removing 1.1 2021-01-15T19:11:00.974 INFO:tasks.workunit.client.0.smithi145.stdout:Remove successful 2021-01-15T19:11:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2021-01-15T19:11:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: local dir=td/pg-split-merge 2021-01-15T19:11:01.380 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: shift 2021-01-15T19:11:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local id=2 2021-01-15T19:11:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2021-01-15T19:11:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local osd_data=td/pg-split-merge/2 2021-01-15T19:11:01.381 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local 'ceph_args=--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:11:01.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:11:01.382 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:11:01.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:11:01.383 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2021-01-15T19:11:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2021-01-15T19:11:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --chdir=' 2021-01-15T19:11:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+= 2021-01-15T19:11:01.384 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2021-01-15T19:11:01.384 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: get_asok_path 2021-01-15T19:11:01.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:11:01.385 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:11:01.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:11:01.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:01.385 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:01.386 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:01.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:01.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:11:01.386 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2021-01-15T19:11:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2021-01-15T19:11:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:11:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:11:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:11:01.387 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' ' 2021-01-15T19:11:01.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+= 2021-01-15T19:11:01.388 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: mkdir -p td/pg-split-merge/2 2021-01-15T19:11:01.389 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.2 2021-01-15T19:11:01.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: echo start osd.2 2021-01-15T19:11:01.389 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: ceph-osd -i 2 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:11:01.390 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: cat td/pg-split-merge/2/whoami 2021-01-15T19:11:01.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: '[' 2 = 2 ']' 2021-01-15T19:11:01.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: ceph osd dump --format=json 2021-01-15T19:11:01.390 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: jq '.flags_set[]' 2021-01-15T19:11:01.391 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: grep -q '"noup"' 2021-01-15T19:11:01.405 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:01.404+0000 7f3a963f8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:01.408 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:01.407+0000 7f3a963f8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:01.409 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:01.408+0000 7f3a963f8080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:01.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: wait_for_osd up 2 2021-01-15T19:11:01.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:11:01.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=2 2021-01-15T19:11:01.788 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:11:01.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:11:01.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:11:01.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:11:01.789 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:11:01.790 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:11:01.790 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:11:01.926 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:01.925+0000 7f3a963f8080 -1 Falling back to public interface 2021-01-15T19:11:02.180 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:11:02.918 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:02.916+0000 7f3a963f8080 -1 osd.2 28 log_to_monitors {default=true} 2021-01-15T19:11:03.181 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:11:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:11:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:11:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:11:03.182 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.2 up' 2021-01-15T19:11:03.183 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:11:03.589 INFO:tasks.workunit.client.0.smithi145.stdout:osd.2 up in weight 1 up_from 30 up_thru 17 down_at 29 last_clean_interval [15,28) [v2:127.0.0.1:6818/1003272,v1:127.0.0.1:6819/1003272] [v2:127.0.0.1:6820/1003272,v1:127.0.0.1:6821/1003272] exists,up 4f78930a-7a21-47df-952b-118abb81cae5 2021-01-15T19:11:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:11:03.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:11:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:11:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:59: TEST_a_merge_empty: wait_for_clean 2021-01-15T19:11:03.591 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:11:03.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:11:03.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:11:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:11:03.593 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:11:03.593 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:11:03.594 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:11:03.594 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:11:03.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:11:03.595 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:11:03.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:11:03.677 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:11:03.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:11:03.678 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:11:03.678 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:11:04.068 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:11:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:11:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:11:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:11:04.069 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:04.070 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:11:04.205 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836497 2021-01-15T19:11:04.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836497 2021-01-15T19:11:04.206 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497' 2021-01-15T19:11:04.207 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:04.207 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:11:04.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672975 2021-01-15T19:11:04.342 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672975 2021-01-15T19:11:04.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672975' 2021-01-15T19:11:04.343 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:04.343 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:11:04.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018883 2021-01-15T19:11:04.478 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018883 2021-01-15T19:11:04.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836497 1-42949672975 2-128849018883' 2021-01-15T19:11:04.479 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:04.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836497 2021-01-15T19:11:04.480 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:04.480 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:11:04.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836497 2021-01-15T19:11:04.481 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:04.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836497 2021-01-15T19:11:04.482 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2021-01-15T19:11:04.482 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:11:04.482 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836497 2021-01-15T19:11:04.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836495 -lt 21474836497 2021-01-15T19:11:04.872 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:11:05.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:11:05.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:11:06.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836497 -lt 21474836497 2021-01-15T19:11:06.269 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:06.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2021-01-15T19:11:06.270 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:06.270 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:11:06.271 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672975 2021-01-15T19:11:06.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:06.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672975 2021-01-15T19:11:06.272 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2021-01-15T19:11:06.272 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:11:06.273 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672975 2021-01-15T19:11:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672975 -lt 42949672975 2021-01-15T19:11:06.671 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:06.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-128849018883 2021-01-15T19:11:06.672 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:06.672 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:11:06.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-128849018883 2021-01-15T19:11:06.673 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:06.673 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018883 2021-01-15T19:11:06.674 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2021-01-15T19:11:06.674 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:11:06.674 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 128849018883 2021-01-15T19:11:07.075 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018883 -lt 128849018883 2021-01-15T19:11:07.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:11:07.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:07.076 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:07.628 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 2 == 0 2021-01-15T19:11:07.629 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:11:07.629 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:11:07.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:11:07.630 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:11:07.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:11:07.631 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:11:07.632 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:11:08.009 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=2 2021-01-15T19:11:08.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:11:08.009 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:08.010 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:08.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 = 2 2021-01-15T19:11:08.519 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:11:08.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:11:08.520 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:65: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2021-01-15T19:11:08.646 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T19:11:08.646 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "" 2021-01-15T19:11:08.646 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T19:11:08.655 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph osd out 0 1 2021-01-15T19:11:09.208 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already out. osd.1 is already out. 2021-01-15T19:11:09.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2021-01-15T19:11:10.084 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: sleep 5 2021-01-15T19:11:15.086 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2021-01-15T19:11:15.223 INFO:tasks.workunit.client.0.smithi145.stdout:{ 2021-01-15T19:11:15.223 INFO:tasks.workunit.client.0.smithi145.stdout: "success": "" 2021-01-15T19:11:15.224 INFO:tasks.workunit.client.0.smithi145.stdout:} 2021-01-15T19:11:15.233 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:73: TEST_a_merge_empty: ceph osd in 0 1 2021-01-15T19:11:15.778 INFO:tasks.workunit.client.0.smithi145.stderr:osd.0 is already in. osd.1 is already in. 2021-01-15T19:11:15.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd pool set foo size 3 2021-01-15T19:11:16.759 INFO:tasks.workunit.client.0.smithi145.stderr:set pool 1 size to 3 2021-01-15T19:11:16.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:76: TEST_a_merge_empty: wait_for_clean 2021-01-15T19:11:16.768 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:11:16.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:11:16.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:11:16.769 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:11:16.769 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:11:16.770 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:11:16.771 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:11:16.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:11:16.772 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:11:16.773 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:11:16.856 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:11:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:11:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:11:16.857 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:11:16.857 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:11:17.428 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ids='0 2021-01-15T19:11:17.429 INFO:tasks.workunit.client.0.smithi145.stderr:1 2021-01-15T19:11:17.429 INFO:tasks.workunit.client.0.smithi145.stderr:2' 2021-01-15T19:11:17.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: seqs= 2021-01-15T19:11:17.430 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:17.431 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-01-15T19:11:17.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=21474836502 2021-01-15T19:11:17.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 21474836502 2021-01-15T19:11:17.570 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502' 2021-01-15T19:11:17.571 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:17.571 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-01-15T19:11:17.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=42949672979 2021-01-15T19:11:17.786 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 42949672979 2021-01-15T19:11:17.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672979' 2021-01-15T19:11:17.787 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: for osd in $ids 2021-01-15T19:11:17.787 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-01-15T19:11:17.928 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seq=128849018887 2021-01-15T19:11:17.931 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2073: flush_pg_stats: test -z 128849018887 2021-01-15T19:11:17.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-128849018887' 2021-01-15T19:11:17.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:17.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2021-01-15T19:11:17.932 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:17.932 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=0 2021-01-15T19:11:17.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 0-21474836502 2021-01-15T19:11:17.933 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:17.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=21474836502 2021-01-15T19:11:17.933 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2021-01-15T19:11:17.934 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:11:17.934 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.0 seq 21474836502 2021-01-15T19:11:18.439 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836500 -lt 21474836502 2021-01-15T19:11:18.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: sleep 1 2021-01-15T19:11:19.440 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: flush_pg_stats: '[' 300 -eq 0 ']' 2021-01-15T19:11:19.441 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 0 2021-01-15T19:11:19.934 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 21474836502 -lt 21474836502 2021-01-15T19:11:19.935 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:19.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672979 2021-01-15T19:11:19.935 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:19.936 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=1 2021-01-15T19:11:19.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 1-42949672979 2021-01-15T19:11:19.936 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:19.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=42949672979 2021-01-15T19:11:19.937 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2021-01-15T19:11:19.937 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 1 2021-01-15T19:11:19.938 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.1 seq 42949672979 2021-01-15T19:11:20.330 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 42949672980 -lt 42949672979 2021-01-15T19:11:20.331 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: for s in $seqs 2021-01-15T19:11:20.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-128849018887 2021-01-15T19:11:20.331 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 1 2021-01-15T19:11:20.333 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: osd=2 2021-01-15T19:11:20.334 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 2-128849018887 2021-01-15T19:11:20.335 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: cut -d - -f 2 2021-01-15T19:11:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: seq=128849018887 2021-01-15T19:11:20.335 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: echo 'waiting osd.2 seq 128849018887' 2021-01-15T19:11:20.336 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: ceph osd last-stat-seq 2 2021-01-15T19:11:20.336 INFO:tasks.workunit.client.0.smithi145.stdout:waiting osd.2 seq 128849018887 2021-01-15T19:11:20.721 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: test 128849018888 -lt 128849018887 2021-01-15T19:11:20.722 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: get_num_pgs 2021-01-15T19:11:20.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:20.723 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:21.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1587: wait_for_clean: test 1 == 0 2021-01-15T19:11:21.274 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:11:21.274 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:11:21.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:11:21.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:11:21.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:11:21.275 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:11:21.276 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:11:21.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T19:11:21.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:11:21.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:21.698 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:22.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T19:11:22.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' -1 2021-01-15T19:11:22.208 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: loop=0 2021-01-15T19:11:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: num_active_clean=0 2021-01-15T19:11:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:11:22.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:11:22.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:11:22.309 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:11:22.309 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:11:22.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:11:22.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:11:22.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:11:22.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:11:22.310 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:11:22.686 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T19:11:22.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:11:22.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:22.687 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:23.202 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T19:11:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T19:11:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:11:23.203 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:11:23.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:11:23.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:11:23.204 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:11:23.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:11:23.205 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:11:23.712 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4400 2021-01-15T19:11:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4400 '!=' null 2021-01-15T19:11:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T19:11:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:11:23.713 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:11:23.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:11:23.814 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:11:23.814 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:11:23.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:11:23.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:11:23.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:11:23.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:11:23.815 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:11:24.192 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=0 2021-01-15T19:11:24.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:11:24.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:24.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:24.707 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 = 1 2021-01-15T19:11:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: test 0 '!=' 0 2021-01-15T19:11:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: get_is_making_recovery_progress 2021-01-15T19:11:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: local recovery_progress 2021-01-15T19:11:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-01-15T19:11:24.708 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-01-15T19:11:24.709 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1275: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-01-15T19:11:24.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: ceph --format json status 2021-01-15T19:11:24.709 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-01-15T19:11:25.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: local progress=4400 2021-01-15T19:11:25.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1278: get_is_making_recovery_progress: test 4400 '!=' null 2021-01-15T19:11:25.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: loop=0 2021-01-15T19:11:25.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: eval 2021-01-15T19:11:25.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: sleep 0.1 2021-01-15T19:11:25.318 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1609: wait_for_clean: loop+=1 2021-01-15T19:11:25.319 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1591: wait_for_clean: true 2021-01-15T19:11:25.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_active_clean 2021-01-15T19:11:25.319 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: local expression 2021-01-15T19:11:25.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-01-15T19:11:25.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-01-15T19:11:25.320 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: ceph --format json pg dump pgs 2021-01-15T19:11:25.321 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-01-15T19:11:25.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: cur_active_clean=1 2021-01-15T19:11:25.696 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: get_num_pgs 2021-01-15T19:11:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: ceph --format json status 2021-01-15T19:11:25.697 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: get_num_pgs: jq .pgmap.num_pgs 2021-01-15T19:11:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 = 1 2021-01-15T19:11:26.209 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: break 2021-01-15T19:11:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1611: wait_for_clean: return 0 2021-01-15T19:11:26.210 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:80: TEST_a_merge_empty: ceph pg scrub 1.0 2021-01-15T19:11:26.579 INFO:tasks.workunit.client.0.smithi145.stderr:instructing pg 1.0 on osd.1 to scrub 2021-01-15T19:11:26.589 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: sleep 10 2021-01-15T19:11:36.590 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: ceph log last debug 2021-01-15T19:11:37.096 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:16.096361+0000 mon.a (mon.0) 225 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2021-01-15T19:11:37.097 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:16.452097+0000 mon.a (mon.0) 228 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2021-01-15T19:11:37.097 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:16.756113+0000 mon.a (mon.0) 232 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2021-01-15T19:11:37.097 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:16.976425+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2021-01-15T19:11:37.098 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:17.352934+0000 mon.a (mon.0) 235 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2021-01-15T19:11:37.098 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:17.367732+0000 mgr.x (mgr.4100) 36 : cluster [DBG] pgmap v69: 1 pgs: 1 unknown; 57 KiB data, 11 MiB used, 200 GiB / 200 GiB avail 2021-01-15T19:11:37.098 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:17.695922+0000 mon.a (mon.0) 237 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2021-01-15T19:11:37.098 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:18.074863+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2021-01-15T19:11:37.099 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:18.163911+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (25'7,25'7] MIN to 25'7 2021-01-15T19:11:37.099 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:19.368211+0000 mgr.x (mgr.4100) 37 : cluster [DBG] pgmap v72: 1 pgs: 1 active+undersized+remapped+backfilling; 89 KiB data, 17 MiB used, 300 GiB / 300 GiB avail; 14 KiB/s wr, 1 op/s; 9.3 KiB/s, 1 objects/s recovering 2021-01-15T19:11:37.099 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:19.545003+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e45: 3 total, 3 up, 3 in 2021-01-15T19:11:37.099 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:19.862920+0000 mon.a (mon.0) 243 : cluster [DBG] osdmap e46: 3 total, 3 up, 3 in 2021-01-15T19:11:37.100 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:21.368504+0000 mgr.x (mgr.4100) 38 : cluster [DBG] pgmap v75: 1 pgs: 1 active+undersized+remapped+backfilling; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 9.0 KiB/s wr, 0 op/s; 6.1 KiB/s, 0 objects/s recovering 2021-01-15T19:11:37.100 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:23.368834+0000 mgr.x (mgr.4100) 43 : cluster [DBG] pgmap v76: 1 pgs: 1 active+undersized+remapped+backfilling; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 6.3 KiB/s wr, 0 op/s; 4.3 KiB/s, 0 objects/s recovering 2021-01-15T19:11:37.100 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:25.369168+0000 mgr.x (mgr.4100) 46 : cluster [DBG] pgmap v77: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 4.5 KiB/s, 0 objects/s recovering 2021-01-15T19:11:37.100 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:27.369742+0000 mgr.x (mgr.4100) 51 : cluster [DBG] pgmap v78: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 1.0 KiB/s, 0 objects/s recovering 2021-01-15T19:11:37.101 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:29.370113+0000 mgr.x (mgr.4100) 52 : cluster [DBG] pgmap v79: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 846 B/s, 0 objects/s recovering 2021-01-15T19:11:37.101 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:28.874345+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub ok 2021-01-15T19:11:37.101 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:31.370697+0000 mgr.x (mgr.4100) 53 : cluster [DBG] pgmap v80: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 723 B/s, 0 objects/s recovering 2021-01-15T19:11:37.101 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:33.371111+0000 mgr.x (mgr.4100) 54 : cluster [DBG] pgmap v81: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 693 B/s, 0 objects/s recovering 2021-01-15T19:11:37.102 INFO:tasks.workunit.client.0.smithi145.stdout:2021-01-15T19:11:35.371644+0000 mgr.x (mgr.4100) 55 : cluster [DBG] pgmap v82: 1 pgs: 1 active+clean; 89 KiB data, 18 MiB used, 300 GiB / 300 GiB avail; 693 B/s, 0 objects/s recovering 2021-01-15T19:11:37.108 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph pg ls 2021-01-15T19:11:37.479 INFO:tasks.workunit.client.0.smithi145.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP 2021-01-15T19:11:37.479 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 11 0 0 0 91542 0 0 0 active+clean 8s 25'7 46:87 [1,0,2]p1 [1,0,2]p1 2021-01-15T19:11:28.874352+0000 2021-01-15T19:10:42.204018+0000 2021-01-15T19:11:37.479 INFO:tasks.workunit.client.0.smithi145.stdout: 2021-01-15T19:11:37.480 INFO:tasks.workunit.client.0.smithi145.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-01-15T19:11:37.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2021-01-15T19:11:37.491 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: grep ' active.clean ' 2021-01-15T19:11:37.895 INFO:tasks.workunit.client.0.smithi145.stdout:1.0 11 0 0 0 91542 0 0 0 active+clean 9s 25'7 46:87 [1,0,2]p1 [1,0,2]p1 2021-01-15T19:11:28.874352+0000 2021-01-15T19:10:42.204018+0000 2021-01-15T19:11:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2021-01-15T19:11:37.896 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: TEST_import_after_merge_and_gap td/pg-split-merge 2021-01-15T19:11:37.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:88: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2021-01-15T19:11:37.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:90: TEST_import_after_merge_and_gap: setup td/pg-split-merge 2021-01-15T19:11:37.897 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2021-01-15T19:11:37.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2021-01-15T19:11:37.898 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2021-01-15T19:11:37.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-01-15T19:11:37.899 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2021-01-15T19:11:37.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-01-15T19:11:37.899 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-01-15T19:11:37.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-01-15T19:11:37.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-01-15T19:11:37.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-01-15T19:11:38.001 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-01-15T19:11:38.001 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-01-15T19:11:38.002 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-01-15T19:11:38.002 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-01-15T19:11:38.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-01-15T19:11:38.003 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-01-15T19:11:38.003 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-01-15T19:11:38.004 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:11:38.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-01-15T19:11:38.005 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-01-15T19:11:38.006 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-01-15T19:11:38.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-01-15T19:11:38.006 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-01-15T19:11:38.007 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2021-01-15T19:11:38.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-01-15T19:11:38.015 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:38.016 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:38.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.997368 2021-01-15T19:11:38.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-01-15T19:11:38.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-01-15T19:11:38.016 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2021-01-15T19:11:38.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-01-15T19:11:38.018 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:38.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.997368 2021-01-15T19:11:38.019 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-01-15T19:11:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-01-15T19:11:38.019 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-01-15T19:11:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2021-01-15T19:11:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:11:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2021-01-15T19:11:38.020 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-01-15T19:11:38.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-01-15T19:11:38.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-01-15T19:11:38.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2021-01-15T19:11:38.021 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:11:38.311 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-01-15T19:11:38.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:11:38.312 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:11:38.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:11:38.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:38.312 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:38.313 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:38.313 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2021-01-15T19:11:38.495 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: cat 2021-01-15T19:11:38.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a fsid 2021-01-15T19:11:38.497 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:11:38.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:11:38.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=fsid 2021-01-15T19:11:38.498 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .fsid 2021-01-15T19:11:38.498 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:11:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:11:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:11:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:11:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:38.499 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:38.500 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.997368/ceph-mon.a.asok 2021-01-15T19:11:38.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:11:38.500 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.997368/ceph-mon.a.asok config get fsid 2021-01-15T19:11:38.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:486: run_mon: get_config mon a mon_host 2021-01-15T19:11:38.598 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local daemon=mon 2021-01-15T19:11:38.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local id=a 2021-01-15T19:11:38.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: get_config: local config=mon_host 2021-01-15T19:11:38.599 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: get_config: jq -r .mon_host 2021-01-15T19:11:38.599 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: get_asok_path mon.a 2021-01-15T19:11:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-01-15T19:11:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-01-15T19:11:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-01-15T19:11:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:38.600 INFO:tasks.workunit.client.0.smithi145.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:38.601 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.997368/ceph-mon.a.asok 2021-01-15T19:11:38.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: CEPH_ARGS= 2021-01-15T19:11:38.601 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: get_config: ceph --format json daemon /tmp/ceph-asok.997368/ceph-mon.a.asok config get mon_host 2021-01-15T19:11:38.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2021-01-15T19:11:38.701 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: local dir=td/pg-split-merge 2021-01-15T19:11:38.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: shift 2021-01-15T19:11:38.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: local id=x 2021-01-15T19:11:38.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: shift 2021-01-15T19:11:38.702 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: local data=td/pg-split-merge/x 2021-01-15T19:11:38.703 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:558: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2021-01-15T19:11:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: get_asok_path 2021-01-15T19:11:39.190 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:11:39.191 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:11:39.191 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:11:39.192 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:39.192 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:39.192 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:39.193 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:11:39.194 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:560: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-01-15T19:11:39.214 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2021-01-15T19:11:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: local dir=td/pg-split-merge 2021-01-15T19:11:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: shift 2021-01-15T19:11:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: local id=0 2021-01-15T19:11:39.215 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: shift 2021-01-15T19:11:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: local osd_data=td/pg-split-merge/0 2021-01-15T19:11:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: local 'ceph_args=--fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2021-01-15T19:11:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-01-15T19:11:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-journal-size=100' 2021-01-15T19:11:39.216 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-01-15T19:11:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2021-01-15T19:11:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2021-01-15T19:11:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --chdir=' 2021-01-15T19:11:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+= 2021-01-15T19:11:39.217 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2021-01-15T19:11:39.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: get_asok_path 2021-01-15T19:11:39.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-01-15T19:11:39.218 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-01-15T19:11:39.218 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-01-15T19:11:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-01-15T19:11:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.997368 2021-01-15T19:11:39.219 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' 2021-01-15T19:11:39.219 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-osd=20' 2021-01-15T19:11:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-ms=1' 2021-01-15T19:11:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-monc=20' 2021-01-15T19:11:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2021-01-15T19:11:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2021-01-15T19:11:39.220 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-01-15T19:11:39.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-01-15T19:11:39.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:11:39.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' ' 2021-01-15T19:11:39.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+= 2021-01-15T19:11:39.221 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: mkdir -p td/pg-split-merge/0 2021-01-15T19:11:39.222 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: uuidgen 2021-01-15T19:11:39.222 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: local uuid=d58832c9-29f6-43c0-afd8-0dd99e4f8977 2021-01-15T19:11:39.223 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: echo 'add osd0 d58832c9-29f6-43c0-afd8-0dd99e4f8977' 2021-01-15T19:11:39.223 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: ceph-authtool --gen-print-key 2021-01-15T19:11:39.223 INFO:tasks.workunit.client.0.smithi145.stdout:add osd0 d58832c9-29f6-43c0-afd8-0dd99e4f8977 2021-01-15T19:11:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: OSD_SECRET=AQDr6AFgB4C4DRAA3blEZ60Pdh//pM2Bls4z0w== 2021-01-15T19:11:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo '{"cephx_secret": "AQDr6AFgB4C4DRAA3blEZ60Pdh//pM2Bls4z0w=="}' 2021-01-15T19:11:39.231 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph osd new d58832c9-29f6-43c0-afd8-0dd99e4f8977 -i td/pg-split-merge/0/new.json 2021-01-15T19:11:39.581 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:11:39.592 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: rm td/pg-split-merge/0/new.json 2021-01-15T19:11:39.593 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph-osd -i 0 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDr6AFgB4C4DRAA3blEZ60Pdh//pM2Bls4z0w== --osd-uuid d58832c9-29f6-43c0-afd8-0dd99e4f8977 2021-01-15T19:11:39.621 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.616+0000 7fb3b538a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:39.622 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.618+0000 7fb3b538a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:39.623 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.619+0000 7fb3b538a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:39.623 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.620+0000 7fb3b538a080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:11:39.624 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.620+0000 7fb3b538a080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:11:39.624 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.620+0000 7fb3b538a080 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2021-01-15T19:11:39.675 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:39.674+0000 7fb3b538a080 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2021-01-15T19:11:44.370 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: local key_fn=td/pg-split-merge/0/keyring 2021-01-15T19:11:44.371 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: cat 2021-01-15T19:11:44.371 INFO:tasks.workunit.client.0.smithi145.stdout:adding osd0 key to auth repository 2021-01-15T19:11:44.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: echo adding osd0 key to auth repository 2021-01-15T19:11:44.372 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-01-15T19:11:44.900 INFO:tasks.workunit.client.0.smithi145.stdout:start osd.0 2021-01-15T19:11:44.900 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo start osd.0 2021-01-15T19:11:44.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph-osd -i 0 --fsid=e983d0c6-e2b8-477e-833c-3c0527499d78 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.997368/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-01-15T19:11:44.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: ceph osd dump --format=json 2021-01-15T19:11:44.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: jq '.flags_set[]' 2021-01-15T19:11:44.901 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:679: run_osd: grep -q '"noup"' 2021-01-15T19:11:44.923 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:44.922+0000 7f909b30a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:44.944 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:44.942+0000 7f909b30a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:44.946 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:44.945+0000 7f909b30a080 -1 WARNING: all dangerous and experimental features are enabled. 2021-01-15T19:11:45.306 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:682: run_osd: wait_for_osd up 0 2021-01-15T19:11:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local state=up 2021-01-15T19:11:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:917: wait_for_osd: local id=0 2021-01-15T19:11:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: status=1 2021-01-15T19:11:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i=0 )) 2021-01-15T19:11:45.307 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:11:45.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 0 2021-01-15T19:11:45.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:11:45.308 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:11:45.308 INFO:tasks.workunit.client.0.smithi145.stdout:0 2021-01-15T19:11:45.462 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:45.461+0000 7f909b30a080 -1 Falling back to public interface 2021-01-15T19:11:45.696 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: wait_for_osd: sleep 1 2021-01-15T19:11:46.237 INFO:tasks.workunit.client.0.smithi145.stderr:2021-01-15T19:11:46.236+0000 7f909b30a080 -1 osd.0 0 log_to_monitors {default=true} 2021-01-15T19:11:46.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i++ )) 2021-01-15T19:11:46.698 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: (( i < 300 )) 2021-01-15T19:11:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: echo 1 2021-01-15T19:11:46.699 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: ceph osd dump 2021-01-15T19:11:46.700 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: grep 'osd.0 up' 2021-01-15T19:11:46.700 INFO:tasks.workunit.client.0.smithi145.stdout:1 2021-01-15T19:11:47.096 INFO:tasks.workunit.client.0.smithi145.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1006046,v1:127.0.0.1:6803/1006046] [v2:127.0.0.1:6804/1006046,v1:127.0.0.1:6805/1006046] exists,up d58832c9-29f6-43c0-afd8-0dd99e4f8977 2021-01-15T19:11:47.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: status=0 2021-01-15T19:11:47.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:926: wait_for_osd: break 2021-01-15T19:11:47.097 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: wait_for_osd: return 0 2021-01-15T19:11:47.098 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2021-01-15T19:11:47.862 INFO:tasks.workunit.client.0.smithi145.stderr:pool 'foo' already exists 2021-01-15T19:11:47.873 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2021-01-15T19:11:47.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local cmd= 2021-01-15T19:11:47.874 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local num_active_clean=-1 2021-01-15T19:11:47.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local cur_active_clean 2021-01-15T19:11:47.875 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-01-15T19:11:47.875 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: get_timeout_delays 90 .1 2021-01-15T19:11:47.876 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -q -o xtrace 2021-01-15T19:11:47.876 INFO:tasks.workunit.client.0.smithi145.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: echo true 2021-01-15T19:11:47.876 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: local trace=true 2021-01-15T19:11:47.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: true 2021-01-15T19:11:47.877 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_timeout_delays: shopt -u -o xtrace 2021-01-15T19:11:47.963 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -a delays 2021-01-15T19:11:47.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1584: wait_for_clean: local -i loop=0 2021-01-15T19:11:47.964 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: flush_pg_stats 2021-01-15T19:11:47.965 INFO:tasks.workunit.client.0.smithi145.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: flush_pg_stats: local timeout=300 2021-01-15T19:11:47.965 INFO:tasks.workunit.client.0.smithi145.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: ceph osd ls 2021-01-15T19:11:48.032 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2021-01-15T19:11:48.033 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2021-01-15T19:11:48.033 DEBUG:teuthology.task.console_log:Killing console logger for smithi145 2021-01-15T19:11:48.034 DEBUG:teuthology.exit:Finished running handlers